So far:

• T185932: CentralNotice: use EventLogging instead of custom beacon
• T185933: Donatewiki: use EventLogging to log pageloads
• T186047: centralnotice_analytics: adapt ImpressionsQuery for EventLogging-based impressions recording
• T186048: Adapt ingress of CN data into Druid to EventLogging-based impression recording

Also:

• T189613: Make a rough timeline/roadmap to replace usage of Kafkatee by FR with an event-based system to count pageviews to donatewiki

Also: T192839: [Spike] Plan ingress system to use with new Kafka topic for landing page and impression data.

Change 430613 had a related patch set uploaded (by AndyRussG; owner: AndyRussG):
[operations/mediawiki-config@master] CentralNotice EventLogging banner impression data test

https://gerrit.wikimedia.org/r/430613

Change 430613 merged by jenkins-bot:
[operations/mediawiki-config@master] CentralNotice EventLogging banner impression data test

https://gerrit.wikimedia.org/r/430613

Mentioned in SAL (#wikimedia-operations) [2018-05-03T19:01:58Z] <thcipriani@tin> Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:430613|CentralNotice EventLogging banner impression data test]] T183978 (duration: 01m 04s)

Mentioned in SAL (#wikimedia-operations) [2018-05-04T00:55:09Z] <niharika29@tin> Synchronized wmf-config/CommonSettings.php: Temporary low-level activation of Eventlogging impression data for testing T183978 (duration: 01m 16s)

As of this writing, the open tasks attached to this epic do approximate what needs to be done. Probably as we move forward we'll think of more specific ones related to parallel deployment of the new system and switching off of the old one.

What the epic doesn't show is which tasks already have some progress, and what the priorities are. Here are some tasks that are already have some progress. All of these tasks are currently in review.

• T196563: Write a specification for mapping banner/landing page impression event properties -> database schema
• T196564: DB schemas (production changes and test DB) and SQL commands to run for new banner and LP impression data from EL
• T198752: Queries and maybe scripts to verify equivalence of data in new-Kafka-pipeline-testing and pgehres production databases
• T195594: New scripts to ingress data from Kafkatee into MySQL

I'd suggest first we finish there review on these tasks, verify the data on the part of the pipeline that is already active (EventLogging data), then maybe plan more details of the deployment process.

Also quite a bit of work has been done by Analytics folks towards the new Druid dataset (for Turnilo and Superset Web interfaces). See the tasks listed here: T186048#5205429.

Here's an overview of the status of this project:

• Data specification for the new pipeline is ready, except for possible adjustments for empty LandingPage values (see below).
• Log files are created from the new Kafka streams.
• Parallel testing database is on production.
• FRUEC (new ingress script) is functional and can be deployed and run as a job on production.
• No automated tests or CI yet.
• No attempts yet to run real queries for Advancement on data from the new pipeline.
• Major issue requiring attention: discrepancies between data in the old and new pipelines.

Details on discrepancies between new and old data pipelines:

• No significant discrepancies detected due to FRUEC script.
  • One minor, easily fixable issues has been found: different behaviour for empty string language code. See T237997.
  • Other similar, minor issues should be checked for. See T242065.
• Likely no major issues with Kafkatee filtering of events received from Analytics' Kafka streams. Task about further verification: T242022.
• Major issues exist regarding the initial sections of the pipelines, that is, how data is initially collected from the client.
  • For the CN (banner) pipeline, both old and new events are generated client-side.
    • There are 12-15% less events in the new CN pipeline, apparently due to some adblock filters blocking all client-side MW EventLogging events.
    • Analysis of data: T236834.
    • What to do: T239570.
  • For the Landing Page pipeline, the old events are generated from requests for the base page HTML, and new events are generated client-side.
    • There are many more events in the old LandingPage pipeline, probably due to bots, spiders, hacking attempts, and proxies for web accelerators that don't run JavaScript. This is unavoidable but OK, since it's likely that we don't actually want to count all or nearly all of these extra events.
    • Strangely, there are also consistently extra events in the new LandingPage pipeline. These are events in the new pipeline that don't have a corresponding event in the old pipeline. Try to figure out what's going on: T238592.
    • Since the new pipeline events arrive on the same EventLogging URL, it is likely that 12-15% are being blocked by adblock filters, just like CN pipeline events.
    • Analysis of data: T236835.
    • What to do: T237553.