Spinning out a couple related tasks:

• T196563: Write a specification for mapping banner/landing page impression event properties -> database schema
• T196564: DB schemas (production changes and test DB) and SQL commands to run for new banner and LP impression data from EL

As per T186048#4189098, dimensions and measures available in Druid seem unlikely to change. So we shouldn't actually have to change anything in centralnotice_analytics. So I'll close this task as invalid. (If things change, we can re-open it...) Thanks!!!

In T193601#4180834, @Krinkle wrote:

On topic of CentralNotice, the internal review has spawned T192623 which is going to eliminate 1 of the serial roundtrips required for CentralNotice banner display (as well as for everything else!).

CSP warnings have now landed in core, should go out on the train pretty soon: https://gerrit.wikimedia.org/r/#/c/253969

For CentralNotice, it should be pretty simple to set up a hook to run for this (or some similar mechanism). :)

I think this involves:

• a test Kafkatee instance,
• a test database, and
• a test DjangoBannerStats job and config to slurp up the files.

For the realtime job, are there any similar examples that have EventLogging sources that I could more or less copy?

OK! So we'll go with (A), that is, continue to use Kafkatee to write files, and slurp them with the python script.

As mentioned on Gerrit, the patch uploaded is a rough attempt in need of review by those familiar with these systems.

There are two fields that in Druid that we can't currently get with the pure event Data. One is country_matches_geocode, which indicates whether server-based geolocation gave the same result as the Geo cookie. (In a small number of cases, it doesn't.) The other is region.

[mediawiki/extensions/CentralNotice@master] Add geo region to client-side data and impression event

Schema updated, too: https://meta.wikimedia.org/wiki/Schema:CentralNoticeImpression

Mmm just one further note, if we could split the cache on country and get that data into ResourceLoaderContext, we could also send a lot less choices to each client, which would also be a win performancewise.

I don't know if mixins can affect the targeting logic. If they don't, they should only be loaded after all targeting logic has run

Thanks!! :) This is correct, however it's blocked by 1) lack of geolocation information from the load.php entry point and 2) inability to split the RL cache on country. If those limitations are solved, it'd be pretty simple to do this.

There are two fields that in Druid that we can't currently get with the pure event Data. One is country_matches_geocode, which indicates whether server-based geolocation gave the same result as the Geo cookie. (In a small number of cases, it doesn't.) The other is region.

(Note that Pivot is not working properly now for Banner impressions, due to some update somewhere, and new versions of Pivot no longer being FOSS. Replacement: https://superset.wikimedia.org/ . See also https://wikitech.wikimedia.org/wiki/Analytics/Systems/Superset .)

In T189820#4085413, @Ottomata wrote:

Just curious though. Are you sure you want to just write this stuff to a log file? Are you just putting it there for a bit? The less hacky pipeline for yall would be kafka -> into MySQL directly.

In T192839#4169228, @Ottomata wrote:

Yes, please let us know the schema and timeline for this, thank youuuu

In T192839#4158762, @Ottomata wrote:

The main downside is you don't have control over the name of the tables (they will be named SchemaName_<REVISION_ID>), and any new schema revisions will result in new tables being created.

@Jseddon Thanks!!

Remaining to-do:

Some of discussion on this same topic here: T189820: Create job to deliver the eventlogging_CentralNoticeImpression topic.

(Sorry, removed the previous comment since I put it on the wrong task... ;p )

Seems one option is to use Apache Flume with a JDBC sink...

Here's one possible issue: backfill in case of failure. In the current system, I don't think files with this data are ever deleted from the FR cluster, so if there's a problem with any part of the pipeline downstream of the files, we have practically no time limit on our ability to backfill.

It looks like python-kafka is definitely the way to go for receiving data from a Kafka producer and processing it in Python. It looks like we can adapt the current python script to received data this way and still keep much of the processing code as-is. Also probably a good opportunity to remove cruft and refactor parts of the script.

The patch is now deployed, so it should be possible to add a link to the message using wikitext.

It has been suggested that we use python-kafka now... Would this as a complement to, or instead of, the current Python scripts? How would they interact?

Also: T192839: [Spike] Plan ingress system to use with new Kafka topic for landing page and impression data.