Hi! Here's some documentation I made about this process.

Also, before we merge master to wmf_deploy we should properly revert this change that was accidentally +2'ed only on the wmf_deploy. (We thought we'd reverted it but the revert patch in the end didn't merge because of CI. The version of the change on master is waiting for an update from the author.)

In T246461#5945348, @mepps wrote:

These are the patches I see waiting for review (aside from Geotargeting and Banner templates). Do we have others @AndyRussG?

• https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralNotice/+/576528
• https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralNotice/+/549202
• https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralNotice/+/498480

In T245285#5893599, @Ottomata wrote:

There should be 90 days available now, but the sanitize_delayed job will start sanitizing from 45 days ago once refinery is deployed. @AndyRussG, do you need the 90-45 days old data to be manually sanitized into the event_sanitized database too, or is starting from 45 days ago ok.

In T245285#5893524, @Nuria wrote:

This would take effect once we deploy it but Analytics needs to run a refine job from the begging of the data stream being available. cc @fdans (who has the ops week) and @Ottomata

if the sampling ratio of the data on that table is good enough then we can help you set up workflows to read from that table the banner counts in question.

See also T192564: [Epic] Banner history improvements for some ideas we've had over the years on how to improve these logs.

Just a few initial thoughts:

• Pre-processing might include flagging logs as randomly sampled or sent following a click to donate.
• Time zone and geographical region are probably needed. For the pageview on which the log was sent back, this is in the unsanitized Hive table. Better would be to have it for every entry in the log. In either case, legal review would probably be needed. See also T170559.
• The article viewed is purposefully not included in the logs, though it would be possible to find this information for the pageview on which the log was sent to the servers, using the Hive WebRequest table.

In T244882#5878203, @jkumalah wrote:

banner history has been white listed by the analytics team. See https://phabricator.wikimedia.org/T161656. According to Nuria banner data will be stored on events_sanitized just like it is done for other eventlogging schemas.
https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/Data_retention_and_auto-purging#The_white-list

In T244771#5877159, @Ottomata wrote:

FYI, if you need a new datatype, you should just make a new field :) It isn't clear how you'd query a field that has two different types at different dates anyway. :)

Hi! Thanks much, @Nuria, @Ottomata, @DStrine, @jkumalah :) Apologies for the delay in replying...

Here are the queries we used for this:

To test the fix locally:

• Set up two campaigns, one higher priority than the other.
• Include a close button in the banner for the campaign with higher priority.
• The higher priority campaign should always show, until the close button is clicked on the banner. On subsequent pageviews after the button has been clicked, the lower priority campaign should show.
• To test again, clear cookies for the local wiki.

In T240802#5797791, @mepps wrote:

@AndyRussG Okay, I can see what you mean. When I look at the loop what I see is that state.setCampaign is called early on and that should set the status to CAMPAIGN_CHOSEN for the campaign that gets selected. cancelBanner is called if hide.shouldHide() is true. Just recording my findings as I go.

In T240802#5790095, @mepps wrote:

@AndyRussG Is there any documentation of what these status codes mean?

Here's an overview of the status of this project:

Just updated the draft specification for this. We should also look at legacy default values for empty country and project fields.

In T236834#5737459, @Jgreen wrote:

We've set up a new kafkatee collector host at codfw, frban2001, which will replace the aged alnitak once we're confident everything is working properly on Debian Buster etc. In the meantime it's useful for comparison, to help rule out losses in the kafka->kafkatee->logfile segment in the pipeline. I ran all of yesterdays new-pipeline landingpages json logs through a simple perl json decoder and observed that alnitak and frban2001 collected exactly the same data for 12/11. The script I used is at mintaka:/tmp/json-log-filter, and frban2001's log store is mounted at mintaka:/mnt/banner_logs_new. The beacon-impressions logs are harder to compare since they're sampled at 1:10, and the two hosts are expected to collect different messages. So we set frban1001 to collecting beacon-impressions 1:1 for a day, and at the end we'll compare what was collected to what's in Hive.

As per T219604, it's just the RL modules aliases that have changed, but the modules are still around, so we should be able to fix pretty easily.

In T240505#5733732, @Jdforrester-WMF wrote:

jquery.ui.datepicker was removed in wmf.10, yes. wmf_deploy strikes yet again.

Thanks again, @sbassett and @Bawolff!

@Pcoombe Thanks so much for this fix!!! Looks great!

In T239922#5715938, @Bawolff wrote:
an attacker could just edit a page to introduce the trigger code instead of relying on url based injection.

@Pcoombe wrote (on IRC):
presumably the vulnerability is that code could be saved in a page. and they just used preloadtitle to avoid creating a page demonstrating it that anyone can see

@Bawolff, if I understand correctly, this is dependent on the injection into the DOM of content using the preloadtitle URL parameter, correct? I was unaware of this Mediawiki feature... now looking at the doc for this...

In T239922#5715700, @Pcoombe wrote:

I assume since the example contains %AVERAGE% it's the following code (and a similar bit for replacing %MINIMUM%) that occurs in https://meta.wikimedia.org/wiki/MediaWiki:FundraisingBanners/LocalizeJS-2017.js and hence all our banners

In T239778#5715514, @sbassett wrote:

Ugh, thanks for finding that, @Bawolff. I'm going to break that out into a separate high/ubn ticket.
Filed: T239919

@Bawolff thank you so much for finding this!!!! This same code is in live FR banners across enwiki. I've created a separate task: T239922

@Bawolff, thanks so much for finding this!!!!! Could you please confirm that this is the code in question, and explain a few more details about how it works?