In T195594#4582085, @mepps wrote:

@Ejegg I'm seeing a patch merged here. Is this task Pending Deployment?

In T203669#4580601, @elukey wrote:

@AndyRussG Hi! I'd like to ask you a couple of questions before starting to work on this task.. are eventlogging_CentralNoticeImpression and eventlogging_LandingPageImpression good authoritative sources for banner impression data instead of filtering webrequest logs?

@Gilles Apologies for the delay... Is it ok with you if this goes on next week's deployment train? (Also replied to your e-mail... Apologies again!!!)

Hi! Some notes here:

• As far as I can tell, by requesting that CentralNotice administrators refrain from creating or editing banners with translatable messages, we're preventing any affectation of cluster performance due to the specific problem described in this bug (incredibly slow save times for banners with translatable messages).
• Nonetheless, we do need a careful review of CentralNotice-Translate interaction, to improve performance and code sanity in general, which includes this specific bug.
• The option suggested by @Nikerabbit is the immediate action we're pursuing, though that should also not be deployed until we're clear about its impact on all bits and pieces of this complex system.
• @Krenair, indeed we wouldn't disable CN-Translate integration without looking carefully at the impact, and, also, that doesn't look like the right option at this point :)

In T203925#4574882, @Nikerabbit wrote:

While checking callers of TranslateMetadata I found this code: https://gerrit.wikimedia.org/g/mediawiki/extensions/CentralNotice/+/e8f750a205e1ae61d694d6360c050bff724e7d03/includes/Banner.php#1008

It seems to be running MessageIndexRebuildJob during the request! This should happen in the JobQueue. No wonder saving will take a long time if this is indeed the case.

It also calls MessageGroups::clearCache while MessageGroups::recache is preferred to avoid multiple threads hurrying to rebuild the deleted cache.

@jcrespo @Nikerabbit Thanks!!! I think we could potentially look at disabling CentralNotice-Translate integration as a measure of last resort. (Note that we have asked CentralNotice administrators to not use translatable messages in banners for now, also.)

Just to note, though the general slow-to-save banner issue is important, that problem, as far as I can tell, has a different cause from very-very-slow-to-save problem with banners containing translatable messages...

In T203925#4571382, @jcrespo wrote:

Could this be related to T195293 ?

Hi! Looked at this quickly:

• This does not seem to be affecting banner displays, as far as I can see.
• There are a lot of Database warnings related to translatable messages.

Tested the update script locally--works so far!

So, the SQL to create the testing database should just be whatever's in sql/create_tables.sql in FRUEC.

Hi! Thanks much for finding this cruft!!!

Updated SQL to create the testing database, as per

@matmarex @Tgr Interesting! Well, if permission changes are pretty easy and not a risk to CentralNotice infrastructure ahead of the year-end fundraising campaigns, I'm ok with it!!! Moving CN pages out of the Mediawiki namespace is probably eventually a good idea (for previous discussions, see T33595 and this RFC). And, in general, the CentralNotice backend and administration interface need quite a lot of reworking--it has to be carefully planned and executed, though.

Hi! Just a few notes here:

• Inline JS is used quite a lot in CentralNotice banners, for both community and Fundraising campaigns. However, many campaigns don't really need it.
• The reason Meta Admins can edit CentralNotice banners is that banners are stored as Mediawiki namespace pages. So, anyone with the ability to edit interface messages (i.e. Admins) can edit banners.

It looks like there is tracking of all navigation to external sites with this. For example, when I clicked on "Donate Now", the following request was sent:
https://pixel.wp.com/c.gif?s=2&u=https%3A%2F%2Fdonate.wikimedia.org%2F&r=&b=147851870&p=10&rand=0.17689332155574422

There's now (in review) some basic test data for both CentralNotice and LandingPage events. However, at least the CN data should be need to be more varied. Also, for CN, the filenames should be changed to match those currently used on the cluster. Thanks!!!

I think perhaps this is all done? (As per comment on merged-in task, the log files are getting written as needed, thx much!!!)

@Jgreen yeah looks great! Apologies for the delay in checking this out :) Thx much!!!!!

In T200853#4498612, @Pcoombe wrote:

errorMsg

"Banner message key Centralnotice-template-spatton_dsk_sm_optin_pponly could not be found in en"

A first glance at the error logs shows a spike in "Banner message key" warnings during the first week in August. This would likely point to a problem with i18n caching. There's still a lot more digging to do, though... :)

Location of the log files to be consumed into the database in the new system: alnitak:/srv/banner_logs/2018/. (There's a lot in that directory... To find some, try: ls /srv/banner_logs/2018/ | grep -v beacon | tail -n 10

Fantastic, thanks @Pcoombe!!!

Temporary repo: https://github.com/AndrewGreen/fr_user_event_consumer
Notes: https://etherpad.wikimedia.org/p/fr_user_event_consumer_notes

In T198752#4494301, @jgleeson wrote:

Hey Andy, could we chat about this when convenient?

@Pcoombe ok thanks that's an important detail!

In T200449#4469076, @Ejegg wrote:

duplicate of T197602 ?

In T189820#4297415, @Ottomata wrote:

I'm not 100% sure how kafkatee does sampling, but I bet you it just filters for every Nth message. E.g. 1/100 would get you every 100th message.

A few notes from discussions on IRC...

Part of this should be to update or remove and archive oudated documentation about DjangoBannerStats and other bits of the pipeline that have changed. Here are a few bits that I found (though there may well be more):