In T209763#4778844, @MarcoAurelio wrote:

@Ejegg Thanks for your reply. Would then:

"banner-protect": "Protect translatable banner messages handled by the translate extension"

be an appropriate i18n summary to display on Special:ListGroupRights?

@MarcoAurelio @Jdforrester-WMF Thanks!!!!!!

The specific error is related about database replication: "Replication wait failed: MySQL server has gone away". There are other errors on this banner that seem linked to rebuilding the Translate extension MessageGroupStats index. See these Logstash entries.

Thanks, @MarcoAurelio! Indeed, this should be documented properly. The banner-protect right is used to protect translatable banner messages handled by the translate extension. The goal is to ensure that translation managers can approve and publish translations for those messages without permission to modify other banner content.

Thanks @MarcoAurelio and @Seddon! It seems that this is the right way to go. If I understand correctly, the i18n aspect would be mostly of use outside the WMF, on wikis that use CentralNotice and are in languages other than English... Is that correct?

In T203669#4731161, @elukey wrote:

@AndyRussG @Jseddon Hi! So I have something to show to you in: https://turnilo.wikimedia.org/#event_centralnoticeimpression

We have a tool called Eventlogging2Druid now that automagically imports data from Eventlogging/Hive and index it to Druid. We indexed only a couple of days (Oct 27th/28th) in Druid as example, but you can play with the data and see if it fits your needs

@Jdforrester-WMF Thanks!! :)

Here's the Jupyter notebook with the queries used:

Hi! I've found a pretty convincing indication that the problem was with old ChoiceData being sent to browsers.

Some notes:

In T204396#4680574, @Nuria wrote:

Can you answer these questions

Hey... of course, apologies for the delays!!!

@Nuria, @JAllemandou, @mforns thanks so much taking the time to check this out, much appreciated!!! We can dig into it further (as priorities allow). :)

What this isn't:

• Not a more general Mediawiki or database issue, outage, or any known problem on the cluster.
• Not a campaign or banner configuration issue.
• Not a general CentralNotice outage.
• Not a data pipeline issue.

Checked a couple more things:

In T203669#4670568, @elukey wrote:

the size of every month varies a lot, so we were wondering about the consistency of the data over the past months.

In T203925#4668926, @CCicalese_WMF wrote:

What is the status of this with the recent changes merged? Is it still a high priority?

Also just re-checked CentralNotice logs... I don't see any changes in any of them around 18:40 on 2018-10-09, which is when the mobil frFR banner finally went out (according to Druid/Turnilo, see above).

Also nothing stands out in logstash for that time: search 1, search 2.

Just checking stuff to eliminate possible causes: there's nothing odd in the server logs for that time, nor were any actions related to the datacenter switchover set for that day (see T199073).

Thanks much for this!!!! I debugged through the merged patch locally. Unfortunately... it looks like it might not fix the problem, since, at least locally, the mergeable updates are never merged. This is because each is added to DeferredUpdates within the execution of a different AtomicSectionUpdate. (See the Gerrit change for more details.)

Thanks!! Checked the data in Turnilo (formerly Pivot), and it shows the problem happened as described: https://bit.ly/2NzT6OW

@elukey so just to confirm, it's fine to go ahead and use the eventlogging_CentralNoticeImpression stream for this. Thanks so much again and apologies again for the delays in replying!!!

Hi!!! Many apologies for the delay here...

In T206478#4650162, @Krenair wrote:

pgehres hasn't worked here for years

Thanks @Pcoombe... That looks like a significant clue!! In the logs, the related server errors seem to come in clusters... That might be related to when campaigns and tests are running, though. There are other MessageCache issues now (see T203925), so we might want to look at them all together.

In T203925#4626088, @AndyRussG wrote:

Here's a limited, temporary workaround for banners using translatable messages: use only two or three messages. Though save times will be long (1 minute or more), I think it'll work. You can include HTML fragments in the messages, so it should be doable, though cumbersome.

In T203925#4625929, @Krinkle wrote:

My intention with T193271 is to change MessageCache to only pre-cache MW-pages that represent interface message overrides for message keys recognised by the software. See that task for more details.

Currently, CentralNotice uses Revision::getContent() in the administration UI to show the content of a banner's translatable messages. So, as far as that UI is concerned, it's not essential to update MessageCache before the form returns. (Note: that part of the UI has been broken-ish for a while--see T72939.)

Hi! Thanks so much, everyone!!

Hi @zeljkofilipin... Many apologies for not responding here, and thanks for the sample test!!!!! Can you say how urgent this is? Also, is the lack of update maybe related to recent failures on OSX for the legacy system? Thanks so much and apologies again!

This may well be a less severe symptom as the same problem that's causing T203925: Save times for changes to translation variable text in centralnotice paralysingly slow. (More details here: T203925#4609441.)

@Peter, @Gilles, looks great!!! :)

Ooops! I was mistaken... This is not necessary, since we currently don't get data (via either pipeline) from banner previews (makes much more sense, really).

For contrast, the same mwrelp test on enwiki took about 2 seconds:

Looks like I've found the general area of code causing this. :)

In T195594#4582085, @mepps wrote:

@Ejegg I'm seeing a patch merged here. Is this task Pending Deployment?

In T203669#4580601, @elukey wrote:

@AndyRussG Hi! I'd like to ask you a couple of questions before starting to work on this task.. are eventlogging_CentralNoticeImpression and eventlogging_LandingPageImpression good authoritative sources for banner impression data instead of filtering webrequest logs?

@Gilles Apologies for the delay... Is it ok with you if this goes on next week's deployment train? (Also replied to your e-mail... Apologies again!!!)

Hi! Some notes here:

• As far as I can tell, by requesting that CentralNotice administrators refrain from creating or editing banners with translatable messages, we're preventing any affectation of cluster performance due to the specific problem described in this bug (incredibly slow save times for banners with translatable messages).
• Nonetheless, we do need a careful review of CentralNotice-Translate interaction, to improve performance and code sanity in general, which includes this specific bug.
• The option suggested by @Nikerabbit is the immediate action we're pursuing, though that should also not be deployed until we're clear about its impact on all bits and pieces of this complex system.
• @Krenair, indeed we wouldn't disable CN-Translate integration without looking carefully at the impact, and, also, that doesn't look like the right option at this point :)

In T203925#4574882, @Nikerabbit wrote:

While checking callers of TranslateMetadata I found this code: https://gerrit.wikimedia.org/g/mediawiki/extensions/CentralNotice/+/e8f750a205e1ae61d694d6360c050bff724e7d03/includes/Banner.php#1008

It seems to be running MessageIndexRebuildJob during the request! This should happen in the JobQueue. No wonder saving will take a long time if this is indeed the case.

It also calls MessageGroups::clearCache while MessageGroups::recache is preferred to avoid multiple threads hurrying to rebuild the deleted cache.

@jcrespo @Nikerabbit Thanks!!! I think we could potentially look at disabling CentralNotice-Translate integration as a measure of last resort. (Note that we have asked CentralNotice administrators to not use translatable messages in banners for now, also.)

Just to note, though the general slow-to-save banner issue is important, that problem, as far as I can tell, has a different cause from very-very-slow-to-save problem with banners containing translatable messages...

In T203925#4571382, @jcrespo wrote:

Could this be related to T195293 ?

Hi! Looked at this quickly:

• This does not seem to be affecting banner displays, as far as I can see.
• There are a lot of Database warnings related to translatable messages.