Page MenuHomePhabricator

Publishing translations for central notice banners fails
Open, HighPublic2 Story Points

Description

When I was setting workflow state to 'Published', a "Change of state failed" error showed up. After reloading the page, it showed that it was set to 'Published'. But in the actual banner, it only shows part of the translation. See https://meta.wikimedia.org/w/index.php?title=Wikinews/Licensure_Poll/GFDL_CC-BY-SA/O/For&banner=wm2015register&uselang=ja&force=1 for example. https://meta.wikimedia.org/w/index.php?title=Special:Translate&action=proofread&group=Centralnotice-tgroup-wm2015register&language=ja

This is causing issues with the currently up global Wikimania banner.

Event Timeline

Glaisher created this task.Jul 4 2015, 6:03 PM
Glaisher raised the priority of this task from to Unbreak Now!.
Glaisher updated the task description. (Show Details)
Glaisher added subscribers: Glaisher, Nikerabbit, siebrand, ori.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 4 2015, 6:03 PM
Mbch331 added a subscriber: Mbch331.Jul 4 2015, 8:52 PM
atgo added a subscriber: atgo.
atgo added a subscriber: AndyRussG.Jul 6 2015, 8:13 PM

We're not sure if this is a centralnotice issue or something on the translate side but will investigate.

If the state was changed even though Change of state failed was displayed, I suspect that an exception or a fatal error happened during the TranslateEventMessageGroupStateChange hook.

Ejegg added a subscriber: Ejegg.Jul 6 2015, 9:36 PM

CentralNotice creates its messages in the MediaWiki namespace with this naming scheme:
https://meta.wikimedia.org/wiki/MediaWiki:Centralnotice-wm2015register-text1/ja
https://meta.wikimedia.org/wiki/MediaWiki:Centralnotice-wm2015register-text2/ja

Those two look like the same text that's showing up in the banner, so I'd guess this isn't a CentralNotice-specific thing. We do have a 'FundraisingTranslateWorkflow' that changes who is allowed to publish certain fr-related translations, but I don't think it should affect this one. I'll try to make sure that's not a factor.

Ejegg added a comment.Jul 7 2015, 12:55 AM

I don't see any settings on meta changing the FTW title regex from the default ^page-Fundraising/, so that extension wouldn't be causing the interference.

atgo added a comment.Jul 7 2015, 10:39 PM

we're pretty sure this isn't in anything that fundraising owns... is someone digging from the MediaWiki-extensions-Translate side?

Hi, I am the creator of the Translate extension. I said above already, regarding the error message described in the initial report, I found no evidence that it is happening on Translate side either. This is further confirmed that the event got logged at https://meta.wikimedia.org/w/index.php?title=Special%3ALog&type=translationreview&user=Glaisher&page=&year=&month=-1&tagfilter=. Only thing happening after that is the aforementioned hook, and what happens there is not controlled by Translate.

If there are some other symptoms or errors, please clarify.

DStrine moved this task from Done to Doing on the Fundraising Sprint N*E*R*D board.Jul 8 2015, 9:05 PM
DStrine moved this task from Doing to Backlog on the Fundraising Sprint N*E*R*D board.
AndyRussG moved this task from Backlog to Doing on the Fundraising Sprint N*E*R*D board.
AndyRussG set Security to None.

Hi! Two questions:

  • Are there any other reports of this happening?
  • What would you suggest as the best way to fix these specific translations so this banner displays as it should?
atgo added a subscriber: Pcoombe.Jul 9 2015, 5:32 PM
AndyRussG added a comment.EditedJul 10 2015, 5:58 AM

I edited both translated messages (added and removed a space), then set the state back to Published. I got the same "Change of state failed" error.
However, the content of the messages in the Mediawiki namespace was updated. (See https://meta.wikimedia.org/wiki/MediaWiki:Centralnotice-wm2015register-text1/ja and https://meta.wikimedia.org/wiki/MediaWiki:Centralnotice-wm2015register-text2/ja .)
Yet I don't see the updated text showing up in the banner itself... (Maybe there's a longish cache delay?)
If we can unbreak this banner, can we then adjust the priority of this task on the basis of how many reports of the same bug we get?
Thanks!!! :)

I don't have permissions to set the state to published to test myself what the error is.

The banner is fixed now! It started to appear correctly after we manually edited (added and removed a space, again) the messages in the Mediawiki: namespace. We're now digging deeper into what actually happened.

Please let us know if you see any further problems with this specific banner, or if you come across any other issues similar to or possibly related to this. Thanks!! :)

AndyRussG added a comment.EditedJul 11 2015, 2:31 PM

I'm able to reproduce the error by changing the state of the message group for this banner from Published to Ready and then back to Published. (This is the same message group the error was originally reported for: https://meta.wikimedia.org/w/index.php?title=Special:Translate&action=proofread&group=Centralnotice-tgroup-wm2015register&language=ja&filter=translated .)

When I changed the state back to published, there was a background request to api.php that never returned. We haven't yet found anything in the error log. Maybe someone from Operations can help us track it down? Here are the request params:

action:"groupreview"
format:"json"
group:"Centralnotice-tgroup-wm2015register"
language:"ja"
state:"published"
token:"{{{sekret}}}"

The request happened near 23:52:42 UTC on July 10th.

Also, since the banner itself is fixed and we have no other reports of this bug so far, I'm lowering the priority a bit. Hope that's OK!

Thanks!!! :)

AndyRussG lowered the priority of this task from Unbreak Now! to High.Jul 11 2015, 2:32 PM
AndyRussG added a project: acl*sre-team.
Restricted Application added a subscriber: Matanya. · View Herald TranscriptJul 11 2015, 2:32 PM
awight added a subscriber: awight.Jul 12 2015, 5:36 AM

Here's something interesting from fluorine:/a/mw-log/archive/exception.log-20150710.gz:

It takes two to deadlock...

2015-07-10 06:20:00 mw1003 wikimania2015wiki exception INFO: [f3ab2529] /rpc/RunJobs.php?wiki=wikimania2015wiki&type=MessageUpdateJob&maxtime=
60&maxmem=300M   DBQueryError from line 1283 of /srv/mediawiki/php-1.26wmf13/includes/db/Database.php: A database error has occurred. Did you 
forget to run maintenance/update.php after upgrading?  See: https://www.mediawiki.org/wiki/Manual:Upgrading#Run_the_update_script
Query: INSERT IGNORE INTO `translate_groupstats` (tgs_group,tgs_lang,tgs_total,tgs_translated,tgs_fuzzy,tgs_proofread) VALUES ('page-Food and 
Drink','en','40','35','1','0')
Function: MessageGroupStats::queueUpdates
Error: 1213 Deadlock found when trying to get lock; try restarting transaction (XXX)

#0 /srv/mediawiki/php-1.26wmf13/includes/db/Database.php(1238): DatabaseBase->reportQueryError()
#1 /srv/mediawiki/php-1.26wmf13/includes/db/Database.php(2124): DatabaseBase->query()
#2 /srv/mediawiki/php-1.26wmf13/extensions/Translate/utils/MessageGroupStats.php(459): DatabaseBase->insert()
#3 /srv/mediawiki/php-1.26wmf13/extensions/Translate/utils/MessageGroupStats.php(474): Closure$MessageGroupStats::queueUpdates()
#4 /srv/mediawiki/php-1.26wmf13/includes/db/Database.php(3493): Closure$MessageGroupStats::runWithLock()
#5 /srv/mediawiki/php-1.26wmf13/includes/db/Database.php(3656): DatabaseBase->runOnTransactionIdleCallbacks()
#6 /srv/mediawiki/php-1.26wmf13/includes/page/WikiPage.php(1802): DatabaseBase->begin()
#7 /srv/mediawiki/php-1.26wmf13/extensions/Translate/tag/TranslateRenderJob.php(64): WikiPage->doEditContent()
#8 /srv/mediawiki/php-1.26wmf13/extensions/Translate/tag/PageTranslationHooks.php(159): TranslateRenderJob->run()
#9 /srv/mediawiki/php-1.26wmf13/extensions/Translate/tag/PageTranslationHooks.php(139): PageTranslationHooks::updateTranslationPage()
#10 /srv/mediawiki/php-1.26wmf13/includes/Hooks.php(204): PageTranslationHooks::onSectionSave()
#11 /srv/mediawiki/php-1.26wmf13/includes/page/WikiPage.php(1963): Hooks::run()
#12 /srv/mediawiki/php-1.26wmf13/extensions/Translate/utils/MessageUpdateJob.php(44): WikiPage->doEditContent()
#13 /srv/mediawiki/php-1.26wmf13/includes/jobqueue/JobRunner.php(183): MessageUpdateJob->run()
#14 /srv/mediawiki/rpc/RunJobs.php(42): JobRunner->run()
#15 {main} {"private":false}
awight edited a custom field.Jul 22 2015, 7:46 PM

Thanks everyone for looking into this. I am still having issues when changing the state to 'Published' and get the warning message 'change of state failed'

. It would be really great to get this fixed as we rely on this feature for our banners.

I spoke to @Amire80 this morning on a different topic who mentioned that the language engineering team might be able to help to get this fixed. Thanks for letting us know if that is a possibility Amir.

@jrobell thanks a lot for mentioning this! I didn't know it was widespread enough to block anything. We'll try to get to it soon!!!

atgo removed a subscriber: atgo.Mar 30 2016, 10:05 PM

I doubt this would be helped by T116235. This is typically a performance issue caused by the number of message groups, which are not going to decrease if you change them to a different kind of message group.

I doubt this would be helped by T116235. This is typically a performance issue caused by the number of message groups, which are not going to decrease if you change them to a different kind of message group.

Evidence for "large number of message groups causes performance issue"?

mmodell removed a subscriber: awight.Jun 22 2017, 9:38 PM
DStrine removed cwdent as the assignee of this task.May 8 2019, 8:21 PM
DStrine moved this task from Sprint Q -- to Q1 (July-Sept) 2019-20 on the Fundraising-Backlog board.
DStrine added a subscriber: cwdent.