Page MenuHomePhabricator
Create Task
Maniphest T267520

Translation pages are updated after a long delay on Wikimedia sites
Closed, ResolvedPublic
Actions

Description

There has been reports of translation page updates taking a long time, even over 30 minutes. We developers knew that it would get delayed when we switched to using a JobQueue, but this is a huge regression from a couple second of delay it used to be.

Reports:

Suggested plan of action

  1. Confirm that this is just caused by Job Queue slowness and not any errors in Translate
  2. Check if this types of jobs can be prioritized
  3. Considering bringing up these delays to Job Queue infrastructure maintainers, if we see that there are constantly long delays

Details

ProjectBranchLines +/-Subject
operations/deployment-chartsmaster+9 -0JobQueue: Move translation jobs to its own queue
operations/deployment-chartsmaster+5 -0JobQueue: Move LocalGlobalUserPageCacheUpdateJob to it's own queue.
Customize query in gerrit

Related Objects

Event Timeline

Nikerabbit created this task.Nov 9 2020, 8:38 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 9 2020, 8:38 AM
Nikerabbit triaged this task as High priority.Nov 9 2020, 8:41 AM
Nikerabbit moved this task from Backlog to maintenance and operational issues on the MediaWiki-extensions-Translate board.

Why I think this is high priority: We (Translate developers) have been working on making translation page updates robust. The code and funcionality is still fresh in our minds so we should look at and address this issue before we declare ourselves done with this area.

Nikerabbit updated the task description. (Show Details)Nov 9 2020, 8:45 AM
Pchelolo added a subscriber: Pchelolo.Nov 9 2020, 1:37 PM

Can you please list the names of your jobs?

https://grafana.wikimedia.org/d/LSeAShkGz/jobqueue?orgId=1 has a lot of data about delays for the particular jobs, I can help look into it.

Nikerabbit added a comment.Nov 9 2020, 1:43 PM

The relevant jobs would be TranslationsUpdateJob and TranslateRenderJob. More info at https://www.mediawiki.org/wiki/Help:Extension:Translate/Process_flow_in_MediaWiki_jobs

Pchelolo added a project: Platform Team Workboards (Clinic Duty Team).Nov 9 2020, 1:45 PM
Nikerabbit added a comment.Nov 9 2020, 2:32 PM

I no longer find any settings for priority jobs, so I guess such thing no longer exists or is now moved somewhere where codesearch does not find it. In any case it would be problematic as refresh-translatable-pages.php maintenance script would spawn thousands of TranslateRenderJobs that would overwhelm the queue if they had a priority.

In other words, the only option available seems to be to investigate whether the jobs are at fault (errors or being excessively slow) or just taking time to be run through the job queue.

Looking at recent history of channel:Translate.Jobs in Logstash I don't see any obvious issues. Job execution durations are slow, and I do not see big delays between TranslationsUpdateJob and TranslateRenderJobs. There are some errors with code "edit-conflict" and "edit-already-exists". Sometimes few, sometimes a bunch for the same page in a short duration. These jobs are the only actors editing the affected pages, so they can only conflict with themselves (i.e. same job being executed multiple times or more than one job for the same page [which becomes more likely if there are long delays and the first job hasn't had chance to run before the new one(s) are being inserted]).

Pchelolo added a comment.Nov 10 2020, 4:02 PM

Ok, this was an easy one.

The translate jobs are rather low traffic, so they share the queue with other low traffic jobs. Apparently LocalGlobalUserPageCacheUpdateJob is prone to large spikes, blocking the low traffic job queue:

Screen Shot 2020-11-10 at 8.50.27 AM.png (522×2 px, 146 KB)

We are going to move the LocalGlobalUserPageCacheUpdateJob out of the low traffic queue and give it its own queue. This should resolve this problem. If there's ever again a slowdown of translate jobs, we will give them a designated queue.

gerritbot added a comment.Nov 10 2020, 4:03 PM

Change 640446 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/deployment-charts@master] JobQueue: Move LocalGlobalUserPageCacheUpdateJob to it's own queue.

https://gerrit.wikimedia.org/r/640446

gerritbot added a project: Patch-For-Review.Nov 10 2020, 4:03 PM
Pchelolo assigned this task to • Clarakosi.Nov 10 2020, 4:04 PM
Pchelolo moved this task from Inbox to Waiting for deployment on the Platform Team Workboards (Clinic Duty Team) board.
Tacsipacsi mentioned this in T267521: Unnecesary language tags are appearing on source language translation pages.Nov 12 2020, 7:50 PM
Tacsipacsi added a subscriber: Tacsipacsi.
Xiplus added a subscriber: Xiplus.Nov 15 2020, 2:48 PM
gerritbot added a comment.Nov 17 2020, 4:22 PM

Change 640446 merged by jenkins-bot:
[operations/deployment-charts@master] JobQueue: Move LocalGlobalUserPageCacheUpdateJob to it's own queue.

https://gerrit.wikimedia.org/r/640446

Pchelolo closed this task as Resolved.Nov 17 2020, 4:40 PM

This issue should stop occurring now. please reopen if it comes back.

Pchelolo moved this task from Waiting for deployment to Done on the Platform Team Workboards (Clinic Duty Team) board.Nov 17 2020, 4:40 PM
Maintenance_bot removed a project: Patch-For-Review.Nov 17 2020, 5:10 PM
Ciencia_Al_Poder reopened this task as Open.Dec 2 2020, 3:57 PM
Ciencia_Al_Poder added a subscriber: Ciencia_Al_Poder.

Looks like this is happening again. Reported today at Topic:Vyupiut0evae00hd

Looking at Wladek92 contributions, the translation unit Translations:Manual:GenerateJsonI18n.php/Page display title/fr (the first for the Manual:GenerateJsonI18n.php/fr page) was created at 15:12 UTC, but the corresponding page that contains all translations, namely Manual:GenerateJsonI18n.php/fr was created at 15:25 UTC with an edit summary corresponding to the last translation unit edited.

That page was created at the same time as Manual:$wgMessagesDirs/fr (15:25 UTC) but the first translation unit was created at 15:08 UTC (17 minutes of lag).

Wladek92 added a subscriber: Wladek92.EditedDec 2 2020, 6:08 PM

yes, same situation 2 hours later.

Magic trick I early used: I reopen the first msg to translate and add a blank at the end (which changes nothing) and validate. History is unchanged but now 'français' appears in the list !!!

Wladek92 added a comment.Dec 6 2020, 11:43 AM

Normal behaviour has been observed back today. Item shown immediately after translating a random message from within the page. No specific action done.

• Clarakosi added a comment.Dec 9 2020, 3:33 PM

Seems there was another backlog in low traffic jobs that caused this delay on Dec 2nd.

Screen Shot 2020-12-08 at 5.19.40 PM.png (730×2 px, 117 KB)

We'll go ahead and move translate jobs to its own queue.

gerritbot added a comment.Dec 9 2020, 4:56 PM

Change 647307 had a related patch set uploaded (by Clarakosi; owner: Clarakosi):
[operations/deployment-charts@master] JobQueue: Move translation jobs to its own queue

https://gerrit.wikimedia.org/r/647307

gerritbot added a project: Patch-For-Review.Dec 9 2020, 4:56 PM
gerritbot added a comment.Dec 9 2020, 5:54 PM

Change 647307 merged by jenkins-bot:
[operations/deployment-charts@master] JobQueue: Move translation jobs to its own queue

https://gerrit.wikimedia.org/r/647307

Maintenance_bot removed a project: Patch-For-Review.Dec 9 2020, 6:10 PM
• Clarakosi closed this task as Resolved.Dec 9 2020, 6:14 PM

The change has been deployed and shouldn't be happening anymore. Please feel free to reopen if the issue continues

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL