Page MenuHomePhabricator

Translation page does not contain the latest translations/last translation
Open, HighPublic

Description

Current status

  • Waiting for a proposed fix to be deployed - cannot be tested locally

Once a fix is confirmed, we should run refresh-translatable-pages.php script again.

Original report

For eg: Saving https://meta.wikimedia.org/w/index.php?title=Steward_requests/Global_permissions/GS-Header&diff=20168333&oldid=19016463 did not updated the result on https://meta.wikimedia.org/wiki/Special:MyLanguage/Steward_requests/Global_permissions so I have to do a dummy edit https://meta.wikimedia.org/w/index.php?title=Steward_requests/Global_permissions/GS-Header&diff=20168363&oldid=20168333 to make it work.

Previous reports about similar issues:

QA plan

Affected projects: Wikimedia projects with Translate
QA person: @Nikerabbit / @abi_
QA task post deployment: Test (re-)marking page for translations, make changes to the template and see if those are applied after re-marking without need of dummy edits and re-marking.

Expected outcome

All translation pages are up to date without the need to do dummy edits.

Event Timeline

1997kB created this task.Sat, Jun 13, 6:57 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSat, Jun 13, 6:57 AM
Pppery added a subscriber: Pppery.Mon, Jun 15, 4:39 AM
1997kB updated the task description. (Show Details)Fri, Jun 19, 4:49 AM
Nikerabbit updated the task description. (Show Details)Mon, Jun 22, 1:22 PM

I would like to get more feedback to help investigate the task. So far I see a couple of reports where a template update did not show up in a page where the template is tranclused. This is something that is likely out of our control, as MediaWiki core handles dependency tracking and refreshing of pages due to dependency changes.

The example of Nemo_bis is different, however, because even though it is a template page, it clearly shows that a change to the template was not updated to the template itself until a dummy edit happened. This is clearly an issue with our code.

So please, do report more examples of this issue, and do provide the following info:

  • Is it related to updating translations, or when marking a page for translation
  • Link to the original edit
  • Link that shows when the page was marked for translation (if applicable)
  • Link to the page where you did not see the edit, and timestamp when you checked¹

¹ We rely on the job queue for many operations, so for example updating of translation pages is not instant. Do wait at least a few minutes to see if it is just a normal delay.

Looking at example Nemo_bis linked:

WhenWhat
Andre made an edit to Template:OPW2020-05-31T19:19:00‎Z
Shirayuki re-marked the page for translation2020-05-31T22:09:25Z
Nemo_bis made a dummy edit2020-06-12T08:21:24‎Z
Nemo_bis re-marked the page for ftranslation2020-06-12T08:21:32Z
FuzzyBot updates Template:OPW/en¹2020-06-12T08:21:32‎Z

¹It includes changes Andre made, but not the one Nemo_bis made.

Here are logs from around the time Shirayuki re-marked the page for translation:

NOTE: Since Logstash only has one second precisions for events, the ordering of messages may not be accurate.
2020-05-31T22:09:25mw1311INFO[Job: TranslationsUpdateJob][Request ID: 3e0e5da2-b696-4e13-940b-cd5ea86ca2bc][Title: Template:OPW] Starting TranslationsUpdateJob
2020-05-31T22:09:26mw1311INFO[Job: TranslationsUpdateJob][Request ID: 3e0e5da2-b696-4e13-940b-cd5ea86ca2bc][Title: Template:OPW] Finished running 3 MessageUpdate jobs for 3 sections
2020-05-31T22:09:26mw1311INFO[Job: TranslationsUpdateJob][Request ID: 3e0e5da2-b696-4e13-940b-cd5ea86ca2bc][Title: Template:OPW] Cleared caches
2020-05-31T22:09:26mw1311INFO[Job: TranslationsUpdateJob][Request ID: 3e0e5da2-b696-4e13-940b-cd5ea86ca2bc][Title: Template:OPW] Updated the message group stats
2020-05-31T22:09:26mw1311INFO[Job: TranslationsUpdateJob][Request ID: 3e0e5da2-b696-4e13-940b-cd5ea86ca2bc][Title: Template:OPW] Finished purging
2020-05-31T22:09:26mw1311INFO[Job: TranslationsUpdateJob][Request ID: 3e0e5da2-b696-4e13-940b-cd5ea86ca2bc][Title: Template:OPW] Added 3 RenderJobs to the queue
2020-05-31T22:09:26mw1334INFO[Job: TranslateRenderJob][Request ID: 3e0e5da2-b696-4e13-940b-cd5ea86ca2bc][Title: Template:OPW/ja] Starting TranslateRenderJob
2020-05-31T22:09:26mw1302INFO[Job: TranslateRenderJob][Request ID: 3e0e5da2-b696-4e13-940b-cd5ea86ca2bc][Title: Template:OPW/en] Starting TranslateRenderJob
2020-05-31T22:09:26mw1311INFO[Job: TranslationsUpdateJob][Request ID: 3e0e5da2-b696-4e13-940b-cd5ea86ca2bc][Title: Template:OPW] Finished TranslationsUpdateJob
2020-05-31T22:09:26mw1310INFO[MessageIndex] Started rebuild. Initiated by MediaWiki\Extension\EventBus\JobExecutor->execute/MessageIndexRebuildJob->run/MessageIndex->rebuild
2020-05-31T22:09:26mw1310INFO[MessageIndex] Got lock in 0.00070500373840332
2020-05-31T22:09:27mw1334INFO[Job: TranslateRenderJob][Request ID: 3e0e5da2-b696-4e13-940b-cd5ea86ca2bc][Title: Template:OPW/ja] Finished page edit operation
2020-05-31T22:09:27mw1334INFO[Job: TranslateRenderJob][Request ID: 3e0e5da2-b696-4e13-940b-cd5ea86ca2bc][Title: Template:OPW/ja] Finished TranslateRenderJob
2020-05-31T22:09:27mw1302INFO[Job: TranslateRenderJob][Request ID: 3e0e5da2-b696-4e13-940b-cd5ea86ca2bc][Title: Template:OPW/en] Finished TranslateRenderJob
2020-05-31T22:09:27mw1302INFO[Job: TranslateRenderJob][Request ID: 3e0e5da2-b696-4e13-940b-cd5ea86ca2bc][Title: Template:OPW/en] Finished page edit operation
2020-05-31T22:09:31mw1310INFO[MessageIndex] Finished critical section in 4.7386178970337
NOTE: I used wiki:mediawikiwiki AND (channel:Translate OR channel:Translate.Jobs) as the Logstash query

I also checked other logs from wiki:mediawikiwiki between 2020-05-31 22:09:23.939 and 2020-05-31 22:09:32.620 and no relevant warnings or errors.

Conclusion: There seems to be still the case where TranslateRenderJob does not see the latest version.

Also, rETRA09d11f488fc0: Split message index rebuild off of TranslationsUpdateJob and enable… is a recent change that may have introduced the regression.

Nikerabbit added a comment.EditedTue, Jun 23, 11:13 AM

Some quick theoretical musings:

The above code change moved slow message index rebuild job out of the critical path. Now TanslateRenderJobs are processed almost immediately. Previously the slow message index rebuild would hide any replication lag induced issues.

Both loading the message definitions and translations are guarded by:

	public static function getSafeReadDB() {
		$lb = MediaWikiServices::getInstance()->getDBLoadBalancer();
		$index = self::shouldReadFromMaster() ? DB_MASTER : DB_REPLICA;

		return $lb->getConnection( $index );
	}

Which uses:

	public static function shouldReadFromMaster() {
		$lb = MediaWikiServices::getInstance()->getDBLoadBalancer();
		// Parsing APIs need POST for payloads but are read-only, so avoid spamming
		// the master then. No good way to check this at the moment...
		if ( PageTranslationHooks::$renderingContext ) {
			return false;
		}

		return PHP_SAPI === 'cli' ||
			RequestContext::getMain()->getRequest()->wasPosted() ||
			$lb->hasOrMadeRecentMasterChanges();
	}

Looking at the code for Special:RunJobs, it requires the use of POST, so this should always return DB_MASTER in the context of TranslateRenderJob at Wikimedia wikis.

Then I realize, there is still third item that is loaded separately: the template (text outside <translate> tags), and this is exactly the case with this example. Inspecting the code, I see TranslatablePage::getText does not use getSafeReadDB.

Also, the job itself could use LoadBalancer's getMasterPos and waitFor to make all this redundant. But I don't see any extensions using it so it seems a bit risky.

Change 607268 had a related patch set uploaded (by Nikerabbit; owner: Nikerabbit):
[mediawiki/extensions/Translate@master] TranslatablePage: Apply replica/master logic to also getText

https://gerrit.wikimedia.org/r/607268

Change 607268 merged by jenkins-bot:
[mediawiki/extensions/Translate@master] TranslatablePage: Apply replica/master logic to also getText

https://gerrit.wikimedia.org/r/607268

Pppery removed a subscriber: Pppery.Tue, Jun 23, 4:57 PM
Nikerabbit updated the task description. (Show Details)Wed, Jun 24, 8:10 AM
Nikerabbit added a subscriber: abi_.

I would like to get more feedback to help investigate the task. So far I see a couple of reports where a template update did not show up in a page where the template is tranclused. This is something that is likely out of our control, as MediaWiki core handles dependency tracking and refreshing of pages due to dependency changes.

The example of Nemo_bis is different, however, because even though it is a template page, it clearly shows that a change to the template was not updated to the template itself until a dummy edit happened. This is clearly an issue with our code.

So please, do report more examples of this issue, and do provide the following info:

  • Is it related to updating translations, or when marking a page for translation
  • Link to the original edit
  • Link that shows when the page was marked for translation (if applicable)
  • Link to the page where you did not see the edit, and timestamp when you checked¹

¹ We rely on the job queue for many operations, so for example updating of translation pages is not instant. Do wait at least a few minutes to see if it is just a normal delay.

Well looking at template page again it's same issue as Nemo_bis highlighted. See following for example:
Main page which have changes with edit summary dummy edit https://meta.wikimedia.org/w/index.php?title=Template:Steward_requests/Username_changes&action=history
Translation page which do not have latest changes https://meta.wikimedia.org/w/index.php?title=Template:Steward_requests/Username_changes/es&action=history

Nikerabbit triaged this task as High priority.Thu, Jun 25, 11:08 AM