BraceBalanceValidator.php on line 33
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error

MediaWiki version: 1.35.0-wmf.37

message

Argument 1 passed to MediaWiki\Extensions\Translate\MessageValidator\Validators\BraceBalanceValidator::getBalance() must be of the type string, null given, called in /srv/mediawiki/php-1.35.0-wmf.37/extensions/Translate/src/MessageValidator/Validators/BraceBalanceValidator.php on line 33

Impact

Some post-save operations for translation units are failing.

Notes

There were just about 8 of these errors on 1.35.0-wmf.37, currently on group0 and group1. Not enough to be a blocker, but reporting in case it's a programming error that can be fixed.

The exception has been converted to a normal log entry.

Details

Request ID: cbdafd79-b4d9-4bb6-9383-e1691305e68b
Request URL: https://jobrunner.discovery.wmnet/rpc/RunSingleJob.php

Stack Trace

exception.trace

#0 /srv/mediawiki/php-1.35.0-wmf.37/extensions/TransSome messages late/src/MessageValidator/Validators/BraceBalanceValidator.php(33): MediaWiki\Extensions\Translate\MessageValidator\Validators\BraceBalanceValidator->getBalance(NULL, string, string)
#1 /srv/mediawiki/php-1.35.0-wmf.37/extensions/Translate/MessageValidator.php(327): MediaWiki\Extensions\Translate\MessageValidator\Validators\BraceBalanceValidator->getIssues(FatMessage, string)
#2 /srv/mediawiki/php-1.35.0-wmf.37/extensions/Translate/MessageValidator.php(168): MessageValidator->runValidation(array, FatMessage, string, MediaWiki\Extensions\Translate\Validation\ValidationIssues, MediaWiki\Extensions\Translate\Validation\ValidationIssues, boolean)
#3 /srv/mediawiki/php-1.35.0-wmf.37/extensions/Translate/TranslateEditAddons.php(287): MessageValidator->quickValidate(FatMessage, string)
#4 /srv/mediawiki/php-1.35.0-wmf.37/extensions/Translate/TranslateEditAddons.php(224): TranslateEditAddons::checkNeedsFuzzy(MessageHandle, string)
#5 /srv/mediawiki/php-1.35.0-wmf.37/includes/HookContainer/HookContainer.php(319): TranslateEditAddons::onSave(WikiPage, User, WikitextContent, string, integer, NULL, NULL, integer, Revision, Status, boolean, integer)
#6 /srv/mediawiki/php-1.35.0-wmf.37/includes/HookContainer/HookContainer.php(131): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
#7 /srv/mediawiki/php-1.35.0-wmf.37/includes/HookContainer/HookRunner.php(2773): MediaWiki\HookContainer\HookContainer->run(string, array)
#8 /srv/mediawiki/php-1.35.0-wmf.37/includes/Storage/PageUpdater.php(1322): MediaWiki\HookContainer\HookRunner->onPageContentSaveComplete(WikiPage, User, WikitextContent, string, integer, NULL, NULL, integer, Revision, Status, boolean, integer)
#9 /srv/mediawiki/php-1.35.0-wmf.37/includes/libs/rdbms/database/Database.php(4264): MediaWiki\Storage\PageUpdater->MediaWiki\Storage\{closure}(Wikimedia\Rdbms\DatabaseMysqli, string)
#10 /srv/mediawiki/php-1.35.0-wmf.37/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->doAtomicSection(string, Closure)
#11 /srv/mediawiki/php-1.35.0-wmf.37/includes/libs/rdbms/database/DBConnRef.php(641): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#12 /srv/mediawiki/php-1.35.0-wmf.37/includes/deferred/AtomicSectionUpdate.php(39): Wikimedia\Rdbms\DBConnRef->doAtomicSection(string, Closure)
#13 /srv/mediawiki/php-1.35.0-wmf.37/includes/deferred/DeferredUpdates.php(466): AtomicSectionUpdate->doUpdate()
#14 /srv/mediawiki/php-1.35.0-wmf.37/includes/deferred/DeferredUpdates.php(343): DeferredUpdates::attemptUpdate(AtomicSectionUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#15 /srv/mediawiki/php-1.35.0-wmf.37/includes/deferred/DeferredUpdates.php(278): DeferredUpdates::run(AtomicSectionUpdate, Wikimedia\Rdbms\LBFactoryMulti, Monolog\Logger, BufferingStatsdDataFactory, string)
#16 /srv/mediawiki/php-1.35.0-wmf.37/includes/deferred/DeferredUpdates.php(190): DeferredUpdates::handleUpdateQueue(array, string, integer)
#17 /srv/mediawiki/php-1.35.0-wmf.37/extensions/EventBus/includes/JobExecutor.php(105): DeferredUpdates::doUpdates()
#18 /srv/mediawiki/rpc/RunSingleJob.php(76): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#19 {main}

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Check if message definition is null before performing validation	mediawiki/extensions/Translate	master	+30 -0

Customize query in gerrit

Related Objects

Mentioned In: Blog Post: Production Excellence #22: June 2020
T254174: 1.35.0-wmf.37 deployment blockers

Event Timeline

• LarsWirzenius created this task.Jun 17 2020, 12:56 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 17 2020, 12:56 PM

• LarsWirzenius mentioned this in T254174: 1.35.0-wmf.37 deployment blockers.Jun 17 2020, 12:56 PM

• Agusbou2015 subscribed.Jun 17 2020, 1:06 PM

Nikerabbit claimed this task.Jun 17 2020, 2:55 PM

Nikerabbit updated the task description. (Show Details)

Nikerabbit edited Stack Trace. (Show Details)

Nikerabbit added a project: Language-Team (Language-2020-Focus-Sprint).

Nikerabbit moved this task from Priority Backlog to In Progress on the Language-Team (Language-2020-Focus-Sprint) board.

Hmm, initially I thought I will just add null-checks to runValidation, but page should never be saved with a null content, so null is unexpected here.

Actually it's the definition that is missing: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/Translate/+/wmf/1.35.0-wmf.37/src/MessageValidator/Validators/BraceBalanceValidator.php#33

This is also unexpected.

Change 606216 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/Translate@master] Check if message definition is null before performing validation

https://gerrit.wikimedia.org/r/606216

gerritbot added a project: Patch-For-Review.Jun 17 2020, 4:06 PM

thcipriani moved this task from Untriaged to Jun 2020 on the Wikimedia-production-error board.Jun 17 2020, 6:57 PM

Change 606216 merged by jenkins-bot:
[mediawiki/extensions/Translate@master] Check if message definition is null before performing validation

https://gerrit.wikimedia.org/r/606216

ReleaseTaggerBot added a project: MW-1.35-notes (1.35.0-wmf.38; 2020-06-23).Jun 18 2020, 11:00 AM

Maintenance_bot removed a project: Patch-For-Review.Jun 18 2020, 11:10 AM

• Rileych reassigned this task from Nikerabbit to abi_.Jun 18 2020, 1:15 PM

As Niklas said earlier, the message definition should not be null. We need to identify where this is happening so that we can properly fix it,

We've added logging to identify when the message definition is null
We're not running the validators if the message definition is null in order to avoid this error.

So while we will no longer see this error appear, we still have to identify and fix the underlying issue.

Found 5 instances of this in Logstash since it was deployed on mediawikiwiki.

Logstash query:

(channel:Translate OR channel:Translate.Jobs) AND normalized_message:"Message definition is empty! Title: {title}, group: {group}, key: {key}"

Messages:

Time (UTC)	Message	Diff
Jun 24, 2020 @ 21:33:25.213	Message definition is empty! Title: Translations:Template:Wikimedia engineering header/10/en, group: page-Template:Wikimedia engineering header, key: Template:Wikimedia_engineering_header/10	Diff
Jun 23, 2020 @ 22:07:44.754	Message definition is empty! Title: Translations:Help:Lint errors/obsolete-tag/43/en, group: page-Help:Lint errors/obsolete-tag, key: Help:Lint_errors/obsolete-tag/43	Diff
Jun 23, 2020 @ 21:33:07.525	Message definition is empty! Title: Translations:Growth/Personalized first day/Newcomer tasks/155/en, group: page-Growth/Personalized first day/Newcomer tasks, key: Growth/Personalized_first_day/Newcomer_tasks/155	Diff
Jun 23, 2020 @ 21:33:07.281	Message definition is empty! Title: Translations:Growth/Personalized first day/Newcomer tasks/154/en, group: page-Growth/Personalized first day/Newcomer tasks, key: Growth/Personalized_first_day/Newcomer_tasks/154	Diff
Jun 23, 2020 @ 21:33:07.141	Message definition is empty! Title: Translations:Growth/Personalized first day/Newcomer tasks/154/en, group: page-Growth/Personalized first day/Newcomer tasks, key: Growth/Personalized_first_day/Newcomer_tasks/154	Diff

abi_ moved this task from Recheck after deployment to In Progress on the Language-Team (Language-2020-Focus-Sprint) board.Jun 25 2020, 10:40 AM

• LarsWirzenius unsubscribed.Jul 6 2020, 7:11 AM

In the last 24 hours, this happened 82 times.

Here's another instance,

Time (UTC)	Message	Diff
Jul 7, 2020 @ 12:59:39.616	Message definition is empty! Title: Translations:Manual:$wgCookieSameSite/Page display title/en, group: page-Manual:$wgCookieSameSite, key: Manual:$wgCookieSameSite/Page_display_title	Diff
Jul 6, 2020 @ 20:19:57.083	Message definition is empty! Title: Translations:VisualEditor/Newsletter/2020/July/6/en, group: page-VisualEditor/Newsletter/2020/July, key: VisualEditor/Newsletter/2020/July/6	Diff

It appears that the definitions are missing for translation units that are new. For example,

Time: Jul 5, 2020 @ 11:28:44.887
Message: Message definition is empty! Title: Translations:Category:CirrusSearchSimilarityConfig extensions/1/en, group: page-Category:CirrusSearchSimilarityConfig extensions, key: Category:CirrusSearchSimilarityConfig_extensions/1
Timeline:

Jul 5, 2020 @ 11:28:44.404 - [Job: TranslationsUpdateJob][Request ID: dbeb9bd3-9574-49b6-8b88-07b457afdcde][Title: Category:CirrusSearchSimilarityConfig extensions] Starting TranslationsUpdateJob
Jul 5, 2020 @ 11:28:44.541 - [Job: TranslationsUpdateJob][Request ID: dbeb9bd3-9574-49b6-8b88-07b457afdcde][Title: Category:CirrusSearchSimilarityConfig extensions] Finished running 2 MessageUpdate jobs for 2 sections
Jul 5, 2020 @ 11:28:44.667 - [Job: TranslationsUpdateJob][Request ID: dbeb9bd3-9574-49b6-8b88-07b457afdcde][Title: Category:CirrusSearchSimilarityConfig extensions] Cleared caches
Jul 5, 2020 @ 11:28:44.681 - [Job: TranslationsUpdateJob][Request ID: dbeb9bd3-9574-49b6-8b88-07b457afdcde][Title: Category:CirrusSearchSimilarityConfig extensions] Updated the message group stats
Jul 5, 2020 @ 11:28:44.682 - [Job: TranslationsUpdateJob][Request ID: dbeb9bd3-9574-49b6-8b88-07b457afdcde][Title: Category:CirrusSearchSimilarityConfig extensions] Finished purging
Jul 5, 2020 @ 11:28:44.695 - [Job: TranslationsUpdateJob][Request ID: dbeb9bd3-9574-49b6-8b88-07b457afdcde][Title: Category:CirrusSearchSimilarityConfig extensions] Added 1 RenderJobs to the queue
Jul 5, 2020 @ 11:28:44.712 - [Job: TranslationsUpdateJob][Request ID: dbeb9bd3-9574-49b6-8b88-07b457afdcde][Title: Category:CirrusSearchSimilarityConfig extensions] Finished TranslationsUpdateJob
Jul 5, 2020 @ 11:28:44.724 - Message definition is empty! Title: Translations:Category:CirrusSearchSimilarityConfig extensions/Page display title/en, group: page-Category:CirrusSearchSimilarityConfig extensions, key: Category:CirrusSearchSimilarityConfig_extensions/Page_display_title
Jul 5, 2020 @ 11:28:44.737 - [Job: TranslateRenderJob][Request ID: dbeb9bd3-9574-49b6-8b88-07b457afdcde][Title: Category:CirrusSearchSimilarityConfig extensions/en] Starting TranslateRenderJob

Time: Jul 7, 2020 @ 12:59:40.273
Message: Message definition is empty! Title: Translations:Manual:$wgCookieSameSite/5/en, group: page-Manual:$wgCookieSameSite, key: Manual:$wgCookieSameSite/5
Timeline:

Jul 7, 2020 @ 12:59:38.865 - [Job: TranslationsUpdateJob][Request ID: b32c71ad-321f-46ca-899e-b3ce14402436][Title: Manual:$wgCookieSameSite] Starting TranslationsUpdateJob
Jul 7, 2020 @ 12:59:39.086 - [Job: TranslationsUpdateJob][Request ID: b32c71ad-321f-46ca-899e-b3ce14402436][Title: Manual:$wgCookieSameSite] Finished running 6 MessageUpdate jobs for 6 sections
Jul 7, 2020 @ 12:59:39.207 - [Job: TranslationsUpdateJob][Request ID: b32c71ad-321f-46ca-899e-b3ce14402436][Title: Manual:$wgCookieSameSite] Cleared caches
Jul 7, 2020 @ 12:59:39.221 - [Job: TranslationsUpdateJob][Request ID: b32c71ad-321f-46ca-899e-b3ce14402436][Title: Manual:$wgCookieSameSite] Updated the message group stats
Jul 7, 2020 @ 12:59:39.223 - [Job: TranslationsUpdateJob][Request ID: b32c71ad-321f-46ca-899e-b3ce14402436][Title: Manual:$wgCookieSameSite] Finished purging
Jul 7, 2020 @ 12:59:39.588 - [Job: TranslationsUpdateJob][Request ID: b32c71ad-321f-46ca-899e-b3ce14402436][Title: Manual:$wgCookieSameSite] Added 1 RenderJobs to the queue
Jul 7, 2020 @ 12:59:39.599 - [Job: TranslationsUpdateJob][Request ID: b32c71ad-321f-46ca-899e-b3ce14402436][Title: Manual:$wgCookieSameSite] Finished TranslationsUpdateJob
Jul 7, 2020 @ 12:59:39.616 - Message definition is empty! Title: Translations:Manual:$wgCookieSameSite/Page display title/en, group: page-Manual:$wgCookieSameSite, key: Manual:$wgCookieSameSite/Page_display_title
Jul 7, 2020 @ 12:59:39.621 - [MessageIndex] Started rebuild. Initiated by MediaWiki\Extension\EventBus\JobExecutor->execute/MessageIndexRebuildJob->run/MessageIndex->rebuild
Jul 7, 2020 @ 12:59:39.637 - [Job: TranslateRenderJob][Request ID: b32c71ad-321f-46ca-899e-b3ce14402436][Title: Manual:$wgCookieSameSite/en] Starting TranslateRenderJob
Jul 7, 2020 @ 12:59:39.672 - [Job: TranslateRenderJob][Request ID: b32c71ad-321f-46ca-899e-b3ce14402436][Title: Manual:$wgCookieSameSite/en] Cannot render translation page!
Jul 7, 2020 @ 12:59:39.737 - Message definition is empty! Title: Translations:Manual:$wgCookieSameSite/1/en, group: page-Manual:$wgCookieSameSite, key: Manual:$wgCookieSameSite/1
Jul 7, 2020 @ 12:59:39.778 - [MessageIndex] Got lock in 0.00063514709472656
Jul 7, 2020 @ 12:59:39.879 - Message definition is empty! Title: Translations:Manual:$wgCookieSameSite/2/en, group: page-Manual:$wgCookieSameSite, key: Manual:$wgCookieSameSite/2
Jul 7, 2020 @ 12:59:40.021 - Message definition is empty! Title: Translations:Manual:$wgCookieSameSite/3/en, group: page-Manual:$wgCookieSameSite, key: Manual:$wgCookieSameSite/3
Jul 7, 2020 @ 12:59:40.145 - Message definition is empty! Title: Translations:Manual:$wgCookieSameSite/4/en, group: page-Manual:$wgCookieSameSite, key: Manual:$wgCookieSameSite/4
Jul 7, 2020 @ 12:59:40.273 - Message definition is empty! Title: Translations:Manual:$wgCookieSameSite/5/en, group: page-Manual:$wgCookieSameSite, key: Manual:$wgCookieSameSite/5

Krinkle mentioned this in Blog Post: Production Excellence #22: June 2020.Jul 23 2020, 3:25 AM

Nikerabbit updated the task description. (Show Details)Jul 23 2020, 6:48 AM

Pginer-WMF edited projects, added Language-Team (Language-2020-July-September); removed Language-Team (Language-2020-Focus-Sprint).Aug 3 2020, 9:32 AM

Pginer-WMF moved this task from Quarter Backlog to In Progress on the Language-Team (Language-2020-July-September) board.

This is still happening, need to investigate this further but this is not a high priority issue. Moving this to the quarterly backlog.

I can no longer find the relevant fatal error in the logs with Logstash query exception.message:"Argument 1 passed to" AND Translate over the past 30 days.

I also don't see any matches for channel:"Translate" AND message:"Message definition is".