LoadBalancer.php
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

Authored By

	jnuche
	Thu, Apr 4, 11:45 AM

Description

Error

service.version: 1.42.0-wmf.25
trace.id: b346a0e8-4737-4458-bc27-d6b7390ed2a9
Find trace.id in Logstash

labels.normalized_message

[{reqId}] {exception_url}   PHP Notice: Undefined offset: 10

error.stack_trace

from /srv/mediawiki/php-1.42.0-wmf.25/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1849)
#0 /srv/mediawiki/php-1.42.0-wmf.25/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1849): MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/mediawiki/php-1.42.0-wmf.25/includes/api/ApiMain.php(1571): Wikimedia\Rdbms\LoadBalancer->getMaxLag()
#2 /srv/mediawiki/php-1.42.0-wmf.25/includes/api/ApiMain.php(1608): ApiMain->getMaxLag()
#3 /srv/mediawiki/php-1.42.0-wmf.25/includes/api/ApiMain.php(1934): ApiMain->checkMaxLag(Wikibase\Repo\Api\GetEntities, array)
#4 /srv/mediawiki/php-1.42.0-wmf.25/includes/api/ApiMain.php(922): ApiMain->executeAction()
#5 /srv/mediawiki/php-1.42.0-wmf.25/includes/api/ApiMain.php(893): ApiMain->executeActionWithErrorHandling()
#6 /srv/mediawiki/php-1.42.0-wmf.25/includes/api/ApiEntryPoint.php(158): ApiMain->execute()
#7 /srv/mediawiki/php-1.42.0-wmf.25/includes/MediaWikiEntryPoint.php(199): MediaWiki\Api\ApiEntryPoint->execute()
#8 /srv/mediawiki/php-1.42.0-wmf.25/api.php(44): MediaWiki\MediaWikiEntryPoint->run()
#9 /srv/mediawiki/w/api.php(3): require(string)
#10 {main}

Notes

Error started around 05:00 UTC on 2024-04-03 (Wednesday) with wmf.25 at group0 and has affected both wmf.24 and wmf.25. Change https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1004792 modified function getLagTimes which returns the indices that are currently causing the out of range error. However, the change was merged on 2024-03-22 and had already gone out in the wmf.24 train ago. All of this makes me think some other factor started triggering the error on Wednesday.

The issue affects mainly wikidate and happens in bursts, which points at some kind of data dump/snapshot.

Details

Request URL: https://www.wikidata.org/w/api.php?action=wbgetentities&format=*&maxlag=*&sites=*&titles=*

	Subject	Repo	Branch	Lines +/-
	rdbms: Protect against stale cache in LB::getMaxLag()	mediawiki/core	wmf/1.43.0-wmf.2	+3 -1
	rdbms: Protect against stale cache in LB::getMaxLag()	mediawiki/core	master	+3 -1

Customize query in gerrit

Event Timeline

jnuche created this task.Thu, Apr 4, 11:45 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptThu, Apr 4, 11:45 AM

jnuche updated the task description. (Show Details)Thu, Apr 4, 11:46 AM

The exception has actually been happening with different offset indices for at least a week, e.g. on the 2024-03-28 (Thursday) last week during the wmf.24 train. This aligns better with the deployment of https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1004792

jnuche renamed this task from PHP Notice: Undefined offset: 10 in rdbms/loadbalancer/LoadBalancer.php to PHP Notice: Undefined offset in rdbms/loadbalancer/LoadBalancer.php.Thu, Apr 4, 12:29 PM

It's combination of that patch, cache being stalled and depooling replicas for maint. It should be a rather small number from what I'm seeing (3K in a day). I can take care of it once back or some easy patch can be made.

I'm on train duty this week so I'm pinging on this ticket. LoadBalancer:1849 PHP Notice: Undefined offset: 8 is the top error on in logspam-watch output.

Change #1020924 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] rdbms: Protect against stale cache in LB::getMaxLag()

https://gerrit.wikimedia.org/r/1020924

gerritbot added a project: Patch-For-Review.Wed, Apr 17, 9:41 PM

Ladsgroup claimed this task.Wed, Apr 17, 9:41 PM

Ladsgroup triaged this task as Medium priority.

Ladsgroup added a project: DBA.

Ladsgroup moved this task from Triage to In progress on the DBA board.

thcipriani moved this task from Untriaged to Apr 2024 on the Wikimedia-production-error board.Thu, Apr 18, 3:23 PM

Change #1020924 merged by jenkins-bot:

[mediawiki/core@master] rdbms: Protect against stale cache in LB::getMaxLag()

https://gerrit.wikimedia.org/r/1020924

ReleaseTaggerBot added a project: MW-1.43-notes (1.43.0-wmf.3; 2024-04-30).Tue, Apr 23, 4:00 PM

Maintenance_bot removed a project: Patch-For-Review.Fri, Apr 26, 6:48 PM

Change #1025178 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@wmf/1.43.0-wmf.2] rdbms: Protect against stale cache in LB::getMaxLag()

https://gerrit.wikimedia.org/r/1025178

gerritbot added a project: Patch-For-Review.Mon, Apr 29, 8:57 AM

Change #1025178 merged by jenkins-bot:

[mediawiki/core@wmf/1.43.0-wmf.2] rdbms: Protect against stale cache in LB::getMaxLag()

https://gerrit.wikimedia.org/r/1025178

Mentioned in SAL (#wikimedia-operations) [2024-04-29T09:18:13Z] <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:1025178|rdbms: Protect against stale cache in LB::getMaxLag() (T361824)]]

Mentioned in SAL (#wikimedia-operations) [2024-04-29T09:20:44Z] <ladsgroup@deploy1002> ladsgroup: Backport for [[gerrit:1025178|rdbms: Protect against stale cache in LB::getMaxLag() (T361824)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Maintenance_bot removed a project: Patch-For-Review.Mon, Apr 29, 9:32 AM

Mentioned in SAL (#wikimedia-operations) [2024-04-29T09:38:28Z] <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:1025178|rdbms: Protect against stale cache in LB::getMaxLag() (T361824)]] (duration: 20m 15s)

Ladsgroup closed this task as Resolved.Mon, Apr 29, 9:39 AM

Ladsgroup moved this task from In progress to Done on the DBA board.

ReleaseTaggerBot edited projects, added MW-1.43-notes (1.43.0-wmf.2; 2024-04-23); removed MW-1.43-notes (1.43.0-wmf.3; 2024-04-30).Mon, Apr 29, 10:00 AM

PHP Notice: Undefined offset in rdbms/loadbalancer/LoadBalancer.phpClosed, ResolvedPublicPRODUCTION ERRORActions