PHP Warning: Stats: Cannot associate label keys with label values: Not all initialized labels have an assigned value.
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error

service.version: 1.43.0-wmf.19
trace.id: 25ef2295-bdff-4eb9-ab59-adc41883295f
Find trace.id in Logstash

labels.normalized_message

[{reqId}] {exception_url}   PHP Warning: Stats: Cannot associate label keys with label values: Not all initialized labels have an assigned value.

error.stack_trace

from /srv/mediawiki/php-1.43.0-wmf.19/includes/libs/Stats/Metrics/CounterMetric.php(84)
#0 [internal function]: MWExceptionHandler::handleError(int, string, string, int, array)
#1 /srv/mediawiki/php-1.43.0-wmf.19/includes/libs/Stats/Metrics/CounterMetric.php(84): trigger_error(string, int)
#2 /srv/mediawiki/php-1.43.0-wmf.19/extensions/EventBus/includes/EventBus.php(221): Wikimedia\Stats\Metrics\CounterMetric->incrementBy(float)
#3 /srv/mediawiki/php-1.43.0-wmf.19/extensions/EventBus/includes/EventBus.php(507): MediaWiki\Extension\EventBus\EventBus->incrementMetricByValue(string, int, array, array)
#4 /srv/mediawiki/php-1.43.0-wmf.19/extensions/EventBus/includes/Adapters/RCFeed/EventBusRCFeedEngine.php(57): MediaWiki\Extension\EventBus\EventBus->send(array)
#5 /srv/mediawiki/php-1.43.0-wmf.19/includes/deferred/MWCallableUpdate.php(52): MediaWiki\Extension\EventBus\Adapters\RCFeed\EventBusRCFeedEngine::MediaWiki\Extension\EventBus\Adapters\RCFeed\{closure}(string)
#6 /srv/mediawiki/php-1.43.0-wmf.19/includes/deferred/DeferredUpdates.php(495): MediaWiki\Deferred\MWCallableUpdate->doUpdate()
#7 /srv/mediawiki/php-1.43.0-wmf.19/includes/deferred/DeferredUpdates.php(207): MediaWiki\Deferred\DeferredUpdates::attemptUpdate(MediaWiki\Deferred\MWCallableUpdate)
#8 /srv/mediawiki/php-1.43.0-wmf.19/includes/deferred/DeferredUpdates.php(305): MediaWiki\Deferred\DeferredUpdates::run(MediaWiki\Deferred\MWCallableUpdate)
#9 /srv/mediawiki/php-1.43.0-wmf.19/includes/deferred/DeferredUpdatesScope.php(269): MediaWiki\Deferred\DeferredUpdates::MediaWiki\Deferred\{closure}(MediaWiki\Deferred\MWCallableUpdate, int)
#10 /srv/mediawiki/php-1.43.0-wmf.19/includes/deferred/DeferredUpdatesScope.php(198): MediaWiki\Deferred\DeferredUpdatesScope->processStageQueue(int, int, Closure)
#11 /srv/mediawiki/php-1.43.0-wmf.19/includes/deferred/DeferredUpdates.php(308): MediaWiki\Deferred\DeferredUpdatesScope->processUpdates(int, Closure)
#12 /srv/mediawiki/php-1.43.0-wmf.19/includes/deferred/DeferredUpdatesScope.php(269): MediaWiki\Deferred\DeferredUpdates::MediaWiki\Deferred\{closure}(MediaWiki\Deferred\MWCallableUpdate, int)
#13 /srv/mediawiki/php-1.43.0-wmf.19/includes/deferred/DeferredUpdatesScope.php(198): MediaWiki\Deferred\DeferredUpdatesScope->processStageQueue(int, int, Closure)
#14 /srv/mediawiki/php-1.43.0-wmf.19/includes/deferred/DeferredUpdates.php(313): MediaWiki\Deferred\DeferredUpdatesScope->processUpdates(int, Closure)
#15 /srv/mediawiki/php-1.43.0-wmf.19/includes/MediaWikiEntryPoint.php(674): MediaWiki\Deferred\DeferredUpdates::doUpdates()
#16 /srv/mediawiki/php-1.43.0-wmf.19/includes/MediaWikiEntryPoint.php(496): MediaWiki\MediaWikiEntryPoint->restInPeace()
#17 /srv/mediawiki/php-1.43.0-wmf.19/includes/MediaWikiEntryPoint.php(454): MediaWiki\MediaWikiEntryPoint->doPostOutputShutdown()
#18 /srv/mediawiki/php-1.43.0-wmf.19/includes/MediaWikiEntryPoint.php(209): MediaWiki\MediaWikiEntryPoint->postOutputShutdown()
#19 /srv/mediawiki/php-1.43.0-wmf.19/index.php(58): MediaWiki\MediaWikiEntryPoint->run()
#20 /srv/mediawiki/w/index.php(3): require(string)
#21 {main}

Impact

Low; 11 occurrences, first one on Aug 2, 2024 @ 21:27:01.017 (with bogus values), first one with non-bogus values on hewiki being on 1.43.0-wmf.19 on Aug 21, 2024 @ 15:51:48.425.

Notes

Details

MediaWiki Version: 1.43.0-wmf.16
Request URL: https://it.wikipedia.org/wiki/Speciale:Blocca/*?gbdisableUTEdit=*&gbexpiry=*&gbnamespaces=*&gbpages=*&gbpartial=*&gbreason=*

Subject	Repo	Branch	Lines +/-
Checker: ensure all labels are set	mediawiki/extensions/CirrusSearch	wmf/1.43.0-wmf.21	+10 -6
Checker: ensure all labels are set	mediawiki/extensions/CirrusSearch	master	+10 -6
metrics: provide a fallback value for emtpy labels	mediawiki/extensions/EventBus	master	+14 -6

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved	Release	dancy	T373640 1.43.0-wmf.21 deployment blockers
		Resolved	PRODUCTION ERROR	gmodena	T373086 PHP Warning: Stats: Cannot associate label keys with label values: Not all initialized labels have an assigned value.

Event Timeline

Aklapper created this task.Aug 22 2024, 10:59 AM

colewhite edited projects, added Event-Platform; removed MediaWiki-libs-Stats.Aug 27 2024, 11:02 PM

Restricted Application added a project: Data-Engineering. · View Herald TranscriptAug 27 2024, 11:02 PM

Introduced in https://gerrit.wikimedia.org/r/c/mediawiki/extensions/EventBus/+/1049831

@gmodena @Ottomata

gmodena claimed this task.Aug 28 2024, 8:49 AM

gmodena moved this task from Incoming (new tickets) to Q1 2024 July 1st - September 30th on the Data-Engineering board.Aug 28 2024, 8:52 AM

gmodena edited projects, added Data-Engineering (Q1 2024 July 1st - September 30th); removed Data-Engineering.

gmodena moved this task from Next Up to In progress on the Data-Engineering (Q1 2024 July 1st - September 30th) board.

@colewhite ack. Thanks for the heads up.

Change #1067997 had a related patch set uploaded (by Gmodena; author: Gmodena):

[mediawiki/extensions/EventBus@master] metrics: provide a fallback value for emtpy labels

https://gerrit.wikimedia.org/r/1067997

gerritbot added a project: Patch-For-Review.Aug 28 2024, 1:26 PM

Change #1067997 merged by jenkins-bot:

[mediawiki/extensions/EventBus@master] metrics: provide a fallback value for emtpy labels

https://gerrit.wikimedia.org/r/1067997

gmodena moved this task from In progress to Done on the Data-Engineering (Q1 2024 July 1st - September 30th) board.Aug 28 2024, 2:34 PM

ReleaseTaggerBot added a project: MW-1.43-notes (1.43.0-wmf.21; 2024-09-03).Aug 28 2024, 3:00 PM

thcipriani moved this task from Untriaged to Aug 2024 on the Wikimedia-production-error board.Aug 29 2024, 3:26 PM

https://gerrit.wikimedia.org/r/1067997 was included in this week's train branch (wmf/1.43.0-wmf.21) but I still see many Stats: Cannot associate label keys with label values: Not all initialized labels have an assigned value and Stats: Cannot associate label keys with label values: Not all initialized labels have an assigned value warnings being logged for testwikidatawiki or test.wikidata.org today.

Coming from CirrusSearch extension:

from /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/Stats/Metrics/CounterMetric.php(84)
#0 [internal function]: MWExceptionHandler::handleError(int, string, string, int, array)
#1 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/Stats/Metrics/CounterMetric.php(84): trigger_error(string, int)
#2 /srv/mediawiki/php-1.43.0-wmf.21/extensions/CirrusSearch/includes/Sanity/Checker.php(184): Wikimedia\Stats\Metrics\CounterMetric->incrementBy(float)
#3 /srv/mediawiki/php-1.43.0-wmf.21/extensions/CirrusSearch/includes/Api/CheckSanity.php(104): CirrusSearch\Sanity\Checker->check(array)
#4 /srv/mediawiki/php-1.43.0-wmf.21/extensions/CirrusSearch/includes/Api/CheckSanity.php(60): CirrusSearch\Api\CheckSanity->check(CirrusSearch\Sanity\Checker, int, int)
#5 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiMain.php(1948): CirrusSearch\Api\CheckSanity->execute()
#6 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiMain.php(926): ApiMain->executeAction()
#7 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiMain.php(897): ApiMain->executeActionWithErrorHandling()
#8 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiEntryPoint.php(153): ApiMain->execute()
#9 /srv/mediawiki/php-1.43.0-wmf.21/includes/MediaWikiEntryPoint.php(200): MediaWiki\Api\ApiEntryPoint->execute()
#10 /srv/mediawiki/php-1.43.0-wmf.21/api.php(44): MediaWiki\MediaWikiEntryPoint->run()
#11 /srv/mediawiki/w/api.php(3): require(string)
#12 {main}

Restricted Application added a project: Discovery-Search. · View Herald TranscriptSep 3 2024, 3:25 PM

Maintenance_bot removed a project: Patch-For-Review.Sep 3 2024, 3:30 PM

Change #1070298 had a related patch set uploaded (by DCausse; author: DCausse):

[mediawiki/extensions/CirrusSearch@master] Checker: ensure all labels are set

https://gerrit.wikimedia.org/r/1070298

gerritbot added a project: Patch-For-Review.Sep 3 2024, 4:22 PM

dancy triaged this task as Unbreak Now! priority.Sep 3 2024, 6:15 PM

dancy added a parent task: T373640: 1.43.0-wmf.21 deployment blockers.

The warning rate ramped up a lot (~860/15 minutes) after rolling 1.43.0-wmf.21 to group0 today, so I'm making this task a train blocker.

Change #1070298 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Checker: ensure all labels are set

https://gerrit.wikimedia.org/r/1070298

Maintenance_bot removed a project: Patch-For-Review.Sep 3 2024, 8:30 PM

ReleaseTaggerBot edited projects, added MW-1.43-notes (1.43.0-wmf.22; 2024-09-10); removed MW-1.43-notes (1.43.0-wmf.21; 2024-09-03).Sep 3 2024, 9:00 PM

Change #1070531 had a related patch set uploaded (by DCausse; author: DCausse):

[mediawiki/extensions/CirrusSearch@wmf/1.43.0-wmf.21] Checker: ensure all labels are set

https://gerrit.wikimedia.org/r/1070531

gerritbot added a project: Patch-For-Review.Sep 4 2024, 7:46 AM

@dancy @dcausse

Ack. Investigating.
My original patch only addressed EventBus. I can't seem to find any new warn entry for that service in logstash.

Looks like the spike is only related to CirrusSearch?

In T373086#10116237, @gmodena wrote:

Looks like the spike is only related to CirrusSearch?

yes this is my understanding as well, it is quite noisy so it might hide others, I'm shipping a fix so hopefully we'll get some more clarity soon.

dcausse claimed this task.Sep 4 2024, 8:39 AM

Change #1070531 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@wmf/1.43.0-wmf.21] Checker: ensure all labels are set

https://gerrit.wikimedia.org/r/1070531

Mentioned in SAL (#wikimedia-operations) [2024-09-04T08:52:40Z] <dcausse@deploy1003> Started scap sync-world: Backport for [[gerrit:1070531|Checker: ensure all labels are set (T373086)]]

Mentioned in SAL (#wikimedia-operations) [2024-09-04T08:54:52Z] <dcausse@deploy1003> dcausse: Backport for [[gerrit:1070531|Checker: ensure all labels are set (T373086)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-09-04T08:59:29Z] <dcausse@deploy1003> Finished scap sync-world: Backport for [[gerrit:1070531|Checker: ensure all labels are set (T373086)]] (duration: 06m 48s)

ReleaseTaggerBot edited projects, added MW-1.43-notes (1.43.0-wmf.21; 2024-09-03); removed MW-1.43-notes (1.43.0-wmf.22; 2024-09-10).Sep 4 2024, 9:00 AM

Seems to be fixed, tentatively closing

Maintenance_bot removed a project: Patch-For-Review.Sep 4 2024, 9:30 AM

In T373086#10116254, @dcausse wrote:

In T373086#10116237, @gmodena wrote:

Looks like the spike is only related to CirrusSearch?

yes this is my understanding as well, it is quite noisy so it might hide others, I'm shipping a fix so hopefully we'll get some more clarity soon.

Just an ack that I don't see any EventBus (or other extensions) entry in logstash.

colewhite mentioned this in T377597: Provide a way to specify metric labels at metric initialization time..Oct 18 2024, 3:18 PM

PHP Warning: Stats: Cannot associate label keys with label values: Not all initialized labels have an assigned value.Closed, ResolvedPublicPRODUCTION ERRORActions