Maniphest T252077

WikimediaPrometheusQueryServiceLagProvider: PHP Warning: A non-numeric value encountered
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error

MediaWiki version: 1.35.0-wmf.31

message

PHP Warning: A non-numeric value encountered

Impact

Unclear.

Notes

40+ of these since 1.35.0-wmf.31.

Details

Request ID: 7498d5ef1f8de0a661f1c74f
Request URL: n/a

Stack Trace

exception.trace

#0 /srv/mediawiki/php-1.35.0-wmf.31/extensions/Wikidata.org/src/QueryServiceLag/WikimediaPrometheusQueryServiceLagProvider.php(110): MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/mediawiki/php-1.35.0-wmf.31/extensions/Wikidata.org/src/QueryServiceLag/WikimediaPrometheusQueryServiceLagProvider.php(59): WikidataOrg\QueryServiceLag\WikimediaPrometheusQueryServiceLagProvider->getLags()
#2 /srv/mediawiki/php-1.35.0-wmf.31/extensions/Wikidata.org/maintenance/updateQueryServiceLag.php(84): WikidataOrg\QueryServiceLag\WikimediaPrometheusQueryServiceLagProvider->getLag()
#3 /srv/mediawiki/php-1.35.0-wmf.31/maintenance/doMaintenance.php(105): WikidataOrg\UpdateQueryServiceLag->execute()
#4 /srv/mediawiki/php-1.35.0-wmf.31/extensions/Wikidata.org/maintenance/updateQueryServiceLag.php(107): require_once(string)
#5 /srv/mediawiki/multiversion/MWScript.php(101): require_once(string)
#6 {main}

	Subject	Repo	Branch	Lines +/-
	WikimediaPrometheusQueryServiceLagProvider Nan values	mediawiki/extensions/Wikidata.org	master	+28 -6

Customize query in gerrit

Related Objects

Mentioned In: T321710: WikimediaPrometheusQueryServiceLagProvider: PHP Warning: A non-numeric value encountered
T263670: Prometheus API error when requesting Blazegraph lag for WDQS
T252739: Define a hard upper limit of query service based maxlag
T253098: "PHP Warning: A non-numeric value encountered" from NamespaceInfo.php via SpecialRecentChanges.php - NamespaceInfo::isTalk called with non-integer (string) namespace '-1'
T249963: 1.35.0-wmf.31 deployment blockers
Mentioned Here: P11165 (An Untitled Masterwork)
rEWOR5e3b0d5fb662: Switch query service maxlag to be median +1

Event Timeline

brennen created this task.May 6 2020, 9:52 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 6 2020, 9:52 PM

brennen triaged this task as Unbreak Now! priority.May 6 2020, 10:03 PM

Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptMay 6 2020, 10:03 PM

brennen added a parent task: T249963: 1.35.0-wmf.31 deployment blockers.May 6 2020, 10:04 PM

Just realizing this is due to /srv/mediawiki-staging/multiversion/MWScript.php extensions/Wikidata.org/maintenance/updateQueryServiceLag.php --wiki wikidatawiki --cluster wdqs --prometheus prometheus.svc.eqiad.wmnet --prometheus prometheus.svc.codfw.wmnet.

5e3b0d5f?

Presumably this is cron'd? Unclear whether this warrants rollback.

brennen moved this task from Backlog to Logs/Train on the User-brennen board.May 6 2020, 10:17 PM

Reedy added a subscriber: Wikidata-Campsite.May 6 2020, 10:31 PM

After discussion in #wikimedia-operations, unblocking train on the assumption that this is likely to be bad data from Prometheus.

16:25 <brennen> meanwhile:  does T252077 warrant a rollback?
16:26 <Reedy> https://github.com/wikimedia/puppet/blob/6b0dc71f153b6f052eb117c72ed365aaedc12a4d/modules/profile/manifests/mediawiki/maintenance/wikidata.pp#L73
16:26 <Reedy> (it is a cron, yeah)
16:27  * Reedy looks
16:27 <brennen> thx.
16:30 <Reedy> brennen: I'm presuming time() isn't broken in PHP...
16:30 <brennen> well one can hope.
16:30 <Reedy> :)
16:30 <Reedy> So I'm guessing it's bad data from the prometheus service
16:30 <Reedy> No recent changes to the code
16:31 <brennen> yeah, makes sense.  in that case i'll unblock.

brennen removed a parent task: T249963: 1.35.0-wmf.31 deployment blockers.May 6 2020, 10:34 PM

I can confirm no rollback or blocking is needed

Addshore removed a subscriber: Wikidata-Campsite.May 6 2020, 11:12 PM

Looking at P11165 prometheus seems to be returning a NaN value, which is no accounted for in the code.

	      {
        "metric": {
          "__name__": "blazegraph_lastupdated",
          "cluster": "wdqs",
          "instance": "wdqs2001:9193",
          "job": "blazegraph",
          "site": "codfw"
        },
        "value": [
          1588807140.848,
          "NaN"
        ]
      },

Looks like that host is doing something else now.

I think the below is related.

21:07 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer
21:05 ryankemper@cumin1001: END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99)
21:04 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer

I guess the code needs to account for NaN values and count them as not in the pool of servers to look at.

brennen mentioned this in T249963: 1.35.0-wmf.31 deployment blockers.May 6 2020, 11:34 PM

Maintenance_bot moved this task from incoming to in progress on the Wikidata board.May 7 2020, 12:15 AM

WMDE-leszek subscribed.May 7 2020, 5:26 AM

Addshore triaged this task as Medium priority.May 11 2020, 9:21 AM

Gehel moved this task from Incoming to Watching / Waiting on the Wikidata-Query-Service board.May 13 2020, 12:36 PM

Krinkle moved this task from Untriaged to May 2020 on the Wikimedia-production-error board.May 13 2020, 6:10 PM

KartikMistry mentioned this in T253098: "PHP Warning: A non-numeric value encountered" from NamespaceInfo.php via SpecialRecentChanges.php - NamespaceInfo::isTalk called with non-integer (string) namespace '-1'.May 19 2020, 8:05 AM

Addshore claimed this task.May 20 2020, 8:19 AM

Addshore moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

Restricted Application added a project: User-Addshore. · View Herald TranscriptMay 20 2020, 8:19 AM

Change 597474 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikidata.org@master] WikimediaPrometheusQueryServiceLagProvider Nan values

https://gerrit.wikimedia.org/r/597474

gerritbot added a project: Patch-For-Review.May 20 2020, 8:39 AM

Addshore moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.May 20 2020, 8:40 AM

Addshore mentioned this in T252739: Define a hard upper limit of query service based maxlag.May 20 2020, 6:28 PM

hoo moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.May 21 2020, 3:17 AM

Change 597474 merged by jenkins-bot:
[mediawiki/extensions/Wikidata.org@master] WikimediaPrometheusQueryServiceLagProvider Nan values

https://gerrit.wikimedia.org/r/597474

ReleaseTaggerBot added a project: MW-1.35-notes (1.35.0-wmf.34; 2020-05-26).May 21 2020, 4:00 AM