| Tarrow | |
| Feb 22 2022, 7:08 PM |
| F34962209: image.png | |
| Feb 22 2022, 7:27 PM |
I saw:
[1] Firing Labels alertname = Max Lag above 10 for 1 hour metric = wikibase-queryservice runbook = https://wikitech.wikimedia.org/wiki/WMDE/Wikidata/Alerts severity = critical team = wikidata Annotations description = Maxlag is above ten seconds for more than an hour summary = Max Lag above 10 for 1 hour
around 18:40 UTC
Looks like due to an ongoing incident affecting the codfw queryservice servers the lag on them is high. Due to T238751 being unresolved this causes high maxlag.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Gehel | T301147 The WDQS streaming updater went unstable for several hours (2022-02-06T23:00:00 - 2022-02-07T06:20:00) | |||
| Resolved | Gehel | T302330 Wikidata MaxLag above 10 for 1hr |
Looks like altering this to not read from codfw prometheus might be sensible https://gerrit.wikimedia.org/g/operations/puppet/+/b81506884faa046e4103412f1928ece934a157dc/modules/profile/manifests/mediawiki/maintenance/wikidata.pp#15
Change 764875 had a related patch set uploaded (by Addshore; author: Addshore):
[operations/puppet@production] Temp remove codfw
Change 764875 merged by Ryan Kemper:
[operations/puppet@production] Temp remove codfw from wikidata updateQueryServiceLag check
Mentioned in SAL (#wikimedia-operations) [2022-02-22T19:25:32Z] <ryankemper> T302330 ryankemper@cumin1001:~$ sudo -E cumin '*mwmaint*' 'run-puppet-agent' (getting https://gerrit.wikimedia.org/r/c/operations/puppet/+/764875 out)
We might want to tweak the script so that normal (non-puppet) deployers can fix these things (e.g. they read from mw-config or something)
Change 764830 had a related patch set uploaded (by Addshore; author: Addshore):
[operations/puppet@production] Revert \"Temp remove codfw from wikidata updateQueryServiceLag check\"
Change 764830 merged by Bking:
[operations/puppet@production] Revert "Temp remove codfw from wikidata updateQueryServiceLag check"
