What happens?:
As with T279809 in the past, the outage was too short for me to run more tests.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Reedy | T266419 Composer 2.0 updates and support | |||
| Resolved | Reedy | T266421 Rebuild mediawiki-vendor on composer 2.0 | |||
| Resolved | Reedy | T275824 Bump wikimedia/composer-merge-plugin in vendor | |||
| Resolved | tstarling | T248908 composer-merge-plugin support for composer 2.0 | |||
| Restricted Task | |||||
| Resolved | Jdforrester-WMF | T279857 Re-build CI containers with Composer 2.x | |||
| Resolved | Jdforrester-WMF | T281294 Drop CI for REL1_31 branch once it's EOL | |||
| Resolved | Aklapper | T230582 Archive Oracle Database project | |||
| Resolved | Aklapper | T230583 Archive MSSQL project | |||
| Resolved | Reedy | T279858 Formally EOL REL1_31 | |||
| Resolved | Reedy | T285404 Release MediaWiki 1.31.16/1.35.4/1.36.2 | |||
| Resolved | Reedy | T285405 Tracking bug for MediaWiki 1.31.16/1.35.4/1.36.2 | |||
| Resolved | BUG REPORT | Ladsgroup | T290379 2021-09-04 enwiki was down at 10:44 (UTC) | ||
| Restricted Task |
There indeed was an alert for text request volume that matches your timing:
10:45:57 <jinxer-wm> (VarnishTrafficDrop) firing: 62% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org 10:50:57 <jinxer-wm> (VarnishTrafficDrop) resolved: 67% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org
https://grafana.wikimedia.org/d/000000278/mysql-aggregated?orgId=1&from=1630736814684&to=1630758414684&var-site=codfw&var-group=core&var-shard=s1&var-role=All shows a drop in enwiki queries and an increase in max response time after that (although it recovered suspiciously quickly)
I'm on my phone but just to mention that the queries dropping is probably a consequence of something else and not the consequence. There's a huge spike and then the drop, which makes me thing we had something else failing in between.
From what I can see all the API enwiki hosts got this query around the reported time which is pretty crazy (hiding the query for obvious reasons): {P17211}
Pasting the explain as that is fine - we can see it is doing a full table scan:
+------+-------------+-----------+------+----------------------+---------+---------+---------------------+----------+----------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-----------+------+----------------------+---------+---------+---------------------+----------+----------------------------------------------+ | 1 | SIMPLE | page | ALL | PRIMARY | NULL | NULL | NULL | 52516309 | Using where; Using temporary; Using filesort | | 1 | SIMPLE | pagelinks | ref | PRIMARY,pl_namespace | PRIMARY | 4 | enwiki.page.page_id | 28 | Using where; Using index | +------+-------------+-----------+------+----------------------+---------+---------+---------------------+----------+----------------------------------------------+ 2 rows in set (0.037 sec)
That caused a massive spike on all of them and maybe got them unavailable : https://grafana.wikimedia.org/d/000000273/mysql?viewPanel=3&orgId=1&from=1630744072686&to=1630753109255&var-server=db2092&var-port=9104
The private task T290394 has been resolved after the patch to mitigate this was pushed. Closing this task too.
Change 725056 had a related patch set uploaded (by Reedy; author: Amir Sarabadani):
[mediawiki/core@REL1_31] SECURITY: Add straight join to ApiQueryBacklinks
Change 725056 merged by jenkins-bot:
[mediawiki/core@REL1_31] SECURITY: Add straight join to ApiQueryBacklinks
Change 725062 had a related patch set uploaded (by Reedy; author: Amir Sarabadani):
[mediawiki/core@REL1_35] SECURITY: Add straight join to ApiQueryBacklinks
Change 725062 merged by jenkins-bot:
[mediawiki/core@REL1_35] SECURITY: Add straight join to ApiQueryBacklinks
Change 725067 had a related patch set uploaded (by Reedy; author: Amir Sarabadani):
[mediawiki/core@REL1_36] SECURITY: Add straight join to ApiQueryBacklinks
Change 725072 had a related patch set uploaded (by Reedy; author: Amir Sarabadani):
[mediawiki/core@REL1_37] SECURITY: Add straight join to ApiQueryBacklinks
Change 725075 had a related patch set uploaded (by Reedy; author: Amir Sarabadani):
[mediawiki/core@master] SECURITY: Add straight join to ApiQueryBacklinks
Change 725067 merged by jenkins-bot:
[mediawiki/core@REL1_36] SECURITY: Add straight join to ApiQueryBacklinks
Change 725072 merged by jenkins-bot:
[mediawiki/core@REL1_37] SECURITY: Add straight join to ApiQueryBacklinks
Change 725075 merged by jenkins-bot:
[mediawiki/core@master] SECURITY: Add straight join to ApiQueryBacklinks