The following hosts are running 10.4.13, we've seen a bug that made mariadb crash (T278408), although index creations are done under controlled circumstances in MW hosts, let's upgrade them to a newer version.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| db-eqiad.php: Disable writes on es4 | operations/mediawiki-config | master | +1 -1 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | jcrespo | T278408 db2135 crashed | |||
| Resolved | • Marostegui | T279281 Upgrade 10.4.13 hosts to a higher version | |||
| Resolved | • Marostegui | T279625 Upgrade mysql on db1132 (phabricator db master) | |||
| Resolved | • Marostegui | T276448 Failover m1 master: db1080 -> db1159 Wed 14th April at 10 AM UTC | |||
| Resolved | Request | • Cmjohnson | T280121 decommission db1080.eqiad.mnet | ||
| Resolved | • Marostegui | T279657 Upgrade mysql on db1128 (m5 db master) | |||
| Resolved | • Marostegui | T280251 Upgrade mysql on db1107 (m2 db master) | |||
| Resolved | • Marostegui | T281212 Restart x1 database master (db1103) | |||
| Resolved | Trizek-WMF | T281375 Read only time for extension 1 (x1) primary database on 2021-05-05 |
Mentioned in SAL (#wikimedia-operations) [2021-04-08T06:56:27Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es1023 to upgrade kernel and mysql, remove weight from es1021, to leave it as it was yesterday T279281', diff saved to https://phabricator.wikimedia.org/P15231 and previous config saved to /var/cache/conftool/dbconfig/20210408-065627-marostegui.json
Mentioned in SAL (#wikimedia-operations) [2021-04-14T12:03:00Z] <marostegui> Upgrade mysql on db1080 T279281
Mentioned in SAL (#wikimedia-operations) [2021-04-14T12:07:25Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es1025 for kernel and mysql upgrade T279281', diff saved to https://phabricator.wikimedia.org/P15333 and previous config saved to /var/cache/conftool/dbconfig/20210414-120724-marostegui.json
@razzi can you take care of these two (kernel + mariadb upgrade) ? It can be done before refreshing the replica. The things to do are:
@razzi ping :) Can you update this task with a timeline about when the job will be done?
Change 682358 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Disable writes on es4
Change 682358 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Disable writes on es4
Mentioned in SAL (#wikimedia-operations) [2021-04-26T11:55:10Z] <marostegui@deploy1002> Synchronized wmf-config/db-eqiad.php: Disable writes on es4 T279281 (duration: 00m 56s)
Mentioned in SAL (#wikimedia-operations) [2021-04-26T11:56:34Z] <marostegui> Restart es4 primary master - T279281
Mentioned in SAL (#wikimedia-operations) [2021-04-26T12:00:44Z] <marostegui@deploy1002> Synchronized wmf-config/db-eqiad.php: Enable writes on es4 T279281 (duration: 00m 56s)
Mentioned in SAL (#wikimedia-operations) [2021-04-27T06:55:28Z] <elukey> upgrade mariadb to 10.4.18-1 + reboot on db1108 - T279281
@razzi work done to unblock data persistence, also ran mysql_upgrade after a chat with Manuel. All good from the Analytics side @Marostegui !