Page MenuHomePhabricator

Upgrade 10.4.13 hosts to a higher version
Open, HighPublic

Description

The following hosts are running 10.4.13, we've seen a bug that made mariadb crash (T278408), although index creations are done under controlled circumstances in MW hosts, let's upgrade them to a newer version.

  • db1080
  • db1099:3311
  • db1099:3318
  • db1103
  • db1107
  • db1108:3351
  • db1108:3352
  • db1113:3315
  • db1113:3316
  • db1119
  • db1132
  • db1141
  • db1149
  • db2075
  • db2081
  • db2083
  • db2088:3311
  • db2088:3312
  • db2091
  • db2092
  • db2096
  • db2111
  • db2117
  • db2119
  • db2130
  • db2131
  • db2134
  • db2136
  • db2137:3314
  • db2137:3315
  • es1020
  • es1021
  • es1023
  • es1025

Event Timeline

Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui added a subscriber: elukey.

@elukey can you take care of db1108?
Thanks!

@elukey can you also please upgrade its kernel for T273281?

Marostegui renamed this task from Upgrade 10.4.13 hosts to Upgrade 10.4.13 hosts to a higher version.Mon, Apr 5, 1:14 PM
Marostegui updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2021-04-08T06:56:27Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es1023 to upgrade kernel and mysql, remove weight from es1021, to leave it as it was yesterday T279281', diff saved to https://phabricator.wikimedia.org/P15231 and previous config saved to /var/cache/conftool/dbconfig/20210408-065627-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2021-04-14T12:07:25Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es1025 for kernel and mysql upgrade T279281', diff saved to https://phabricator.wikimedia.org/P15333 and previous config saved to /var/cache/conftool/dbconfig/20210414-120724-marostegui.json

es1021 is es4 master, requires write depooling from MW. I will do that next week

@elukey can you take care of db1108?
Thanks!

@elukey can you also please upgrade its kernel for T273281?

@razzi can you take care of these two (kernel + mariadb upgrade) ? It can be done before refreshing the replica. The things to do are:

  • stop the replicas following the usual procedure (stop replica, stop mysql, etc..)
  • umount /srv, swapoff -a
  • verify that no backup is in progress
  • apt-get install wmf-mariadb104 (verify with apt-cache policy wmf-mariadb104 what version will be installed etc..)
  • reboot
  • start all the replicas and verify that everything is working