Change Details

This will pick up the kernel fix for T261389. As we're not changing the es5 master, just rebooting it, we can use a simplified version of the normal maintenance procedure. **Time**: Wed, 2020-11-25, 0800 UTC (0900 CET) Steps: [x] Downtime all es5 hosts: `sudo -H cookbook sre.hosts.downtime --minutes 30 -r "Reboot es1024 for kernel upgrade T268469" 'A:db-section-es5'` [x] Merge mediawiki-config CR to disable writes to es5 https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/643030 [x] Deploy MW change from deploy1001: `cd /srv/mediawiki-staging/; git status; git fetch; git rebase; scap sync-file wmf-config/db-eqiad.php "Disable writes to es5 T268469"` [ ] Check that es5 is read-only (only heartbeat update statements in mysqlbinlog) [ ] Stop mariadb on es1024 [ ] Check [[ https://i.imgur.com/o5CW9v5.png | kibana ]] to ensure that MW is coping fine. [ ] Reboot es1024 [ ] Start mariadb: `systemctl start mariadb` [ ] Restart prom exporter: `systemctl restart prometheus-mysqld-exporter` [ ] Check that replication is working correctly: `sudo -H db-replication-tree es1024` [ ] Revert MW change [ ] Deploy MW revert from deploy1001: `cd /srv/mediawiki-staging/; git status; git fetch; git rebase; scap sync-file wmf-config/db-eqiad.php "Re-enable writes to es5 T268469"` [ ] Check that icinga is all green

This will pick up the kernel fix for T261389. As we're not changing the es5 master, just rebooting it, we can use a simplified version of the normal maintenance procedure. **Time**: Wed, 2020-11-25, 0800 UTC (0900 CET) Steps: [x] Downtime all es5 hosts: `sudo -H cookbook sre.hosts.downtime --minutes 30 -r "Reboot es1024 for kernel upgrade T268469" 'A:db-section-es5'` [x] Merge mediawiki-config CR to disable writes to es5 https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/643030 [x] Deploy MW change from deploy1001: `cd /srv/mediawiki-staging/; git status; git fetch; git rebase; scap sync-file wmf-config/db-eqiad.php "Disable writes to es5 T268469"` [x] Check that es5 is read-only (only heartbeat update statements in mysqlbinlog) [x] Stop mariadb on es1024 [x] Check [[ https://i.imgur.com/o5CW9v5.png | kibana ]] to ensure that MW is coping fine. [ ] Reboot es1024 [ ] Start mariadb: `systemctl start mariadb` [ ] Restart prom exporter: `systemctl restart prometheus-mysqld-exporter` [ ] Check that replication is working correctly: `sudo -H db-replication-tree es1024` [ ] Revert MW change [ ] Deploy MW revert from deploy1001: `cd /srv/mediawiki-staging/; git status; git fetch; git rebase; scap sync-file wmf-config/db-eqiad.php "Re-enable writes to es5 T268469"` [ ] Check that icinga is all green

This will pick up the kernel fix for T261389. As we're not changing the es5 master, just rebooting it, we can use a simplified version of the normal maintenance procedure. **Time**: Wed, 2020-11-25, 0800 UTC (0900 CET) Steps: [x] Downtime all es5 hosts: `sudo -H cookbook sre.hosts.downtime --minutes 30 -r "Reboot es1024 for kernel upgrade T268469" 'A:db-section-es5'` [x] Merge mediawiki-config CR to disable writes to es5 https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/643030 [x] Deploy MW change from deploy1001: `cd /srv/mediawiki-staging/; git status; git fetch; git rebase; scap sync-file wmf-config/db-eqiad.php "Disable writes to es5 T268469"` [ x] Check that es5 is read-only (only heartbeat update statements in mysqlbinlog) [ x] Stop mariadb on es1024 [ x] Check [[ https://i.imgur.com/o5CW9v5.png | kibana ]] to ensure that MW is coping fine. [ ] Reboot es1024 [ ] Start mariadb: `systemctl start mariadb` [ ] Restart prom exporter: `systemctl restart prometheus-mysqld-exporter` [ ] Check that replication is working correctly: `sudo -H db-replication-tree es1024` [ ] Revert MW change [ ] Deploy MW revert from deploy1001: `cd /srv/mediawiki-staging/; git status; git fetch; git rebase; scap sync-file wmf-config/db-eqiad.php "Re-enable writes to es5 T268469"` [ ] Check that icinga is all green