Page MenuHomePhabricator
Create Task
Maniphest T280751

Upgrade s6 to Debian Buster and MariaDB 10.4
Closed, ResolvedPublic
Actions

Description

s6 should be the first section to be upgraded entirely to 10.4 and Buster.

Steps to upgrade:

Please read the doc about procedure for more details.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
• Marostegui added a comment.EditedApr 21 2021, 2:27 PM

db1165 is ready to take over db1085 in s6 as sanitarium master.
They both need to be stopped at the same time and move db1155 (sanitarium for s2) under it. db1125 (old sanitarium) doesn't need to be moved, as it will be repurposed somewhere else (T258361)

Kormat claimed this task.Apr 26 2021, 9:17 AM
Kormat updated the task description. (Show Details)
Kormat added a comment.EditedMay 3 2021, 12:55 PM

Steps to swap db1085 -> db1165:

  • Silence db1085, db1165 & db1155, and wikireplicas:
sudo cookbook sre.hosts.downtime --minutes 60 -r "Replace db1085 with db1165 T280751"  'A:db-clouddb or A:db-labsdb or P{db1085* or db1155* or db1165*}'
  • Depool instances:
sudo dbctl instance db1085 depool
sudo dbctl instance db1165 depool
sudo dbctl config commit -m "Depooling for sanitarium master switch T280751"
  • db-stop-in-sync db1085 db1165, and wait for db1155:s6 to catch up.
  • Switch db1155 over:
master-pos db1165.eqiad.wmnet
sudo mysql.py -h db1155:s6
> stop slave;
> reset slave all;
> change master to ...;
> start slave;
• Marostegui added a comment.EditedMay 4 2021, 4:53 AM
In T280751#7052806, @Kormat wrote:

Steps to swap db1085 -> db1165:

  • Silence db1085, db1165 & db1155.
  • Depool instances:
sudo dbctl instance db1085 depool
sudo dbctl instance db1165 depool
sudo dbctl config commit -m "Depooling for sanitarium master switch T280751"
  • db-stop-in-sync db1085 db1165, and wait for db1155:s6 to catch up.
  • Switch db1155 over:
master-pos db1165.eqiad.wmnet
sudo mysql.py -h db1155:s6
> stop slave;
> change master to ...;
> start slave;
  • Start replication again on db1165: sudo mysql.py -h db1165 -e 'start slave'
  • Wait for db1165 to catch up, then repool it.
  • Merge puppet commit with the heira changes
  • Remove db1085 from dbctl?

This works for me, small notes:

  • you might want to silence also the cloudb* hosts, as those will also alert on IRC.
  • After the stop slave you might want also to run reset slave all
  • Once db1155:s6 has caught up enable GTID: STOP SLAVE; CHANGE MASTER TO MASTER_USE_GTID=Slave_pos; START SLAVE;
  • Start replication on db1085
  • I would leave db1085 depooled (but still on dbctl) and if after a few days everything goes well, decom it!.

Thanks!

Stashbot added a comment.May 4 2021, 12:34 PM

Mentioned in SAL (#wikimedia-operations) [2021-05-04T12:34:36Z] <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 15 hosts with reason: Replace db1085 with db1165 T280751

Stashbot added a comment.May 4 2021, 12:34 PM

Mentioned in SAL (#wikimedia-operations) [2021-05-04T12:34:45Z] <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Replace db1085 with db1165 T280751

Stashbot added a comment.May 4 2021, 12:35 PM

Mentioned in SAL (#wikimedia-operations) [2021-05-04T12:35:38Z] <kormat@cumin1001> dbctl commit (dc=all): 'Depooling for sanitarium master switch T280751', diff saved to https://phabricator.wikimedia.org/P15714 and previous config saved to /var/cache/conftool/dbconfig/20210504-123537-kormat.json

Stashbot added a comment.May 4 2021, 12:46 PM

Mentioned in SAL (#wikimedia-operations) [2021-05-04T12:46:48Z] <kormat@cumin1001> dbctl commit (dc=all): 'Repooling after sanitarium master switch T280751', diff saved to https://phabricator.wikimedia.org/P15715 and previous config saved to /var/cache/conftool/dbconfig/20210504-124647-kormat.json

gerritbot added a comment.May 4 2021, 12:49 PM

Change 684895 had a related patch set uploaded (by Kormat; author: Kormat):

[operations/puppet@production] mariadb: Remove obsolete comment.

https://gerrit.wikimedia.org/r/684895

gerritbot added a comment.May 4 2021, 12:50 PM

Change 684895 merged by Kormat:

[operations/puppet@production] mariadb: Remove obsolete comment.

https://gerrit.wikimedia.org/r/684895

Kormat updated the task description. (Show Details)May 4 2021, 12:51 PM
jcrespo updated the task description. (Show Details)May 5 2021, 12:28 PM
Kormat updated the task description. (Show Details)May 5 2021, 12:29 PM

Turns out the candidate master for s6/codfw (db2114) is already running buster/10.4.

jcrespo updated the task description. (Show Details)May 5 2021, 12:30 PM
gerritbot added a comment.May 5 2021, 1:08 PM

Change 685440 had a related patch set uploaded (by Kormat; author: Kormat):

[operations/puppet@production] install_server: Switch db2129 to buster.

https://gerrit.wikimedia.org/r/685440

gerritbot added a comment.May 5 2021, 1:10 PM

Change 685440 merged by Kormat:

[operations/puppet@production] install_server: Switch db2129 to buster.

https://gerrit.wikimedia.org/r/685440

Stashbot added a comment.May 5 2021, 1:12 PM

Mentioned in SAL (#wikimedia-operations) [2021-05-05T13:12:30Z] <kormat> reimaging db2129 to buster T280751

ops-monitoring-bot added a comment.May 5 2021, 1:15 PM

Script wmf-auto-reimage was launched by kormat on cumin2001.codfw.wmnet for hosts:

['db2129.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202105051315_kormat_6374.log.

Stashbot added a comment.May 5 2021, 1:19 PM

Mentioned in SAL (#wikimedia-operations) [2021-05-05T13:19:49Z] <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 9 hosts with reason: Reimage db2129 T280751

Stashbot added a comment.May 5 2021, 1:19 PM

Mentioned in SAL (#wikimedia-operations) [2021-05-05T13:19:57Z] <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 9 hosts with reason: Reimage db2129 T280751

• Marostegui mentioned this in T266486: Schema change to turn user_last_timestamp.user_newtalk to binary(14).May 5 2021, 1:57 PM
• Marostegui mentioned this in T273360: Schema change for dropping default of img_timestamp and making it binary(14).
• Marostegui mentioned this in T268392: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14).
Stashbot added a comment.May 5 2021, 2:28 PM

Mentioned in SAL (#wikimedia-operations) [2021-05-05T14:28:47Z] <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 10 hosts with reason: Reimage db2129 T280751

Stashbot added a comment.May 5 2021, 2:28 PM

Mentioned in SAL (#wikimedia-operations) [2021-05-05T14:28:56Z] <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 10 hosts with reason: Reimage db2129 T280751

ops-monitoring-bot added a comment.May 5 2021, 2:49 PM

Completed auto-reimage of hosts:

['db2129.codfw.wmnet']

and were ALL successful.

Kormat updated the task description. (Show Details)May 5 2021, 2:57 PM
gerritbot added a comment.May 5 2021, 3:02 PM

Change 681621 merged by Jcrespo:

[operations/puppet@production] dbbackups: Switchover s6 codfw database backups from db2097 to db2141

https://gerrit.wikimedia.org/r/681621

Stashbot added a comment.May 5 2021, 3:10 PM

Mentioned in SAL (#wikimedia-operations) [2021-05-05T15:10:01Z] <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on 10 hosts with reason: Table check on db2129 T280751

Stashbot added a comment.May 5 2021, 3:10 PM

Mentioned in SAL (#wikimedia-operations) [2021-05-05T15:10:08Z] <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 10 hosts with reason: Table check on db2129 T280751

jcrespo updated the task description. (Show Details)May 5 2021, 3:12 PM
gerritbot added a comment.May 5 2021, 3:18 PM

Change 685494 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Switchover backup generation for s6 on eqiad from db1139 to db1140

https://gerrit.wikimedia.org/r/685494

jcrespo updated the task description. (Show Details)May 5 2021, 3:19 PM

^I've prepared the backup failover for eqiad :-)

Kormat added a comment.May 6 2021, 8:30 AM

mysqlcheck --all-databases completed successfully on db2129.

gerritbot added a comment.May 6 2021, 8:39 AM

Change 685726 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: remove db2097 s6 section for this codfw backup source

https://gerrit.wikimedia.org/r/685726

gerritbot added a comment.May 6 2021, 11:08 AM

Change 685753 had a related patch set uploaded (by Kormat; author: Kormat):

[operations/puppet@production] install_server: switch db1173 to buster

https://gerrit.wikimedia.org/r/685753

gerritbot added a comment.May 6 2021, 11:09 AM

Change 685754 had a related patch set uploaded (by Kormat; author: Kormat):

[operations/puppet@production] db1173: Disable notifications

https://gerrit.wikimedia.org/r/685754

gerritbot added a comment.May 6 2021, 11:10 AM

Change 685753 merged by Kormat:

[operations/puppet@production] install_server: switch db1173 to buster

https://gerrit.wikimedia.org/r/685753

gerritbot added a comment.May 6 2021, 11:10 AM

Change 685754 merged by Kormat:

[operations/puppet@production] db1173: Disable notifications

https://gerrit.wikimedia.org/r/685754

Stashbot added a comment.May 6 2021, 11:12 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-06T11:12:25Z] <kormat> reimaging db1173 to buster T280751

Stashbot added a comment.May 6 2021, 11:12 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-06T11:12:56Z] <kormat@cumin1001> dbctl commit (dc=all): 'db1173 depooling: Reimage to buster T280751', diff saved to https://phabricator.wikimedia.org/P15824 and previous config saved to /var/cache/conftool/dbconfig/20210506-111256-kormat.json

ops-monitoring-bot added a comment.May 6 2021, 11:18 AM

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['db1173.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202105061118_kormat_25433.log.

ops-monitoring-bot added a comment.May 6 2021, 11:43 AM

Completed auto-reimage of hosts:

['db1173.eqiad.wmnet']

and were ALL successful.

Kormat updated the task description. (Show Details)May 6 2021, 11:54 AM

db1173 (candidate master in eqiad) reimaged to buster, mysqlcheck --all-databases running now.

Kormat moved this task from Ready to In progress on the DBA board.May 6 2021, 12:24 PM
Stashbot added a comment.May 7 2021, 11:33 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-07T11:33:56Z] <kormat@cumin1001> dbctl commit (dc=all): 'db1173 (re)pooling @ 25%: reimaged to buster T280751', diff saved to https://phabricator.wikimedia.org/P15853 and previous config saved to /var/cache/conftool/dbconfig/20210507-113355-kormat.json

Stashbot added a comment.May 7 2021, 11:49 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-07T11:49:01Z] <kormat@cumin1001> dbctl commit (dc=all): 'db1173 (re)pooling @ 50%: reimaged to buster T280751', diff saved to https://phabricator.wikimedia.org/P15854 and previous config saved to /var/cache/conftool/dbconfig/20210507-114859-kormat.json

Stashbot added a comment.May 7 2021, 12:04 PM

Mentioned in SAL (#wikimedia-operations) [2021-05-07T12:04:04Z] <kormat@cumin1001> dbctl commit (dc=all): 'db1173 (re)pooling @ 75%: reimaged to buster T280751', diff saved to https://phabricator.wikimedia.org/P15855 and previous config saved to /var/cache/conftool/dbconfig/20210507-120404-kormat.json

Stashbot added a comment.May 7 2021, 12:19 PM

Mentioned in SAL (#wikimedia-operations) [2021-05-07T12:19:08Z] <kormat@cumin1001> dbctl commit (dc=all): 'db1173 (re)pooling @ 100%: reimaged to buster T280751', diff saved to https://phabricator.wikimedia.org/P15856 and previous config saved to /var/cache/conftool/dbconfig/20210507-121908-kormat.json

Kormat closed subtask T282124: Switchover s6 from db1131 to db1173 as Resolved.May 17 2021, 5:26 AM
gerritbot added a comment.May 17 2021, 9:07 AM

Change 685494 merged by Jcrespo:

[operations/puppet@production] dbbackups: Switchover backup generation for s6 on eqiad from db1139 to db1140

https://gerrit.wikimedia.org/r/685494

Kormat updated the task description. (Show Details)May 17 2021, 9:40 AM
Kormat updated the task description. (Show Details)
gerritbot added a comment.May 17 2021, 12:28 PM

Change 681622 abandoned by Jcrespo:

[operations/puppet@production] dbbackups: Switchover s6 eqiad database backups from db1139 to db1140

Reason:

duplicate of https://gerrit.wikimedia.org/r/c/operations/puppet/ /685494

https://gerrit.wikimedia.org/r/681622

gerritbot added a comment.May 17 2021, 12:52 PM

Change 692314 had a related patch set uploaded (by Kormat; author: Kormat):

[operations/puppet@production] install_server: Switch db1131 to buster.

https://gerrit.wikimedia.org/r/692314

gerritbot added a comment.May 17 2021, 12:52 PM

Change 692315 had a related patch set uploaded (by Kormat; author: Kormat):

[operations/puppet@production] db1131: Disable notifications.

https://gerrit.wikimedia.org/r/692315

gerritbot added a comment.May 17 2021, 12:57 PM

Change 692314 merged by Kormat:

[operations/puppet@production] install_server: Switch db1131 to buster.

https://gerrit.wikimedia.org/r/692314

gerritbot added a comment.May 17 2021, 12:57 PM

Change 692315 merged by Kormat:

[operations/puppet@production] db1131: Disable notifications.

https://gerrit.wikimedia.org/r/692315

ops-monitoring-bot added a comment.May 17 2021, 1:09 PM

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['db1131.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202105171309_kormat_32150.log.

ops-monitoring-bot added a comment.May 17 2021, 1:57 PM

Completed auto-reimage of hosts:

['db1131.eqiad.wmnet']

and were ALL successful.

Kormat added a comment.May 17 2021, 2:35 PM

db1131 (old s6 eqiad primary instance) has been reimaged to buster. It's now running mariadb-check -A.

gerritbot added a comment.May 17 2021, 2:41 PM

Change 692341 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Remove s6 stretch backup source instance on eqiad

https://gerrit.wikimedia.org/r/692341

gerritbot added a comment.May 17 2021, 5:05 PM

Change 685726 merged by Jcrespo:

[operations/puppet@production] dbbackups: remove db2097 s6 section for this codfw backup source

https://gerrit.wikimedia.org/r/685726

jcrespo added a comment.May 17 2021, 5:34 PM

db2097:s6 removed, only db1139:s6 left (to do next week).

Stashbot added a comment.May 18 2021, 9:01 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-18T09:01:57Z] <kormat@cumin1001> dbctl commit (dc=all): 'Remove s6 eqiad primary from 'api' group T280751', diff saved to https://phabricator.wikimedia.org/P16043 and previous config saved to /var/cache/conftool/dbconfig/20210518-090156-kormat.json

Stashbot added a comment.May 18 2021, 9:02 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-18T09:02:16Z] <kormat@cumin1001> dbctl commit (dc=all): 'db1131 (re)pooling @ 25%: reimaged to buster T280751', diff saved to https://phabricator.wikimedia.org/P16045 and previous config saved to /var/cache/conftool/dbconfig/20210518-090215-kormat.json

Kormat updated the task description. (Show Details)May 18 2021, 9:03 AM
Stashbot added a comment.May 18 2021, 9:04 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-18T09:04:50Z] <kormat@cumin1001> dbctl commit (dc=all): 'Set db1131 to weight 400 in s6/eqiad T280751', diff saved to https://phabricator.wikimedia.org/P16046 and previous config saved to /var/cache/conftool/dbconfig/20210518-090449-kormat.json

Stashbot added a comment.May 18 2021, 9:17 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-18T09:17:26Z] <kormat@cumin1001> dbctl commit (dc=all): 'db1131 (re)pooling @ 50%: reimaged to buster T280751', diff saved to https://phabricator.wikimedia.org/P16049 and previous config saved to /var/cache/conftool/dbconfig/20210518-091725-kormat.json

Stashbot added a comment.May 18 2021, 9:32 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-18T09:32:29Z] <kormat@cumin1001> dbctl commit (dc=all): 'db1131 (re)pooling @ 75%: reimaged to buster T280751', diff saved to https://phabricator.wikimedia.org/P16050 and previous config saved to /var/cache/conftool/dbconfig/20210518-093228-kormat.json

Kormat updated the task description. (Show Details)May 18 2021, 9:34 AM

db1131 completed the databases check successfully, and is now being repooled.

Stashbot added a comment.May 18 2021, 9:47 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-18T09:47:33Z] <kormat@cumin1001> dbctl commit (dc=all): 'db1131 (re)pooling @ 100%: reimaged to buster T280751', diff saved to https://phabricator.wikimedia.org/P16053 and previous config saved to /var/cache/conftool/dbconfig/20210518-094732-kormat.json

Kormat reassigned this task from Kormat to jcrespo.May 19 2021, 8:59 AM
jcrespo updated the task description. (Show Details)May 19 2021, 9:02 AM
gerritbot added a comment.May 26 2021, 8:18 AM

Change 692341 merged by Jcrespo:

[operations/puppet@production] dbbackups: Remove s6 stretch backup source instance on eqiad

https://gerrit.wikimedia.org/r/692341

jcrespo updated the task description. (Show Details)May 26 2021, 8:34 AM
jcrespo reassigned this task from jcrespo to Kormat.May 26 2021, 8:39 AM

AFAICS, all of s6 is in buster/10.4:

Screenshot from 2021-05-26 10-38-40.png (868×2 px, 587 KB)

Kormat closed this task as Resolved.May 26 2021, 8:47 AM

🎉

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL