Page MenuHomePhabricator
Create Task
Maniphest T283131

Upgrade s3 to Debian Buster and MariaDB 10.4
Closed, ResolvedPublic
Actions

Description

Steps to upgrade:

Please read the doc about procedure for more details.

Original master: db1123
Original candidate master: db1157

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
db1123: Disable notifications.Kormatoperations/puppetproduction+1 -0
install_server: switch db1123 to buster.Kormatoperations/puppetproduction+0 -1
dbbackups: Switchover s3 backup source from db1171 to db1102 (buster)Jcrespooperations/puppetproduction+6 -6
db1157: Disable notifications.Kormatoperations/puppetproduction+1 -0
install_server: switch db1157 to busterKormatoperations/puppetproduction+0 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.May 19 2021, 6:54 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 19 2021, 6:54 AM
Marostegui changed the task status from Open to Stalled.May 19 2021, 6:57 AM
Marostegui assigned this task to • Kormat.
Marostegui triaged this task as Medium priority.
Marostegui moved this task from Triage to Blocked on the DBA board.
Marostegui added a subscriber: • Kormat.

Assigning this to @Kormat as she's done s6 already. This should be stalled for now and only to be done once we are happy with s6's performance/stability in around 3-4 weeks or so.
codfw is entirely done, so is sanitarium mastesr, so effectively what is pending:

  • Confirm backup sources are good to go (cc @jcrespo
  • Upgrade eqiad candidate master to buster and 10.4
  • Perform eqiad master switchover
  • Upgrade old master to buster and 10.4
Marostegui updated the task description. (Show Details)May 19 2021, 6:58 AM
• jcrespo updated the task description. (Show Details)May 19 2021, 6:58 AM
Marostegui updated the task description. (Show Details)May 19 2021, 6:59 AM
gerritbot added a comment.May 19 2021, 7:05 AM

Change 692845 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Switchover s3 backup source from db1171 to db1102 (buster)

https://gerrit.wikimedia.org/r/692845

gerritbot added a project: Patch-For-Review.May 19 2021, 7:05 AM
• jcrespo updated the task description. (Show Details)May 19 2021, 7:06 AM
Marostegui mentioned this in T266486: Schema change to turn user_last_timestamp.user_newtalk to binary(14).May 20 2021, 10:26 AM
Marostegui mentioned this in T268392: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14).
Marostegui mentioned this in T273360: Schema change for dropping default of img_timestamp and making it binary(14).
Marostegui added a comment.May 26 2021, 5:40 AM

Upgraded all s3 codfw to 10.4.19

Marostegui added a comment.Jun 3 2021, 9:06 AM

s6 hasn't given any issues, so maybe we can start working on this next week (after 3 weeks since we switched s6) and attempt to do the switchover the 17th?
@Kormat thoughts?

Marostegui added a parent task: T266486: Schema change to turn user_last_timestamp.user_newtalk to binary(14).Jun 4 2021, 8:59 AM
Marostegui added a parent task: T268392: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14).
Marostegui added a parent task: T273360: Schema change for dropping default of img_timestamp and making it binary(14).
• Kormat added a comment.Jun 4 2021, 10:25 AM
In T283131#7131272, @Marostegui wrote:

s6 hasn't given any issues, so maybe we can start working on this next week (after 3 weeks since we switched s6) and attempt to do the switchover the 17th?
@Kormat thoughts?

Sounds good :)

• Kormat updated the task description. (Show Details)Jun 7 2021, 9:55 AM
gerritbot added a comment.Jun 7 2021, 10:05 AM

Change 698469 had a related patch set uploaded (by Kormat; author: Kormat):

[operations/puppet@production] db1157: Disable notificcations.

https://gerrit.wikimedia.org/r/698469

Stashbot added a comment.Jun 7 2021, 10:08 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-07T10:08:23Z] <kormat@cumin1001> dbctl commit (dc=all): 'db1157 depooling: reimage to buster T283131', diff saved to https://phabricator.wikimedia.org/P16311 and previous config saved to /var/cache/conftool/dbconfig/20210607-100822-kormat.json

gerritbot added a comment.Jun 7 2021, 10:09 AM

Change 698469 merged by Kormat:

[operations/puppet@production] db1157: Disable notifications.

https://gerrit.wikimedia.org/r/698469

gerritbot added a comment.Jun 7 2021, 10:11 AM

Change 698471 had a related patch set uploaded (by Kormat; author: Kormat):

[operations/puppet@production] install_server: switch db1157 to buster

https://gerrit.wikimedia.org/r/698471

gerritbot added a comment.Jun 7 2021, 10:12 AM

Change 698471 merged by Kormat:

[operations/puppet@production] install_server: switch db1157 to buster

https://gerrit.wikimedia.org/r/698471

ops-monitoring-bot added a comment.Jun 7 2021, 10:21 AM

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['db1157.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202106071021_kormat_12649.log.

Stashbot added a comment.Jun 7 2021, 10:28 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-07T10:28:57Z] <kormat> reimaging db1157 T283131

ops-monitoring-bot added a comment.Jun 7 2021, 10:46 AM

Completed auto-reimage of hosts:

['db1157.eqiad.wmnet']

and were ALL successful.

• Kormat added a comment.Jun 7 2021, 11:56 AM

db1157 upgraded to buster. Running mysqlcheck now.

Stashbot added a comment.Jun 8 2021, 10:53 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-08T10:53:47Z] <kormat@cumin1001> dbctl commit (dc=all): 'db1157 (re)pooling @ 25%: reimaged to buster T283131', diff saved to https://phabricator.wikimedia.org/P16326 and previous config saved to /var/cache/conftool/dbconfig/20210608-105346-kormat.json

• Kormat added a comment.Jun 8 2021, 10:53 AM

db1157 had a clean mysqlcheck run, repooling it now.

Stashbot added a comment.Jun 8 2021, 11:08 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-08T11:08:51Z] <kormat@cumin1001> dbctl commit (dc=all): 'db1157 (re)pooling @ 50%: reimaged to buster T283131', diff saved to https://phabricator.wikimedia.org/P16327 and previous config saved to /var/cache/conftool/dbconfig/20210608-110850-kormat.json

Stashbot added a comment.Jun 8 2021, 11:23 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-08T11:23:54Z] <kormat@cumin1001> dbctl commit (dc=all): 'db1157 (re)pooling @ 75%: reimaged to buster T283131', diff saved to https://phabricator.wikimedia.org/P16328 and previous config saved to /var/cache/conftool/dbconfig/20210608-112354-kormat.json

Stashbot added a comment.Jun 8 2021, 11:38 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-08T11:38:58Z] <kormat@cumin1001> dbctl commit (dc=all): 'db1157 (re)pooling @ 100%: reimaged to buster T283131', diff saved to https://phabricator.wikimedia.org/P16329 and previous config saved to /var/cache/conftool/dbconfig/20210608-113857-kormat.json

• Kormat updated the task description. (Show Details)Jun 8 2021, 12:10 PM
gerritbot added a comment.Jun 21 2021, 5:23 AM

Change 700463 had a related patch set uploaded (by Kormat; author: Kormat):

[operations/puppet@production] db1123: Disable notifications.

https://gerrit.wikimedia.org/r/700463

gerritbot added a comment.Jun 21 2021, 5:24 AM

Change 700463 merged by Kormat:

[operations/puppet@production] db1123: Disable notifications.

https://gerrit.wikimedia.org/r/700463

gerritbot added a comment.Jun 21 2021, 5:27 AM

Change 692845 merged by Jcrespo:

[operations/puppet@production] dbbackups: Switchover s3 backup source from db1171 to db1102 (buster)

https://gerrit.wikimedia.org/r/692845

Stashbot added a comment.Jun 21 2021, 5:31 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-21T05:31:19Z] <kormat> stopping replication on db1123 T283131

• Kormat updated the task description. (Show Details)Jun 21 2021, 5:32 AM
• jcrespo updated the task description. (Show Details)Jun 21 2021, 5:33 AM
Maintenance_bot removed a project: Patch-For-Review.Jun 21 2021, 6:10 AM
• Kormat closed subtask T284648: Switchover s3 from db1123 to db1157 as Resolved.Jun 21 2021, 11:33 AM
gerritbot added a comment.Jun 21 2021, 1:42 PM

Change 700613 had a related patch set uploaded (by Kormat; author: Kormat):

[operations/puppet@production] install_server: switch db1123 to buster.

https://gerrit.wikimedia.org/r/700613

gerritbot added a project: Patch-For-Review.Jun 21 2021, 1:42 PM

Change 700613 merged by Kormat:

[operations/puppet@production] install_server: switch db1123 to buster.

https://gerrit.wikimedia.org/r/700613

ops-monitoring-bot added a comment.Jun 21 2021, 2:01 PM

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['db1123.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202106211401_kormat_3114.log.

ops-monitoring-bot added a comment.Jun 21 2021, 2:31 PM

Completed auto-reimage of hosts:

['db1123.eqiad.wmnet']

and were ALL successful.

• Kormat added a comment.Jun 21 2021, 2:41 PM

db1123 is reimaged to buster, mysqlcheck --all-databases running now. As this is s3, this is going to take A While.

Stashbot added a comment.Jun 22 2021, 10:21 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-22T10:21:09Z] <kormat@cumin1001> dbctl commit (dc=all): 'db1123 (re)pooling @ 25%: reimaged to buster T283131', diff saved to https://phabricator.wikimedia.org/P16688 and previous config saved to /var/cache/conftool/dbconfig/20210622-102108-kormat.json

• Kormat added a comment.Jun 22 2021, 10:21 AM

db1123 had a clean mysqlcheck run, repooling it now.

• Kormat updated the task description. (Show Details)Jun 22 2021, 10:21 AM
Stashbot added a comment.Jun 22 2021, 10:36 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-22T10:36:12Z] <kormat@cumin1001> dbctl commit (dc=all): 'db1123 (re)pooling @ 50%: reimaged to buster T283131', diff saved to https://phabricator.wikimedia.org/P16689 and previous config saved to /var/cache/conftool/dbconfig/20210622-103612-kormat.json

Stashbot added a comment.Jun 22 2021, 10:51 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-22T10:51:16Z] <kormat@cumin1001> dbctl commit (dc=all): 'db1123 (re)pooling @ 75%: reimaged to buster T283131', diff saved to https://phabricator.wikimedia.org/P16690 and previous config saved to /var/cache/conftool/dbconfig/20210622-105115-kormat.json

Stashbot added a comment.Jun 22 2021, 11:06 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-22T11:06:20Z] <kormat@cumin1001> dbctl commit (dc=all): 'db1123 (re)pooling @ 100%: reimaged to buster T283131', diff saved to https://phabricator.wikimedia.org/P16691 and previous config saved to /var/cache/conftool/dbconfig/20210622-110619-kormat.json

• Kormat reassigned this task from • Kormat to • jcrespo.Jun 22 2021, 12:40 PM

@jcrespo : assigning to you for the final backups step. Please resolve the task when that's done. Thanks!

• jcrespo changed the task status from Stalled to Open.Jun 28 2021, 9:27 AM
• jcrespo closed this task as Resolved.
• jcrespo reassigned this task from • jcrespo to • Kormat.
• jcrespo updated the task description. (Show Details)

There should be no more s3 servers with 10.1 available, that I am aware, but feel free to double check.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits