Page MenuHomePhabricator
Create Task
Maniphest T307101

Reboot pc1013
Closed, ResolvedPublic
Actions

Description

Steps:

  • Stop replication on pc1014.
  • Merge CR to change pc3 primary to be pc1014: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/787497
  • Deploy: scap sync-file wmf-config/ProductionServices.php "Set pc1014 as pc3 primary T307101"
  • Downtime pc3: sudo cookbook sre.hosts.downtime --hours 1 -r "Rebooting pc1013 T307101" 'A:db-section-pc3'
  • Reboot pc1013: SKIP_DBCTL=1 SKIP_START_SLAVE=1 ~kormat/bin/reboot-host T303174 pc1013.eqiad.wmnet
  • Revert previous CR
  • Deploy: scap sync-file wmf-config/ProductionServices.php "Set pc1013 as pc3 primary T307101"
  • Start replication on pc1014.

Note that pc2013 will stay trying to replicate from pc1013 during this time.

Afterwards:

  • Move pc1014 back to pc1
  • Truncate all tables on pc1014

Details

SubjectRepoBranchLines +/-
pc1014: Move back to pc1.operations/puppetproduction+1 -1
ProductionServices: Promote pc1014 to primary of pc3operations/mediawiki-configmaster+2 -2
pc1014: Move to pc3.operations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Restricted Task
 Resolved• KormatT307101 Reboot pc1013

Event Timeline

• Kormat created this task.Apr 28 2022, 1:03 PM
gerritbot added a comment.Apr 28 2022, 1:04 PM

Change 787496 had a related patch set uploaded (by Kormat; author: Kormat):

[operations/puppet@production] pc1014: Move to pc3.

https://gerrit.wikimedia.org/r/787496

gerritbot added a project: Patch-For-Review.Apr 28 2022, 1:04 PM

Change 787496 merged by Kormat:

[operations/puppet@production] pc1014: Move to pc3.

https://gerrit.wikimedia.org/r/787496

• Kormat triaged this task as Medium priority.Apr 28 2022, 1:09 PM
• Kormat updated the task description. (Show Details)
gerritbot added a comment.Apr 28 2022, 1:10 PM

Change 787497 had a related patch set uploaded (by Kormat; author: Kormat):

[operations/mediawiki-config@master] ProductionServices: Promote pc1014 to primary of pc3

https://gerrit.wikimedia.org/r/787497

• Kormat updated the task description. (Show Details)Apr 28 2022, 1:10 PM
• Kormat updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)Apr 28 2022, 1:32 PM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)

Made some fixes - it looks good

gerritbot added a comment.Apr 28 2022, 1:40 PM

Change 787497 merged by jenkins-bot:

[operations/mediawiki-config@master] ProductionServices: Promote pc1014 to primary of pc3

https://gerrit.wikimedia.org/r/787497

Stashbot added a comment.Apr 28 2022, 1:42 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-28T13:42:06Z] <kormat@deploy1002> Synchronized wmf-config/ProductionServices.php: Set pc1014 as pc3 primary T307101 (duration: 00m 52s)

Stashbot added a comment.Apr 28 2022, 1:42 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-28T13:42:49Z] <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on pc2013.codfw.wmnet,pc[1013-1014].eqiad.wmnet with reason: Rebooting pc1013 T307101

Stashbot added a comment.Apr 28 2022, 1:42 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-28T13:42:54Z] <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2013.codfw.wmnet,pc[1013-1014].eqiad.wmnet with reason: Rebooting pc1013 T307101

• Kormat updated the task description. (Show Details)Apr 28 2022, 1:43 PM
Stashbot added a comment.Apr 28 2022, 2:08 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-28T14:08:48Z] <kormat@deploy1002> Synchronized wmf-config/ProductionServices.php: Set pc1013 as pc3 primary T307101 (duration: 00m 54s)

gerritbot added a comment.Apr 28 2022, 2:24 PM

Change 787511 had a related patch set uploaded (by Kormat; author: Kormat):

[operations/puppet@production] pc1014: Move back to pc1.

https://gerrit.wikimedia.org/r/787511

• Kormat updated the task description. (Show Details)Apr 28 2022, 2:24 PM
gerritbot added a comment.Apr 28 2022, 2:25 PM

Change 787511 merged by Kormat:

[operations/puppet@production] pc1014: Move back to pc1.

https://gerrit.wikimedia.org/r/787511

• Kormat closed this task as Resolved.Apr 28 2022, 2:31 PM
• Kormat updated the task description. (Show Details)
Maintenance_bot removed a project: Patch-For-Review.Apr 28 2022, 2:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits