Page MenuHomePhabricator
Create Task
Maniphest T358200

Database post-switchover tasks March 2024
Closed, ResolvedPublic
Actions

Description

Stop and disconnect replication codfw -> eqiad
To be done: TBD

  • s1
  • s2
  • s3
  • s4
  • s5
  • s6
  • s7
  • s8
  • x1
  • es4
  • es5

*pc and x2 hosts are excluded as we maintain multi-master replication at all times.

Enable GTID on codfw masters

  • s1
  • s2
  • s3
  • s4
  • s5
  • s6
  • s7
  • s8
  • x1
  • es4
  • es5

Related Objects
Search...

Event Timeline

• Marostegui created this task.Feb 22 2024, 10:49 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 22 2024, 10:49 AM
• Marostegui changed the task status from Open to Stalled.Feb 22 2024, 10:49 AM
• Marostegui triaged this task as Medium priority.
• Marostegui moved this task from Triage to Blocked on the DBA board.

Blocked until the DC switch is done

• Marostegui updated the task description. (Show Details)Feb 22 2024, 10:50 AM
• Marostegui mentioned this in T360332: Make the cupe_actor column nullable on WMF wikis.Mar 19 2024, 10:18 AM
• Marostegui changed the task status from Stalled to Open.Mar 20 2024, 3:07 PM
• Marostegui claimed this task.
• Marostegui moved this task from Blocked to In progress on the DBA board.
Stashbot added a comment.Mar 20 2024, 3:10 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T15:10:45Z] <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 0:30:00 on 16 hosts with reason: Remove circular replication in x1 T358200

Stashbot added a comment.Mar 20 2024, 3:11 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T15:11:10Z] <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 16 hosts with reason: Remove circular replication in x1 T358200

• Marostegui updated the task description. (Show Details)Mar 20 2024, 3:12 PM
Stashbot added a comment.Mar 20 2024, 3:49 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T15:49:17Z] <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 0:30:00 on 6 hosts with reason: Remove circular replication in es5 T358200

Stashbot added a comment.Mar 20 2024, 3:49 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T15:49:23Z] <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 6 hosts with reason: Remove circular replication in es5 T358200

Stashbot added a comment.Mar 20 2024, 3:50 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T15:50:48Z] <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 0:30:00 on 6 hosts with reason: Remove circular replication in es4 T358200

Stashbot added a comment.Mar 20 2024, 3:51 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T15:51:05Z] <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 6 hosts with reason: Remove circular replication in es4 T358200

• Marostegui updated the task description. (Show Details)Mar 20 2024, 3:52 PM
Stashbot added a comment.Mar 20 2024, 3:53 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T15:53:11Z] <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 0:30:00 on 27 hosts with reason: Remove circular replication in s6 T358200

Stashbot added a comment.Mar 20 2024, 3:53 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T15:53:34Z] <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 27 hosts with reason: Remove circular replication in s6 T358200

• Marostegui updated the task description. (Show Details)Mar 20 2024, 3:54 PM
Stashbot added a comment.Mar 20 2024, 3:54 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T15:54:29Z] <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 0:30:00 on 29 hosts with reason: Remove circular replication in s2 T358200

Stashbot added a comment.Mar 20 2024, 3:54 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T15:54:54Z] <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 29 hosts with reason: Remove circular replication in s2 T358200

• Marostegui updated the task description. (Show Details)Mar 20 2024, 3:55 PM
Stashbot added a comment.Mar 20 2024, 3:56 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T15:56:05Z] <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 0:30:00 on 26 hosts with reason: Remove circular replication in s3 T358200

Stashbot added a comment.Mar 20 2024, 3:56 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T15:56:28Z] <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 26 hosts with reason: Remove circular replication in s3 T358200

• Marostegui updated the task description. (Show Details)Mar 20 2024, 3:57 PM
Stashbot added a comment.Mar 20 2024, 3:58 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T15:58:52Z] <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 0:30:00 on 31 hosts with reason: Remove circular replication in s7 T358200

Stashbot added a comment.Mar 20 2024, 3:59 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T15:59:20Z] <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 31 hosts with reason: Remove circular replication in s7 T358200

• Marostegui updated the task description. (Show Details)Mar 20 2024, 4:00 PM
Stashbot added a comment.Mar 20 2024, 4:01 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T16:01:21Z] <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 0:10:00 on 27 hosts with reason: Remove circular replication in s5 T358200

Stashbot added a comment.Mar 20 2024, 4:01 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T16:01:46Z] <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on 27 hosts with reason: Remove circular replication in s5 T358200

• Marostegui updated the task description. (Show Details)Mar 20 2024, 4:02 PM
Stashbot added a comment.Mar 20 2024, 4:03 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T16:03:40Z] <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 0:10:00 on 34 hosts with reason: Remove circular replication in s8 T358200

Stashbot added a comment.Mar 20 2024, 4:04 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T16:04:06Z] <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on 34 hosts with reason: Remove circular replication in s8 T358200

• Marostegui updated the task description. (Show Details)Mar 20 2024, 4:04 PM
Stashbot added a comment.Mar 20 2024, 4:05 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T16:05:02Z] <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 0:10:00 on 36 hosts with reason: Remove circular replication in s4 T358200

Stashbot added a comment.Mar 20 2024, 4:05 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T16:05:33Z] <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on 36 hosts with reason: Remove circular replication in s4 T358200

• Marostegui updated the task description. (Show Details)Mar 20 2024, 4:06 PM
Stashbot added a comment.Mar 20 2024, 4:07 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T16:07:06Z] <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 0:10:00 on 37 hosts with reason: Remove circular replication in s1 T358200

Stashbot added a comment.Mar 20 2024, 4:07 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T16:07:39Z] <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on 37 hosts with reason: Remove circular replication in s1 T358200

• Marostegui closed this task as Resolved.Mar 20 2024, 4:08 PM
• Marostegui updated the task description. (Show Details)

This is all done

Maintenance_bot moved this task from In progress to Done on the DBA board.Mar 20 2024, 4:29 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL