Page MenuHomePhabricator
Create Task
Maniphest T331541

14 March 2023 eqiad Service repooling
Closed, ResolvedPublic
Actions

Description

A half-hour before (10:00UTC)

  • !log Locking scap deployment for service switchover - T331541
  • Add a scap lock on deploy2002.codfw.wmnet echo "Deployment lock for service switchover - T331541" | sudo tee -a /var/lock/scap-global-lock

All services

  • !log Running sre.switchdc.mediawiki.00-optional-warmup-caches - T331541
  • Run sudo cookbook sre.switchdc.mediawiki.00-optional-warmup-caches --live-test eqiad codfw
  • !log Repooling all active/active services in eqiad - T331541
  • Run sudo cookbook sre.discovery.datacenter pool eqiad --reason "Datacenter Switchover - eqiad RO repool" --task-id T331541
  • Run sudo cookbook sre.discovery.service-route --reason T331541 depool --wipe-cache codfw restbase-async
  • Unlock scap sudo rm -v /var/lock/scap-global-lock
  • !log All active/active services in eqiad repooled, scap unlocked - T331541

Related Objects
Search...

Event Timeline

Clement_Goubert created this task.Mar 8 2023, 3:29 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 8 2023, 3:29 PM
Clement_Goubert triaged this task as Medium priority.Mar 8 2023, 3:29 PM
Clement_Goubert updated the task description. (Show Details)
Clement_Goubert moved this task from Incoming 🐫 to this.quarter 🍕 on the serviceops board.
Marostegui subscribed.Mar 8 2023, 3:35 PM
Clement_Goubert changed the task status from Open to In Progress.Mar 14 2023, 9:56 AM
Clement_Goubert updated the task description. (Show Details)
Clement_Goubert updated the task description. (Show Details)Mar 14 2023, 10:00 AM
Clement_Goubert updated the task description. (Show Details)
Stashbot added a comment.Mar 14 2023, 10:02 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-14T10:02:37Z] <claime> Locking scap deployment for service switchover - T331541

Clement_Goubert updated the task description. (Show Details)Mar 14 2023, 10:25 AM
Stashbot added a comment.Mar 14 2023, 10:28 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-14T10:28:19Z] <claime> Running sre.switchdc.mediawiki.00-optional-warmup-caches - T331541

Clement_Goubert updated the task description. (Show Details)Mar 14 2023, 10:29 AM
Stashbot added a comment.Mar 14 2023, 10:32 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-14T10:32:51Z] <claime> Repooling all active/active services in eqiad - T331541

Clement_Goubert updated the task description. (Show Details)Mar 14 2023, 10:32 AM
ops-monitoring-bot added a comment.Mar 14 2023, 10:33 AM

cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: Datacenter Switchover - eqiad RO repool - T331541 started.

Stashbot added a comment.Mar 14 2023, 10:33 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-14T10:33:13Z] <cgoubert@cumin1001> START - Cookbook sre.discovery.datacenter pool all active/active services in eqiad: Datacenter Switchover - eqiad RO repool - T331541

ops-monitoring-bot added a comment.Mar 14 2023, 10:47 AM

cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: Datacenter Switchover - eqiad RO repool - T331541 completed.

Stashbot added a comment.Mar 14 2023, 10:47 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-14T10:47:52Z] <cgoubert@cumin1001> END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in eqiad: Datacenter Switchover - eqiad RO repool - T331541

Stashbot added a comment.Mar 14 2023, 10:48 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-14T10:48:12Z] <cgoubert@cumin1001> START - Cookbook sre.discovery.service-route depool restbase-async in codfw: T331541

Stashbot added a comment.Mar 14 2023, 10:48 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-14T10:48:16Z] <cgoubert@cumin1001> END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool restbase-async in codfw: T331541

Stashbot added a comment.Mar 14 2023, 11:13 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-14T11:13:37Z] <claime> We are encountering unexpected DNS anycast issued following T331541, latencies are increased but no production outage.

Clement_Goubert added a parent task: T327920: March 2023 Datacenter Switchover.Mar 14 2023, 11:38 AM
Clement_Goubert mentioned this in T331981: Warmup script does not warm memcached enough.Mar 14 2023, 11:41 AM
Stashbot added a comment.Mar 14 2023, 11:51 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-14T11:51:35Z] <cgoubert@cumin1001> START - Cookbook sre.discovery.service-route depool appservers-ro in eqiad: T331541

Stashbot added a comment.Mar 14 2023, 11:52 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-14T11:52:21Z] <cgoubert@cumin1001> END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool appservers-ro in eqiad: T331541

Stashbot added a comment.Mar 14 2023, 12:06 PM

Mentioned in SAL (#wikimedia-operations) [2023-03-14T12:06:53Z] <claime> Unlocked scap deployments - T331541

Clement_Goubert updated the task description. (Show Details)Mar 14 2023, 12:08 PM
Stashbot added a comment.Mar 14 2023, 12:08 PM

Mentioned in SAL (#wikimedia-operations) [2023-03-14T12:08:14Z] <cgoubert@cumin1001> START - Cookbook sre.discovery.service-route pool appservers-ro in eqiad: T331541

Stashbot added a comment.Mar 14 2023, 12:13 PM

Mentioned in SAL (#wikimedia-operations) [2023-03-14T12:13:18Z] <cgoubert@cumin1001> END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool appservers-ro in eqiad: T331541

Clement_Goubert mentioned this in T332009: Cookbooks that do DNS discovery change should check recdns.Mar 14 2023, 2:12 PM
Stashbot added a comment.Mar 14 2023, 2:16 PM

Mentioned in SAL (#wikimedia-operations) [2023-03-14T14:16:41Z] <claime> All active/active services in eqiad repooled, DNS issues resolved - T331541

Clement_Goubert updated the task description. (Show Details)Mar 14 2023, 2:17 PM
Clement_Goubert closed this task as Resolved.Mar 14 2023, 2:19 PM

We ran into a powerdns configuration issue which meant that instead of traffic being spread over both datacenters, we completely switched RO traffic over to eqiad.
Google doc for the issue

This has been fixed by https://gerrit.wikimedia.org/r/898738 and we are now in a stable state with eqiad pooled for RO traffic.

Tgr mentioned this in T332650: Frequent OAuth failures on Wikimedia wikis since eqiad was repooled due to db-mainstash replication lag.Mar 21 2023, 4:00 AM
Clement_Goubert closed subtask T331981: Warmup script does not warm memcached enough as Resolved.May 5 2023, 8:23 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits