Page MenuHomePhabricator
Create Task
Maniphest T330271

March 2023 Datacenter Switchover live test
Closed, ResolvedPublic
Actions

Description

Meta-task for live-test tracking and possible work to do afterwards.

Related Objects
Search...

Event Timeline

Clement_Goubert changed the task status from Open to In Progress.Feb 22 2023, 10:33 AM
Clement_Goubert triaged this task as High priority.
Clement_Goubert created this task.
Clement_Goubert moved this task from Incoming 🐫 to Doing 😎 on the serviceops board.
Clement_Goubert updated the task description. (Show Details)EditedFeb 22 2023, 10:36 AM
10:35 <+logmsgbot> !log cgoubert@cumin1001 START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl
10:35 <+logmsgbot> !log cgoubert@cumin1001 END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0)
Clement_Goubert added a comment.Feb 22 2023, 10:48 AM

Skipping 00-optional-warmup-caches as the node script is broken and the replacement python script hasn't been reviewed yet.

Clement_Goubert added a comment.Feb 22 2023, 11:01 AM
11:01 <+logmsgbot> !log cgoubert@cumin1001 START - Cookbook sre.switchdc.mediawiki.00-disable-puppet
11:01 <+logmsgbot> !log cgoubert@cumin1001 END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0)
Clement_Goubert added a comment.Feb 22 2023, 11:02 AM
11:01 <+logmsgbot> !log cgoubert@cumin1001 START - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks
11:01 <+logmsgbot> !log cgoubert@cumin1001 END (PASS) - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks (exit_code=0)
Clement_Goubert added a comment.Feb 22 2023, 11:03 AM
11:02 <+logmsgbot> !log cgoubert@cumin1001 START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance
11:02 <+logmsgbot> !log cgoubert@cumin1001 END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0)
Clement_Goubert added a comment.Feb 22 2023, 11:05 AM
11:03 <+logmsgbot> !log cgoubert@cumin1001 START - Cookbook sre.switchdc.mediawiki.02-set-readonly
11:03 <+logmsgbot> !log cgoubert@cumin1001 [DRY-RUN] MediaWiki read-only period starts at: 2023-02-22 11:03:19.149671
11:03 <+logmsgbot> !log cgoubert@cumin1001 END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0)
11:03 <+logmsgbot> !log cgoubert@cumin1001 START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly
11:04 <+logmsgbot> !log cgoubert@cumin1001 END (FAIL) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=99)
Clement_Goubert added a comment.Feb 22 2023, 11:05 AM
spicerack.mysql_legacy.MysqlLegacyError: Unable to get heartbeat from master db1118.eqiad.wmnet for section s1
Clement_Goubert added a comment.Feb 22 2023, 11:12 AM
This comment was removed by Clement_Goubert.
Clement_Goubert added a comment.EditedFeb 22 2023, 11:15 AM
11:13 <+logmsgbot> !log cgoubert@cumin1001 START - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki
11:13 <+logmsgbot> !log cgoubert@cumin1001 END (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0)
11:13 <+logmsgbot> !log cgoubert@cumin1001 START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite
11:13 <+logmsgbot> !log cgoubert@cumin1001 END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0)
11:13 <+logmsgbot> !log cgoubert@cumin1001 START - Cookbook sre.switchdc.mediawiki.07-set-readwrite
11:13 <+logmsgbot> !log cgoubert@cumin1001 [DRY-RUN] MediaWiki read-only period ends at: 2023-02-22 11:13:51.466468
11:13 <+logmsgbot> !log cgoubert@cumin1001 END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0)
11:14 <+logmsgbot> !log cgoubert@cumin1001 START - Cookbook sre.switchdc.mediawiki.08-restart-envoy-on-jobrunners
11:14 <+logmsgbot> !log cgoubert@cumin1001 END (PASS) - Cookbook sre.switchdc.mediawiki.08-restart-envoy-on-jobrunners (exit_code=0)
11:14 <+logmsgbot> !log cgoubert@cumin1001 START - Cookbook sre.switchdc.mediawiki.08-start-maintenance
11:16:26     +logmsgbot │ !log cgoubert@cumin1001 END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0)
Clement_Goubert added a comment.EditedFeb 22 2023, 11:24 AM
11:18 <+logmsgbot> !log cgoubert@cumin1001 START - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters
11:24 <+logmsgbot> !log cgoubert@cumin1001 END (PASS) - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters (exit_code=0)
Clement_Goubert mentioned this in T330300: sre.switchdc.mediawiki.07-set-readwrite doesn't reset both datacenter to rw.Feb 22 2023, 3:13 PM
Clement_Goubert closed subtask T330300: sre.switchdc.mediawiki.07-set-readwrite doesn't reset both datacenter to rw as Resolved.Feb 23 2023, 12:50 PM
Clement_Goubert added a comment.Feb 23 2023, 4:14 PM

Incident doc relating the minor editing incident due to T330300: sre.switchdc.mediawiki.07-set-readwrite doesn't reset both datacenter to rw https://wikitech.wikimedia.org/wiki/Incidents/2023-02-22_read_only

Clement_Goubert closed this task as Resolved.Feb 27 2023, 2:36 PM
Clement_Goubert closed subtask T330302: sre.switchdc.mediawiki.03-set-db-readonly fails in live-test mode as Resolved.

All code paths exercised and fixes applied and tested. Resolving.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits