Change Details

At Thu Nov 16 17:15 server db1063, master of the s5 shard of databases (dewiki and wikidata) started to have storage crashes (T180714#3767308) **Update Fri Nov 17 07:55:06 UTC 2017:** Pending things: [] Revert schema change on codfw [ONGOING] [] move dbstore1001 under the new master (db1070) [ONGOING] [] revert schema change on dbstore1001 [x] move dbstore1002 under the new master (db1070) [DONE] [x] revert schema change on dbstore1002 [ONGOING] [] decide what to do with db1063 (definitely not a master for anything else, we might rebuild it as vslow from db1071 for example) as it is broken: https://phabricator.wikimedia.org/P6337 [x] db1071 moved under db1070 IR document: https://wikitech.wikimedia.org/wiki/Incident_documentation/20171116-s5-dewiki-wikidata

At Thu Nov 16 17:15 server db1063, master of the s5 shard of databases (dewiki and wikidata) started to have storage crashes (T180714#3767308) **Update Fri Nov 17 07:55:06 UTC 2017:** Pending things: [] Revert schema change on codfw [ONGOING] [x] move dbstore1001 under the new master (db1070) [DONE] [] revert schema change on dbstore1001 [ONGOING] [x] move dbstore1002 under the new master (db1070) [DONE] [x] revert schema change on dbstore1002 [ONGOING] [] decide what to do with db1063 (definitely not a master for anything else, we might rebuild it as vslow from db1071 for example) as it is broken: https://phabricator.wikimedia.org/P6337 [x] db1071 moved under db1070 IR document: https://wikitech.wikimedia.org/wiki/Incident_documentation/20171116-s5-dewiki-wikidata

At Thu Nov 16 17:15 server db1063, master of the s5 shard of databases (dewiki and wikidata) started to have storage crashes (T180714#3767308) **Update Fri Nov 17 07:55:06 UTC 2017:** Pending things: [] Revert schema change on codfw [ONGOING] [x] move dbstore1001 under the new master (db1070) [ONGOINGDONE] [] revert schema change on dbstore1001 [ONGOING] [x] move dbstore1002 under the new master (db1070) [DONE] [x] revert schema change on dbstore1002 [ONGOING] [] decide what to do with db1063 (definitely not a master for anything else, we might rebuild it as vslow from db1071 for example) as it is broken: https://phabricator.wikimedia.org/P6337 [x] db1071 moved under db1070 IR document: https://wikitech.wikimedia.org/wiki/Incident_documentation/20171116-s5-dewiki-wikidata