Page MenuHomePhabricator

Use EtcdConfig in production to allow automation of a datacenter switch
Open, MediumPublic

Description

The switchover of the master datacenter in mediawiki requires the following steps:

  1. Set the databases in the master datacenter in read-only mode
  2. Switch the wmfMasterDatacenter variable to the new datacenter
  3. Switch the databases in the new master datacenter

each one of these changes requires traditionally a git commit and a deploy. Our goal is to make switching easier, and - per T149617 - we've chosen to use Etcd to that end.

We have now EtcdConfig that can do that inside mediawiki-config, we just need to thoroughly test it and bring that to production.

Our objectives are thus as follows

  • Test EtcdConfig in different failure scenarios T185078
  • Prepare conftool for safely editing mediawiki-config values (additional validation, separate etcd credentials)
  • Deploy EtcdConfig to production
  • Allow use of EtcdConfig to configure slave databases (stretch) T185084

Event Timeline

Joe created this task.Dec 11 2017, 3:35 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 11 2017, 3:35 PM
Volans triaged this task as Medium priority.Dec 19 2017, 11:33 AM
Volans added a subscriber: Volans.
Joe renamed this task from [DRAFT] Use EtcdConfig in production to allow automation of a datacenter switch to Use EtcdConfig in production to allow automation of a datacenter switch.Jan 8 2018, 2:37 PM
Volans updated the task description. (Show Details)Jan 17 2018, 10:00 AM
elukey added a subscriber: elukey.Feb 12 2018, 4:21 PM

Change 411286 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[mediawiki/core@master] Expose the latest modified index seen by EtcdConfig

https://gerrit.wikimedia.org/r/411286

Change 411296 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/mediawiki-config@master] Enable EtcdConfig on the debug hosts

https://gerrit.wikimedia.org/r/411296

Change 411296 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable EtcdConfig on the debug hosts

https://gerrit.wikimedia.org/r/411296

Change 413355 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] Icinga: add sync check for MW config on etcd

https://gerrit.wikimedia.org/r/413355

Change 413356 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] Icinga: add EtcdConfig sync check on MW hosts

https://gerrit.wikimedia.org/r/413356

Change 411286 merged by jenkins-bot:
[mediawiki/core@master] Expose the latest modified index seen by EtcdConfig

https://gerrit.wikimedia.org/r/411286

Change 416411 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[mediawiki/core@wmf/1.31.0-wmf.23] Expose the latest modified index seen by EtcdConfig

https://gerrit.wikimedia.org/r/416411

Change 416470 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/mediawiki-config@master] Fetch the last modified index in etcd.php, and expose it via siteinfo.

https://gerrit.wikimedia.org/r/416470

Change 416482 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/mediawiki-config@master] Fetch data from etcd on every server

https://gerrit.wikimedia.org/r/416482

Change 416483 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/mediawiki-config@master] Enable use of EtcdConfig everywhere.

https://gerrit.wikimedia.org/r/416483

Change 416411 merged by Giuseppe Lavagetto:
[mediawiki/core@wmf/1.31.0-wmf.23] Expose the latest modified index seen by EtcdConfig

https://gerrit.wikimedia.org/r/416411

Change 416470 merged by jenkins-bot:
[operations/mediawiki-config@master] Fetch the last modified index in etcd.php, and expose it via siteinfo.

https://gerrit.wikimedia.org/r/416470

Change 416482 merged by jenkins-bot:
[operations/mediawiki-config@master] Fetch data from etcd on every server

https://gerrit.wikimedia.org/r/416482

Change 413355 merged by Volans:
[operations/puppet@production] Icinga: add sync check for MW config on etcd

https://gerrit.wikimedia.org/r/413355

Change 417232 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] Icinga: use systemd::service for etcd_mw_config

https://gerrit.wikimedia.org/r/417232

Change 417232 merged by Giuseppe Lavagetto:
[operations/puppet@production] Icinga: use systemd::service for etcd_mw_config

https://gerrit.wikimedia.org/r/417232

Change 413356 merged by Volans:
[operations/puppet@production] Icinga: add EtcdConfig sync check on MW hosts

https://gerrit.wikimedia.org/r/413356

Change 416483 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable use of EtcdConfig everywhere.

https://gerrit.wikimedia.org/r/416483

Joe updated the task description. (Show Details)Mar 8 2018, 3:32 PM
Joe closed subtask T188922: EtcdConfig: add Icinga check as Resolved.

The main part of the goal is now done. We will try to work on the stretch part if we get to it.

Marostegui updated the task description. (Show Details)Aug 21 2019, 10:57 AM