Page MenuHomePhabricator
Create Task
Maniphest T205814

Switch the main etcd cluster in eqiad to use conf1004-1006
Closed, ResolvedPublic
Actions

Description

Our old servers (conf1001-1003) are old and out of warranty, so we want to switch etcd over to the new cluster.

The servers conf1004-1006 have etcd3; we will still use the v2 store for now though.

The plan is as follows:

  1. Start replicating from the codfw cluster to the new etcd cluster in eqiad using etcdmirror
  2. Fail over write and read traffic from eqiad to codfw (see https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster#Master_cluster_switchover)
  3. Test that applications can work with the new software (possibly one by one)
  4. [Optional] Switchover codfw => eqiad and upgrade the codfw cluster to stretch too.

The first 3 points in this ticket would be better done *before* we switch back on the eqiad dc for mediawiki and traffic.

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Joe created this task.Oct 1 2018, 6:33 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 1 2018, 6:33 AM
Joe added a comment.Oct 2 2018, 2:06 PM

More detailed plan:

  • Replicate from the codfw etcd cluster to the new etcd cluster in eqiad. This means setting up an instance of profile::etcd::replication as active on one of the new hosts. Given the ports used are different on the new cluster, we will need to refactor the class a bit
  • Set etcd as read-only. This is achieved by setting profile::etcd::tlsproxy::read_only to true in puppet.
  • Move the client r-w records ( _etcd._tcp.conftool and _etcd._tcp) in DNS to point to the codfw cluster globally
  • Stop replicating eqiad old => codfw. This is done by changing a setting in hiera profile::etcd::replication::active to false for conf2002
  • Revert etcd read-only
  • Move selected applications in eqiad (stil inactive!) to read from the new cluster. I would suggest starting with pybal, then with any place where we use confd.
  • Move eqiad, esams to read from eqiad's etcd
  • Decom the old eqiad cluster
  • Once all is done, repeat the same procedure to move writes back to eqiad, and possibly upgrade the conf2* servers to stretch.
gerritbot subscribed.Oct 3 2018, 11:16 AM

Change 464128 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::etcd::replication: allow the config of src/dst repl ports

https://gerrit.wikimedia.org/r/464128

gerritbot added a project: Patch-For-Review.Oct 3 2018, 11:16 AM
gerritbot added a comment.Oct 3 2018, 4:42 PM

Change 464128 merged by Elukey:
[operations/puppet@production] profile::etcd::replication: allow the config of src/dst repl ports

https://gerrit.wikimedia.org/r/464128

gerritbot added a comment.Oct 4 2018, 8:45 AM

Change 464492 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::configcluster_stretch: enable etcd replication

https://gerrit.wikimedia.org/r/464492

gerritbot added a comment.Oct 4 2018, 10:06 AM

Change 464503 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/dns@master] Move _etcd._tcp* SRV records to etcd codfw

https://gerrit.wikimedia.org/r/464503

gerritbot added a comment.Oct 4 2018, 10:29 AM

Change 464507 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/software/etcd-mirror@debian] Repackaging for stretch

https://gerrit.wikimedia.org/r/464507

Stashbot subscribed.Oct 4 2018, 10:36 AM

Mentioned in SAL (#wikimedia-operations) [2018-10-04T10:36:32Z] <_joe_> uploading etcd-mirror to stretch-wikimedia T205814

gerritbot added a comment.Oct 4 2018, 12:23 PM

Change 464492 merged by Elukey:
[operations/puppet@production] role::configcluster_stretch: enable etcd replication

https://gerrit.wikimedia.org/r/464492

Stashbot added a comment.Oct 4 2018, 12:23 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-04T12:23:56Z] <elukey> deploy etcdmirror on conf1005 - T205814

gerritbot added a comment.Oct 5 2018, 8:53 AM

Change 464503 merged by Elukey:
[operations/dns@master] Move _etcd._tcp* SRV records to etcd codfw

https://gerrit.wikimedia.org/r/464503

gerritbot added a comment.Oct 5 2018, 9:00 AM

Change 464772 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Stop replicating etcd data from conf100[1-3] to codfw

https://gerrit.wikimedia.org/r/464772

gerritbot added a comment.Oct 5 2018, 9:02 AM

Change 464772 merged by Elukey:
[operations/puppet@production] Stop replicating etcd data from conf100[1-3] to codfw

https://gerrit.wikimedia.org/r/464772

gerritbot added a comment.Oct 5 2018, 9:20 AM

Change 464779 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] pybal: stop reading from the old etcd cluster

https://gerrit.wikimedia.org/r/464779

gerritbot added a comment.Oct 5 2018, 9:22 AM

Change 464779 merged by Giuseppe Lavagetto:
[operations/puppet@production] pybal: stop reading from the old etcd cluster

https://gerrit.wikimedia.org/r/464779

gerritbot added a comment.Oct 5 2018, 5:07 PM

Change 464847 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::configcluster: set codfw to read/write and eqiad to readonly

https://gerrit.wikimedia.org/r/464847

gerritbot added a comment.Oct 5 2018, 5:09 PM

Change 464847 merged by Elukey:
[operations/puppet@production] role::configcluster: set codfw to read/write and eqiad to readonly

https://gerrit.wikimedia.org/r/464847

gerritbot added a comment.Oct 8 2018, 6:09 AM

Change 464507 merged by Giuseppe Lavagetto:
[operations/software/etcd-mirror@debian] Repackaging for stretch

https://gerrit.wikimedia.org/r/464507

gerritbot added a comment.Oct 8 2018, 10:23 AM

Change 465138 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] lvs: switch eqiad/esams to the new eqiad etcd cluster

https://gerrit.wikimedia.org/r/465138

gerritbot added a comment.Oct 8 2018, 10:28 AM

Change 465138 merged by Giuseppe Lavagetto:
[operations/puppet@production] lvs: switch eqiad/esams to the new eqiad etcd cluster

https://gerrit.wikimedia.org/r/465138

gerritbot added a comment.Oct 8 2018, 10:56 AM

Change 465142 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] pybal: support etcd connections to ports different from 2379

https://gerrit.wikimedia.org/r/465142

gerritbot added a comment.Oct 8 2018, 11:06 AM

Change 465143 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/dns@master] Move _etcd* SRV records to conf100[4-6] where needed

https://gerrit.wikimedia.org/r/465143

gerritbot added a comment.Oct 8 2018, 11:43 AM

Change 465142 merged by Giuseppe Lavagetto:
[operations/puppet@production] pybal: support etcd connections to ports different from 2379

https://gerrit.wikimedia.org/r/465142

gerritbot added a comment.Oct 8 2018, 1:07 PM

Change 465143 merged by Elukey:
[operations/dns@master] Move _etcd* SRV RO records to conf100[4-6] where needed

https://gerrit.wikimedia.org/r/465143

gerritbot added a comment.Oct 9 2018, 11:05 AM

Change 465389 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] [WIP] Decommission conf100[1-3]

https://gerrit.wikimedia.org/r/465389

gerritbot added a comment.Oct 10 2018, 12:08 PM

Change 465389 merged by Elukey:
[operations/puppet@production] Decommission conf100[1-3]

https://gerrit.wikimedia.org/r/465389

elukey subscribed.Oct 10 2018, 12:42 PM

Opened https://phabricator.wikimedia.org/T206626 to fully decom conf100[1-3] (not in service anymore and with role::spare::system).

The last step is to switch back to eqiad as RW (currently set to codfw) after the MW switchover happens (modifying etcdmirror accordingly).

jijiki triaged this task as Medium priority.Oct 26 2018, 10:15 AM
elukey closed this task as Resolved.Oct 26 2018, 1:16 PM
elukey claimed this task.

This has been done, closing!

RobH closed subtask T206626: Decommission conf100[1-3] as Resolved.Mar 28 2019, 9:00 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits