Page MenuHomePhabricator
Create Task
Maniphest T272963

cloudgw: develop HA setup
Closed, ResolvedPublic
Actions

Description

Introduce keepalived + conntrackd to the setup, document failover procedures.

This development is possible now that we have both cloudgw2001-dev (T271519) and cloudgw2002-dev (T271590) racked and ready to use.

Details

SubjectRepoBranchLines +/-
cloudgw: refresh conntrackd service dependenciesoperations/puppetproduction+2 -0
cloudgw: keepalived: use nopreempt optionoperations/puppetproduction+2 -0
cloudgw: set up conntrack sysctl parametersoperations/puppetproduction+32 -4
cloudgw: let keepalived track additional static routesoperations/puppetproduction+15 -5
cloudgw: use address per interface in the cloud-instance-transport subnetoperations/puppetproduction+26 -13
openstack: cloudgw: allow incoming conntrackd TCP connectionoperations/puppetproduction+5 -0
conntrackd: also install the conntrack tooloperations/puppetproduction+6 -1
cloudgw: interfaces: relax check on routing setup by using 'onlink'operations/puppetproduction+3 -3
cloudgw: switch data place interface config modes to manualoperations/puppetproduction+2 -2
cloudgw: stop setting up VIP addresses that are now handle via keepalived/VRRPoperations/puppetproduction+2 -24
Revert "cloud: hiera: enable back neutron hacks in codfw1dev"operations/puppetproduction+1 -1
Revert "cloud: hiera: connect cloudnet servers back to vlan 2120"operations/puppetproduction+1 -1
Revert "cloud: hiera: add vlan 2120 back into the neutron bridge"operations/puppetproduction+1 -1
cloudgw: introduce HA by using keepalived/VRRPoperations/puppetproduction+172 -1
cloudgw: move common hiera into proper fileoperations/puppetproduction+82 -164
keepalived: add support for custom templateoperations/puppetproduction+9 -1
cloudgw2002-dev: give it proper puppet roleoperations/puppetproduction+115 -6
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

aborrero triaged this task as Medium priority.Jan 26 2021, 10:33 AM
aborrero created this task.
aborrero moved this task from Inbox to Soon! on the cloud-services-team (Kanban) board.
gerritbot added a comment.Feb 12 2021, 10:53 AM

Change 663799 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudgw2002-dev: give it proper puppet role

https://gerrit.wikimedia.org/r/663799

gerritbot added a project: Patch-For-Review.Feb 12 2021, 10:53 AM

Change 663799 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudgw2002-dev: give it proper puppet role

https://gerrit.wikimedia.org/r/663799

ops-monitoring-bot added a comment.Feb 12 2021, 10:57 AM

Script wmf-auto-reimage was launched by aborrero on cumin2001.codfw.wmnet for hosts:

cloudgw2002-dev.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202102121057_aborrero_28819_cloudgw2002-dev_codfw_wmnet.log.

Maintenance_bot removed a project: Patch-For-Review.Feb 12 2021, 11:10 AM
ops-monitoring-bot added a comment.Feb 12 2021, 11:22 AM

Completed auto-reimage of hosts:

['cloudgw2002-dev.codfw.wmnet']

and were ALL successful.

gerritbot added a comment.Feb 12 2021, 11:22 AM

Change 663801 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] keepalived: add support for custom template

https://gerrit.wikimedia.org/r/663801

gerritbot added a project: Patch-For-Review.Feb 12 2021, 11:22 AM
gerritbot added a comment.Feb 12 2021, 12:02 PM

Change 663801 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] keepalived: add support for custom template

https://gerrit.wikimedia.org/r/663801

Maintenance_bot removed a project: Patch-For-Review.Feb 12 2021, 12:10 PM
gerritbot added a comment.Feb 12 2021, 1:00 PM

Change 663823 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudgw: introduce HA by using keepalived/VRRP

https://gerrit.wikimedia.org/r/663823

gerritbot added a project: Patch-For-Review.Feb 12 2021, 1:00 PM
gerritbot added a comment.Feb 15 2021, 11:32 AM

Change 664241 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudgw: move common hiera into proper file

https://gerrit.wikimedia.org/r/664241

gerritbot added a comment.Feb 15 2021, 11:33 AM

Change 664241 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudgw: move common hiera into proper file

https://gerrit.wikimedia.org/r/664241

gerritbot added a comment.Feb 15 2021, 12:49 PM

Change 663823 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudgw: introduce HA by using keepalived/VRRP

https://gerrit.wikimedia.org/r/663823

Stashbot added a comment.Feb 15 2021, 3:45 PM

Mentioned in SAL (#wikimedia-cloud) [2021-02-15T15:45:29Z] <arturo> [codfw1dev] connect virtual router cloudinstances2b-gw to vlan cloud-gw-transport-codfw (185.15.57.10) (T272963)

Stashbot added a comment.Feb 15 2021, 3:45 PM

Mentioned in SAL (#wikimedia-cloud) [2021-02-15T15:45:54Z] <arturo> [codfw1dev] drop subnet definition for cloud-instances-transport1-b-codfw (T272963)

gerritbot added a comment.Feb 15 2021, 3:47 PM

Change 664255 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] Revert "cloud: hiera: add vlan 2120 back into the neutron bridge"

https://gerrit.wikimedia.org/r/664255

gerritbot added a comment.Feb 15 2021, 3:49 PM

Change 664255 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] Revert "cloud: hiera: add vlan 2120 back into the neutron bridge"

https://gerrit.wikimedia.org/r/664255

gerritbot added a comment.Feb 15 2021, 3:51 PM

Change 664256 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] Revert "cloud: hiera: connect cloudnet servers back to vlan 2120"

https://gerrit.wikimedia.org/r/664256

gerritbot added a comment.Feb 15 2021, 3:53 PM

Change 664257 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] Revert "cloud: hiera: enable back neutron hacks in codfw1dev"

https://gerrit.wikimedia.org/r/664257

gerritbot added a comment.Feb 15 2021, 3:53 PM

Change 664256 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] Revert "cloud: hiera: connect cloudnet servers back to vlan 2120"

https://gerrit.wikimedia.org/r/664256

gerritbot added a comment.Feb 15 2021, 3:58 PM

Change 664257 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] Revert "cloud: hiera: enable back neutron hacks in codfw1dev"

https://gerrit.wikimedia.org/r/664257

gerritbot added a comment.Feb 15 2021, 4:02 PM

Change 664257 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] Revert "cloud: hiera: enable back neutron hacks in codfw1dev"

https://gerrit.wikimedia.org/r/664257

gerritbot added a comment.Feb 15 2021, 4:11 PM

Change 664307 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudgw: stop setting up VIP addresses that are now handle via keepalived/VRRP

https://gerrit.wikimedia.org/r/664307

gerritbot added a comment.Feb 15 2021, 4:18 PM

Change 664307 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudgw: stop setting up VIP addresses that are now handle via keepalived/VRRP

https://gerrit.wikimedia.org/r/664307

Stashbot added a comment.Feb 15 2021, 4:25 PM

Mentioned in SAL (#wikimedia-cloud) [2021-02-15T16:25:24Z] <arturo> [codfw1dev] rebooting all cloudgw200x-dev / cloudnet200x-dev servers (T272963)

gerritbot added a comment.Feb 15 2021, 4:49 PM

Change 664311 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudgw: switch data place interface config modes to manual

https://gerrit.wikimedia.org/r/664311

gerritbot added a comment.Feb 15 2021, 4:52 PM

Change 664311 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudgw: switch data place interface config modes to manual

https://gerrit.wikimedia.org/r/664311

gerritbot added a comment.Feb 15 2021, 5:55 PM

Change 664317 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudgw: interfaces: relax check on routing setup by using 'onlink'

https://gerrit.wikimedia.org/r/664317

gerritbot added a comment.Feb 15 2021, 5:57 PM

Change 664317 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudgw: interfaces: relax check on routing setup by using 'onlink'

https://gerrit.wikimedia.org/r/664317

gerritbot added a comment.Feb 16 2021, 10:20 AM

Change 664521 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] conntrackd: also install the conntrack tool

https://gerrit.wikimedia.org/r/664521

gerritbot added a comment.Feb 16 2021, 10:22 AM

Change 664521 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] conntrackd: also install the conntrack tool

https://gerrit.wikimedia.org/r/664521

gerritbot added a comment.Feb 16 2021, 11:14 AM

Change 664538 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: cloudgw: allow incoming conntrackd TCP connection

https://gerrit.wikimedia.org/r/664538

gerritbot added a comment.Feb 16 2021, 11:18 AM

Change 664538 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: cloudgw: allow incoming conntrackd TCP connection

https://gerrit.wikimedia.org/r/664538

gerritbot added a comment.Feb 16 2021, 12:23 PM

Change 664549 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudgw: use address per interface in the cloud-instance-transport subnet

https://gerrit.wikimedia.org/r/664549

gerritbot added a comment.Feb 16 2021, 12:30 PM

Change 664549 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudgw: use address per interface in the cloud-instance-transport subnet

https://gerrit.wikimedia.org/r/664549

gerritbot added a comment.Feb 16 2021, 5:28 PM

Change 664603 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudgw: let keepalived track static routes

https://gerrit.wikimedia.org/r/664603

gerritbot added a comment.Feb 16 2021, 6:16 PM

Change 664603 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudgw: let keepalived track additional static routes

https://gerrit.wikimedia.org/r/664603

gerritbot added a comment.Feb 17 2021, 10:23 AM

Change 664785 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudgw: set up conntrack sysctl parameters

https://gerrit.wikimedia.org/r/664785

gerritbot added a comment.Feb 17 2021, 10:25 AM

Change 664785 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudgw: set up conntrack sysctl parameters

https://gerrit.wikimedia.org/r/664785

gerritbot added a comment.Feb 17 2021, 10:50 AM

Change 664789 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudgw: keepalived: use nopreempt option

https://gerrit.wikimedia.org/r/664789

gerritbot added a comment.Feb 17 2021, 10:54 AM

Change 664789 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudgw: keepalived: use nopreempt option

https://gerrit.wikimedia.org/r/664789

gerritbot added a comment.Feb 17 2021, 12:01 PM

Change 664800 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudgw: start conntrackd before keepalived

https://gerrit.wikimedia.org/r/664800

gerritbot added a comment.Feb 17 2021, 12:18 PM

Change 664800 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudgw: refresh conntrackd service dependencies

https://gerrit.wikimedia.org/r/664800

aborrero added a comment.Feb 17 2021, 12:37 PM

This is in very good shape. I tested several failover scenarios:

  • manually stop keepalived in the primary VRRP node
  • reboot of the primary VRRP node
  • flapping (backup -> primary -> backup -> primary)

How I tested this:

  • ssh tools-codfw1dev-k8s-worker-1.tools-codfw1dev.codfw1dev.wikimedia.cloud
  • aborrero@tools-codfw1dev-k8s-worker-1:~$ wget https://network-tests.toolforge.org/files/1GB.bin -O /dev/null
  • ssh cloudgw2001-dev.codfw.wmnet --> reboot if primary
  • ssh cloudgw2002-dev.codfw.wmnet --> if new primary, watch traffic flowing
  • watch wget download still flowing despite several failovers
aborrero closed this task as Resolved.Feb 25 2021, 3:34 PM
aborrero mentioned this in T294956: Avoid unnecessary keepalived flap after rebooting servers.Nov 3 2021, 6:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL