Page MenuHomePhabricator

Migrate nova proxies to eqiad1
Closed, ResolvedPublic

Description

Currently this project is puppetized on wikitech via https://wikitech.wikimedia.org/wiki/Hiera:Project-proxy -- I'm going to fix that /after/ this fail-over is done.

The API service mentioned below is a uwsgi service called 'invisible_unicorn'.

These steps will not result in downtime:

  • Create new eqiad1 proxy nodes, proxy-01 and proxy-02
  • copy certs over by hand from nova-proxy-01
  • Add proxy-01 and proxy-02 to $all_proxies, let puppet update
  • ensure that redis is syncing properly between regions
  • Update proxy DNS record for a test proxy, ensure that proxy-01 handles it correctly
  • Update proxy DNS records to point to the eqiad1 proxy (proxy-01)
  • test some more
  • update hieradata/eqiad/profile/openstack/main/nova/network.yaml with the new active proxy IP

These steps will result in partial downtime with creating/deleting proxies:

  • Set $active_proxy to point to proxy-01, let puppet update
  • stop puppet and the API on novaproxy-01
  • stop api on proxy-01, restore database (it's on NFS, available to all nodes), restart API there
  • Update proxy endpoints in keystone to point to the new proxy
  • Test!

Cleanup:

  • move project-wide puppet off of wikitech and into horizon
  • Wait 24 hours for DNS caches to update
  • Shut down novaproxy-01 and novaproxy-02
  • Wait another few days before deleting

Event Timeline

Andrew created this task.Jan 11 2019, 2:04 PM
Andrew triaged this task as Normal priority.
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptJan 11 2019, 2:04 PM
Andrew updated the task description. (Show Details)Jan 11 2019, 2:09 PM
Andrew updated the task description. (Show Details)Jan 11 2019, 3:35 PM
Andrew updated the task description. (Show Details)Jan 11 2019, 3:44 PM
Andrew updated the task description. (Show Details)Jan 11 2019, 4:11 PM

Change 483793 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] dynamicproxy api: include sqlite3 package

https://gerrit.wikimedia.org/r/483793

Change 483794 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add new nova proxy IPs to some firewall defs

https://gerrit.wikimedia.org/r/483794

Change 483793 merged by Andrew Bogott:
[operations/puppet@production] dynamicproxy api: include sqlite3 package

https://gerrit.wikimedia.org/r/483793

Change 483794 merged by Andrew Bogott:
[operations/puppet@production] Add new nova proxy IPs to some firewall defs

https://gerrit.wikimedia.org/r/483794

Andrew updated the task description. (Show Details)Jan 11 2019, 4:52 PM
Andrew updated the task description. (Show Details)Jan 11 2019, 4:55 PM
Andrew updated the task description. (Show Details)Jan 12 2019, 1:46 AM

Change 483902 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmcs: add a script to update VPS proxies

https://gerrit.wikimedia.org/r/483902

Change 483902 merged by Andrew Bogott:
[operations/puppet@production] wmcs: add a script to update VPS proxies

https://gerrit.wikimedia.org/r/483902

Mentioned in SAL (#wikimedia-cloud) [2019-01-13T17:11:33Z] <andrewbogott> moving all VPS dynamic proxies to proxy-eqiad1.wmflabs.org aka proxy-01.project-proxy.eqiad.wmflabs, as per T213540

Change 484024 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Update IPs for the new VPS proxy host

https://gerrit.wikimedia.org/r/484024

Change 484024 merged by Andrew Bogott:
[operations/puppet@production] Update IPs for the new VPS proxy host

https://gerrit.wikimedia.org/r/484024

Andrew updated the task description. (Show Details)Jan 13 2019, 7:07 PM

Mentioned in SAL (#wikimedia-cloud) [2019-01-13T19:12:34Z] <andrewbogott> moving the VPS proxy API backend to proxy-01.project-proxy.eqiad.wmflabs, as per T213540

Andrew updated the task description. (Show Details)Jan 13 2019, 7:50 PM
Andrew updated the task description. (Show Details)Jan 13 2019, 7:59 PM

Mentioned in SAL (#wikimedia-cloud) [2019-01-13T20:00:00Z] <andrewbogott> VPS proxies are now running in eqiad1 on proxy-01. Old VMs will wait a bit for deletion. T213540

Mentioned in SAL (#wikimedia-cloud) [2019-01-16T14:21:04Z] <andrewbogott> stopping old VPS proxies in eqiad — T213540

Mentioned in SAL (#wikimedia-cloud) [2019-01-16T14:21:16Z] <andrewbogott> stopping old VPS proxies in eqiad — T213540

Andrew updated the task description. (Show Details)Jan 16 2019, 2:22 PM
Andrew closed this task as Resolved.Jan 23 2019, 7:42 PM
Andrew updated the task description. (Show Details)