As of now, a new DNS address of osm.db.svc.eqiad.wmflabs is pointed at clouddb1003 which is a read replica of osmdb.eqiad.wmnet.
It should work for read-only loads already for all intents and purposes. Now we need to cut over in order to decommission the old hardware.
This is currently scheduled to take place on 20190404@1700 UTC
The plan looks like this:
- Stage DNS change so that osmdb.eqiad.wmnet will be a CNAME pointing at osm.db.svc.eqiad.wmflabs
- Announce the plan to switch over with impact to a couple tools (notably those owned by @Kolossos and possibly @aude, which have some read-write access) from DNS changing to a read replica temporarily -- with a few days lead time so most people see it
- Switch DNS with a merge and update
- <wait for TTL, which is 5min>
- Stop postgres on master, which will no longer be in use.
- Touch trigger file on clouddb1003, and ensure postgres is now running as the rw primary
- Announce the change with impact is over.
- Switch the puppet role to make this server the primary for purposes of sync jobs, etc.
- Start work on bringing up the new replica on clouddb1004 <-- At this point we are ready to decom the old servers
Will try some quick tests to ensure no network issues before proceeding. As of now for read-access osm.db.svc.eqiad.wmflabs should work as a read replica and copy of osmdb.eqiad.wmnet (if folks on the maps project or similar want to test that theory -- @Awjrichards @Chippyy @cmarqu @dschwen @jeremyb @MaxSem @Multichill @Nosy @TheDJ -- since I think that project uses osmdb).