Change Details

As of now, a new DNS address of osm.db.svc.eqiad.wmflabs is pointed at clouddb1003 which is a read replica of osmdb.eqiad.wmnet. It should work for read-only loads already for all intents and purposes. Now we need to cut over in order to decommission the old hardware. The plan looks like this: - Stage DNS change so that osmdb.eqiad.wmnet will be a CNAME pointing at osm.db.svc.eqiad.wmflabs - Announce the plan to switch over with impact to a couple tools (notably those owned by @Kolossos and possibly @aude, which have some read-write access) from DNS changing to a read replica temporarily -- with a few days lead time so most people see it - Switch DNS with a merge and update - <wait for TTL, which is 5min> - Stop postgres on master, which will no longer be in use. - Touch trigger file on clouddb1003, and ensure postgres is now running as the rw primary - Announce the change with impact is over. - Switch the puppet role to make this server the primary for purposes of sync jobs, etc. - Start work on bringing up the new replica on clouddb1004 <-- At this point we are ready to decom the old servers Will try some quick tests to ensure no network issues before proceeding. As of now for read-access osm.db.svc.eqiad.wmflabs should work as a read replica and copy of osmdb.eqiad.wmnet (if folks on the maps project or similar want to test that theory -- @Awjrichards @Chippyy @cmarqu @dschwen @jeremyb @MaxSem @Multichill @Nosy @TheDJ -- since I think that project uses osmdb).

As of now, a new DNS address of osm.db.svc.eqiad.wmflabs is pointed at clouddb1003 which is a read replica of osmdb.eqiad.wmnet. It should work for read-only loads already for all intents and purposes. Now we need to cut over in order to decommission the old hardware. This is currently scheduled to take place on 20190404 The plan looks like this: - Stage DNS change so that osmdb.eqiad.wmnet will be a CNAME pointing at osm.db.svc.eqiad.wmflabs - Announce the plan to switch over with impact to a couple tools (notably those owned by @Kolossos and possibly @aude, which have some read-write access) from DNS changing to a read replica temporarily -- with a few days lead time so most people see it - Switch DNS with a merge and update - <wait for TTL, which is 5min> - Stop postgres on master, which will no longer be in use. - Touch trigger file on clouddb1003, and ensure postgres is now running as the rw primary - Announce the change with impact is over. - Switch the puppet role to make this server the primary for purposes of sync jobs, etc. - Start work on bringing up the new replica on clouddb1004 <-- At this point we are ready to decom the old servers Will try some quick tests to ensure no network issues before proceeding. As of now for read-access osm.db.svc.eqiad.wmflabs should work as a read replica and copy of osmdb.eqiad.wmnet (if folks on the maps project or similar want to test that theory -- @Awjrichards @Chippyy @cmarqu @dschwen @jeremyb @MaxSem @Multichill @Nosy @TheDJ -- since I think that project uses osmdb).

As of now, a new DNS address of osm.db.svc.eqiad.wmflabs is pointed at clouddb1003 which is a read replica of osmdb.eqiad.wmnet. It should work for read-only loads already for all intents and purposes. Now we need to cut over in order to decommission the old hardware. This is currently scheduled to take place on 20190404 The plan looks like this: - Stage DNS change so that osmdb.eqiad.wmnet will be a CNAME pointing at osm.db.svc.eqiad.wmflabs - Announce the plan to switch over with impact to a couple tools (notably those owned by @Kolossos and possibly @aude, which have some read-write access) from DNS changing to a read replica temporarily -- with a few days lead time so most people see it - Switch DNS with a merge and update - <wait for TTL, which is 5min> - Stop postgres on master, which will no longer be in use. - Touch trigger file on clouddb1003, and ensure postgres is now running as the rw primary - Announce the change with impact is over. - Switch the puppet role to make this server the primary for purposes of sync jobs, etc. - Start work on bringing up the new replica on clouddb1004 <-- At this point we are ready to decom the old servers Will try some quick tests to ensure no network issues before proceeding. As of now for read-access osm.db.svc.eqiad.wmflabs should work as a read replica and copy of osmdb.eqiad.wmnet (if folks on the maps project or similar want to test that theory -- @Awjrichards @Chippyy @cmarqu @dschwen @jeremyb @MaxSem @Multichill @Nosy @TheDJ -- since I think that project uses osmdb).