sre.mysql.depool (and .pool) handle pcX and msX should also work with esX
external store has RO sections and RW sections and they have different cases:
RO section:
if a replica goes down, we simply depool via dbctl instance esXX depool
if a master goes down, we simply should "switch" masters in dbctl (we obviously need to pick a replica within the same section):
dbctl -s $DC section esX set-master esXXXX dbctl instance esXXXX depool dbctl config commit -m "Your commit message XXX"
RW section:
if a replica goes down, we simply depool via dbctl instance esXX depool
if a master goes down, we can remove writes from that section (reads would remain unaffected)
sudo dbctl --scope eqiad section esX ro "Depooling" sudo dbctl --scope codfw section esX ro "Depooling" sudo dbctl config commit -m "Master esX crashed "
(Note: for RW sections we do have to run those two commands regardless of the active dc)
That would allow the replicas to receive reads but no writes.
How to find if it is a writable section may be the tricky part (as they all are RW on dbctl due to MW reasons, but RO ones aren't written) - for now maybe it can be just hardcoded (we spawn RW sections every 5 years or so).
Currently the RO ones: es1 es2 es3 es4 es5
The writable ones are on https://noc.wikimedia.org/conf/highlight.php?file=db-production.php at the bottom but however, we do not automatically translate cluster32 to esX anywhere other than the table names.
At the moment they are: es6 es7