Change Details

sre.mysql.depool (and .pool) handle pcX and msX and would depool the section when a host is depooled. external store has RO sections and RW sections. RO sections aren't needed to be depooled but RW sections are easily entirely depooled from writes by running: ``` sudo dbctl --scope eqiad section es7 ro "Depooling" sudo dbctl --scope codfw section es7 ro "Depooling" sudo dbctl config commit -m "Master crashed" ``` That would allow the replicas to receive reads but no writes. How to find if it is a writable section may be the tricky part (as they all are RW on dbctl due to MW reasons, but RO ones aren't written) - for now maybe it can be just hardcoded (we spawn RW sections every 5 years or so). The writable ones are on https://noc.wikimedia.org/conf/highlight.php?file=db-production.php at the bottom but however, we do not automatically translate `cluster32` to esX anywhere other than the table names.

sre.mysql.depool (and .pool) handle pcX and msX should also work with esX external store has RO sections and RW sections and they have different cases: RO section: if a replica goes down, we simply depool via `dbctl instance esXX depool` if a master goes down, we simply should "switch" masters in dbctl (we obviously need to pick a replica within the same section): ``` dbctl -s $DC section esX set-master esXXXX dbctl instance esXXXX depool dbctl config commit -m "Your commit message XXX" ``` RW section: if a replica goes down, we simply depool via `dbctl instance esXX depool` if a master goes down, we can remove writes from that section (reads would remain unaffected) ``` sudo dbctl --scope eqiad section esX ro "Depooling" sudo dbctl --scope codfw section esX ro "Depooling" sudo dbctl config commit -m "Master esX crashed " ``` (Note: for RW sections we do have to run those two commands regardless of the active dc) That would allow the replicas to receive reads but no writes. How to find if it is a writable section may be the tricky part (as they all are RW on dbctl due to MW reasons, but RO ones aren't written) - for now maybe it can be just hardcoded (we spawn RW sections every 5 years or so). Currently the RO ones: es1 es2 es3 es4 es5 The writable ones are on https://noc.wikimedia.org/conf/highlight.php?file=db-production.php at the bottom but however, we do not automatically translate `cluster32` to esX anywhere other than the table names. At the moment they are: es6 es7

sre.mysql.depool (and .pool) handle pcX and msX and wshould depool the section when a host is depooled.also work with esX external store has RO sections and RW sections. and they have different cases: RO section: if a replica goes down, we simply depool via `dbctl instance esXX depool` RO sections aren't needed to be depooled but RW sections are easily entirely depooled from writes by running:if a master goes down, we simply should "switch" masters in dbctl (we obviously need to pick a replica within the same section): ``` sudo dbctl --scope eqiad-s $DC section es7 ro "Depooling"X set-master esXXXX sudo dbctl instance esXXXX depool dbctl --scope codfw section es7 ro "Depooling"config commit -m "Your commit message XXX" ``` RW section: sudo dbctl config commit -m "Master crashed"if a replica goes down, we simply depool via `dbctl instance esXX depool` if a master goes down, we can remove writes from that section (reads would remain unaffected) ``` sudo dbctl --scope eqiad section esX ro "Depooling" sudo dbctl --scope codfw section esX ro "Depooling" sudo dbctl config commit -m "Master esX crashed " ``` (Note: for RW sections we do have to run those two commands regardless of the active dc) That would allow the replicas to receive reads but no writes. How to find if it is a writable section may be the tricky part (as they all are RW on dbctl due to MW reasons, but RO ones aren't written) - for now maybe it can be just hardcoded (we spawn RW sections every 5 years or so). Currently the RO ones: es1 es2 es3 es4 es5 The writable ones are on https://noc.wikimedia.org/conf/highlight.php?file=db-production.php at the bottom but however, we do not automatically translate `cluster32` to esX anywhere other than the table names. At the moment they are: es6 es7