Ping T239188: Decommission db1062.eqiad.wmnet

In T239188#5708209, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2019-12-03T08:29:15Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db1062 from config T239188 (duration: 01m 08s)

I guess that didn't quite work as intended.

According to https://noc.wikimedia.org/db.php, db1062 is still part of s7.

Also, less frequently, I also see MW errors in Logstash about two other s7 hosts:

[{exception_id}] {exception_url} Wikimedia\Rdbms\DBConnectionError from line 1415 of /srv/mediawiki/php-1.35.0-wmf.8/includes/libs/rdbms/loadbalancer/LoadBalancer.php:
Cannot access the database: Unknown error (10.64.48.109)

10.64.48.109 is the IP of host db1136 (also part of s7)

[{exception_id}] {exception_url} Wikimedia\Rdbms\DBConnectionError from line 1415 of /srv/mediawiki/php-1.35.0-wmf.8/includes/libs/rdbms/loadbalancer/LoadBalancer.php:
Cannot access the database: Unknown error (10.64.48.153)

Where is the IP of host db1094 (also part of s7).

That seems rather concerning, one s7 host is fully unreachable and two others are (sometimes) unreachable?

From https://noc.wikimedia.org/db.php#tabs-8 and https://noc.wikimedia.org/dbconfig/eqiad.json:

db1086: 0
db1062: 0

"s7": [
  {
    "db1086": 0
  },
  {
    "db1062": 0,
    "db1079": 200,
    "db1090:3317": 1,
    "db1094": 500,
    "db1098:3317": 150,
    "db1101:3317": 150,
    "db1136": 400
  }
],

Two hosts that both claim to be the master / have 0 read load assigned. The first one is correct but the second one presumably isn't meant to be there. I don't actually know if it's valid to have a slave with 0 load. I guess that's for a follow-up to reject in the dbctl schema to prevent future issues so that one is forced to either remove it or assign at least 1 load (which is effectively the same as 0 given that 0 doesn't mean no interaction at all per the current issue). Having said that, I also don't know under which circumstances (aside from replag wait) a 0-load slave would be used. Perhaps if others are lagged / unused, which is quite likely given it is the only host with no other read queries so it might win regularly when weighing load against least-lagged meta data. To be continued :)

It seems clear that db1062 shouldn't be pooled anywhere. Ran the dbctl depool utility and it's gone from s7.

Thanks for tackling this and apologies for not depooling it when I should've.
db1062 stopped being a master a week ago

We normally depool it straightaway (that is removing it from the list of hosts, not leaving it with weight 0) but looks like I forgot to do so at that same moment.
Yesterday, as part of the decommissioning process, I stopped MySQL and remove the host from MW config: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/554243/
However, as we normally have depooled it from dbctl, I assumed it was depooled, but it wasn't.

I have amended our master switchover checklists, to reflect the fact that it needs to be depooled from dbctl (if it will go under maintenance/decommissioning) so this doesn't happen again.

Thank you all for solving this and sorry again for the inconveniences.