Change 828677 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[operations/puppet@production] Multi-DC: go back to testwiki only

https://gerrit.wikimedia.org/r/828677

As a followup to this, here or on a a separate tasks, it would be wise to reevaluate paging strategy for x2- so if something doesn't create an outage anymore, removing it from paging (not sure if just the replicas or also the primaries- all depends on the tests results, I guess). Currently it pages for everything, which I believe it is wise in the status during T315274, but might be able to be relaxed if tests work as expected.

Here is a few scenarios that I can think of that could be tested: iptables with REJECT, iptables with DROP, regular lag, replication breaking, high query latency/max_connections overload.

Yeah, the replicas will definitely have notifications disabled.

The dbctl patch (https://gerrit.wikimedia.org/r/c/operations/software/conftool/+/828606) was merged yesterday and x2 is now running with the new flag

root@cumin1001:~#  dbctl -s codfw section x2 get
{
    "tags": "datacenter=codfw",
    "x2": {
        "flavor": "external",
        "master": "db2142",
        "min_replicas": 0,
        "omit_replicas_in_mwconfig": true,
        "readonly": false,
        "ro_reason": "test"
    }
}
root@cumin1001:~#  dbctl -s eqiad section x2 get
{
    "tags": "datacenter=eqiad",
    "x2": {
        "flavor": "external",
        "master": "db1151",
        "min_replicas": 1,
        "omit_replicas_in_mwconfig": true,
        "readonly": false,
        "ro_reason": "test"
    }
}
root@cumin1001:~#

So the replicas aren't supposed to be used from dbctl and this can be visible here: https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-job=All&var-server=db1152&var-port=9104&from=1661993853796&to=1662069003226

Change 828677 merged by Tim Starling:

[operations/puppet@production] Multi-DC: go back to testwiki only

https://gerrit.wikimedia.org/r/828677

Mentioned in SAL (#wikimedia-operations) [2022-09-05T11:15:57Z] <tstarling@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db[2142-2144].codfw.wmnet with reason: T316847 x2 failure test

Mentioned in SAL (#wikimedia-operations) [2022-09-05T11:16:12Z] <tstarling@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db[2142-2144].codfw.wmnet with reason: T316847 x2 failure test

There are log entries like

2022-09-05 10:22:36.951647 [764c3497-8ff4-4ca2-a60c-1a7122c6943f] mw2320 testwiki 1.39.0-wmf.27 DBReplication DEBUG: Wikimedia\Rdbms\ChronologyProtector::applySessionReplicationPosition: extension2 (db2142) has no position

Not sure why it's checking the replication position, but there's no error, just a debug message. Viewing and editing works just fine, using codfw with stopped replication on x2.

Mentioned in SAL (#wikimedia-operations) [2022-09-05T11:29:58Z] <TimStarling> on db2142: set master_delay=30 and restarted replication T316847

Mentioned in SAL (#wikimedia-operations) [2022-09-05T11:37:36Z] <TimStarling> on db2142: dropping inbound mysql traffic T316847

With -j DROP there were plenty of errors logged in the DBConnection channel, and from the log timestamps it looks like the connection timeout was 3s. No problem with page views or edits. Seemed slow, but I tested action=edit again after I dropped the iptables rule, and it was about as slow (~3.5 second response time). There was no spike in the exception channel.

Mentioned in SAL (#wikimedia-operations) [2022-09-05T11:55:22Z] <TimStarling> on db2142: rejecting inbound mysql traffic T316847

In the -j REJECT test I found a better manual test to do, which was to click on the echo bell. It triggered a DBConnection log entry, but no user-visible error.

2022-09-05 11:56:21.663667 [7f29d1c3-a634-4d13-b4ab-f13b7ef46af9] mw2366 testwiki 1.39.0-wmf.27 DBConnection ERROR: Error connecting to db2142 as user wikiuser202206: :real_connect(): (HY000/2002): Connection refused {"db_server":"db2142","db_name":"mainstash","db_user":"wikiuser202206","error":":real_connect(): (HY000/2002): Connection refused","db_log_category":"connection"}

As with -j DROP there was a spike of DBConnection log entries but no associated exception log entries.

In summary, nothing unexpected. Stopping or delaying replication causes a split brain — probably there are subtle consequences, but the site didn't go down, and that's all we were really checking for here. With iptables simulating failure of the x2 master, the logs showed graceful failure after a 3 second connect timeout.

Change 830052 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] x2 replicas: Disable notifications

https://gerrit.wikimedia.org/r/830052

Change 830052 merged by Marostegui:

[operations/puppet@production] x2 replicas: Disable notifications

https://gerrit.wikimedia.org/r/830052

I have disabled notifications on all x2 replicas. They won't page, but they'll still be shown in icinga if something breaks