Mediawiki sets/gets special purge keys ("tombstones") with the prefix /*/mw-wan/. Those keys are broadcasted from the primary to the secondary DC, via mcrouter. While mcrouter directly connects to the memcached cluster on the remote datacentre to set those keys, currently there is no proper provision for a memcached server on the remote DC being down. What we will do is include both gutter pools in each mcrouter config, and failover to the remote DC's gutter pool in case a remote DC memcached server is unavailable
In T240684#5891129 we tested the following config
{
"aliases": [ "/codfw/mw-wan/"],
"route": {
"type": "OperationSelectorRoute",
"default_policy": {
"type": "FailoverRoute",
"children": [ "PoolRoute|eqiad", "PoolRoute|eqiad-gutter" ],
"failover_errors": [ "tko" ]
},
"operation_policies": {
"delete": {
"type": "AllSyncRoute",
"children": [ "AllSyncRoute|Pool|codfw", "AllSyncRoute|Pool|codfw-gutter" ]
},
"set": {
"type": "FailoverRoute",
"children": [ "PoolRoute|codfw", "PoolRoute|codfw-gutter" ],
"failover_errors": [ "tko" ]
}
}
}
}We should deploy this configuration to production as well!