We have switched Thumbor/thumbnails to be active/active T201858, AFAIK we should be able to serve originals too in an active/active fashion. Said feature is driven by FileBackend being in "sync" mode between its eqiad and codfw swift backends.
A list of things to test/verify for completion:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | aaron | T88445 MediaWiki active/active datacenter investigation and work (tracking) | |||
| Open | None | T204245 Run MediaWiki media originals active/active |
As far as I can tell, the math extension is serving images through the REST API (Mathoid) and as such doesn't use the "original" urls:
https://wikimedia.org/api/rest_v1/media/math/render/svg/b9f8284fcea4d88e1ef5816226f189c6b2d2d2ee
Maybe the background Mathoid stores those renders in Swift, but that's irrelevant to making the public-facing https://upload.wikimedia.org/* URLs active-active, imho.
Same for captchas, they are served through these URLs:
https://en.wikipedia.org/w/index.php?title=Special:Captcha/image&wpCaptchaId=290077603
Musical scores are the only special type from that list served from upload.wikimedia.org:
https://upload.wikimedia.org/score/e/g/egv3toazxv8fm3whmr3weuhlx6qgj15/egv3toaz.png
$wmgScoreFileBackend is configured as "global-multiwrite". Which is actually the same used by Math and Captchas.
Here's its configuration:
$globalMultiWriteFileBackend = [
'class' => 'FileBackendMultiWrite',
'name' => 'global-multiwrite',
'wikiId' => "global-data",
'lockManager' => 'redisLockManager',
# DO NOT change the master backend unless it is fully trusted or autoRsync is off
'backends' => [
[ 'template' => 'global-swift-eqiad', 'isMultiMaster' => true ],
],
'replication' => 'sync', // read-after-update for assets
'syncChecks' => ( 1 | 4 ) // (size & sha1)
];
if ( in_array( 'codfw', $datacenters ) ) {
$localMultiWriteFileBackend['backends'][] = [ 'template' => 'local-swift-codfw' ];
$sharedMultiwriteFileBackend['backends'][] = [ 'template' => 'shared-swift-codfw' ];
$globalMultiWriteFileBackend['backends'][] = [ 'template' => 'global-swift-codfw' ];
$sharedTestwikiMultiWriteFileBackend['backends'][] = [ 'template' => 'shared-testwiki-swift-codfw' ];
}This confirms that all those 3 types are configured to write to both DCs when objects are created (even if for the purpose of this task only Score mattered). The replication is synchronous, meaning that if pushing a new object to the secondary DC fails, it should result in an error for the upload. And the web request will wait until the object is pushed to all DCs before completing.
Math/Mathoid's renders are stored in Cassandra, so are already active/active.
@fgiunchedi / @Gilles what is left to do here for us to be able to serve files from Swift in active/active (apart from config changes and potentially some quick VCL) ?
As far as I can tell, all it would take is uncommenting one line of yaml in hieradata/role/common/cache/upload.yaml
Change 496872 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[operations/puppet@production] Varnish: serve Swift traffic in active/active mode
Awesome, thank you @Gilles ! I put up a patch for that (^) so that we don't forget it :)
Change 496872 had a related patch set uploaded (by Alexandros Kosiaris; owner: Mobrovac):
[operations/puppet@production] Varnish: serve Swift traffic in active/active mode
Change 496872 merged by Alexandros Kosiaris:
[operations/puppet@production] Varnish: serve Swift traffic in active/active mode
Change 502453 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/dns@master] swift-rw: Mock it as a geo-resource
Change 502456 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] swift: Revert swift-rw to active/passive
Change 502457 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] swift: Introduce new swift.discovery.wmnet stanza
Change 502458 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] trafficserver: Switch to using swift.discovery.wmnet
Change 502456 merged by Alexandros Kosiaris:
[operations/puppet@production] swift: Revert swift-rw to active/passive
Change 502457 merged by Alexandros Kosiaris:
[operations/puppet@production] swift: Introduce new swift.discovery.wmnet stanza
Change 502453 merged by Alexandros Kosiaris:
[operations/dns@master] swift-rw: Mock it as a geo-resource
Change 503274 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/dns@master] Add a new swift.discovery.wmnet resource
Change 503274 merged by Alexandros Kosiaris:
[operations/dns@master] Add a new swift.discovery.wmnet resource
Change 504331 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] swift: new cert for ms-fe.svc.codfw.wmnet
Mentioned in SAL (#wikimedia-operations) [2019-04-16T14:04:29Z] <ema> test https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/504331/ on ms-fe2005 T204245
Change 504331 merged by Ema:
[operations/puppet@production] swift: new cert for ms-fe.svc.codfw.wmnet
Mentioned in SAL (#wikimedia-operations) [2019-04-16T14:30:22Z] <ema> swift-fe-codfw: nginx reload for new TLS certificate T204245
Change 504340 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] swift: new cert for ms-fe.svc.eqiad.wmnet
Mentioned in SAL (#wikimedia-operations) [2019-04-16T14:45:36Z] <ema> test https://gerrit.wikimedia.org/r/504340 on ms-fe1005 T204245
Change 504340 merged by Ema:
[operations/puppet@production] swift: new cert for ms-fe.svc.eqiad.wmnet
Mentioned in SAL (#wikimedia-operations) [2019-04-16T14:56:29Z] <ema> swift-fe-eqiad: nginx reload for new TLS certificate T204245
Change 504349 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] Revert "ATS: use 'swift-ro' as the origin server for thumb traffic"
Change 504349 merged by Ema:
[operations/puppet@production] Revert "ATS: use 'swift-ro' as the origin server for thumb traffic"
Change 502458 merged by Ema:
[operations/puppet@production] trafficserver: Switch to using swift.discovery.wmnet
I think we have evidence of this not always happening in T231086: Picture from Commons not found from Singapore