A general issue is also that of config. MediaWiki has lots of config deciding what DB/swift to talk to. If stale config is still running in prod due to network partitions (breaking scap and so on), we can still have problems. One option is to have config switches around "what DC is active".
The active DC name could be pulled from a file, managed by ecd eventually (instead of the less reliable scap). We could have apaches go read only if that file is out of contact with etcd.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | aaron | T88445 MediaWiki active/active datacenter investigation and work (tracking) | |||
| Resolved | Joe | T114273 Cleanup active-DC based MW config code and make it more robust and easy to change | |||
| Resolved | • jcrespo | T124795 codfw is in read only according to mediawiki |
Other things will probably need the same kind of switching:
First thing that comes to mind is Elasticsearch
Change 266509 had a related patch set uploaded (by Giuseppe Lavagetto):
Rationalize definition of service hosts
Change 266510 had a related patch set uploaded (by Giuseppe Lavagetto):
Define Production service entries for InitialiseSettings
Change 266511 had a related patch set uploaded (by Giuseppe Lavagetto):
Reduce poolcounter configuration complexity
Change 266512 had a related patch set uploaded (by Giuseppe Lavagetto):
Add references to wmfServices for Cirrusearch.
Change 266513 had a related patch set uploaded (by Giuseppe Lavagetto):
Use wmfMasterDatacenter for picking the master redis config
Change 266513 merged by jenkins-bot:
Use wmfMasterDatacenter for picking the master redis config
Change 279350 had a related patch set uploaded (by Giuseppe Lavagetto):
Use ProductionServices for the jobqueue configuration
Change 279355 had a related patch set uploaded (by Giuseppe Lavagetto):
Use local resources in codfw for parsoid, url-downloader and mathoid
Change 279350 merged by jenkins-bot:
Use ProductionServices for the jobqueue configuration
Change 279355 merged by jenkins-bot:
Use local resources in codfw for parsoid, url-downloader and mathoid