Page MenuHomePhabricator
Create Task
Maniphest T114273

Cleanup active-DC based MW config code and make it more robust and easy to change
Closed, ResolvedPublic
Actions

Description

A general issue is also that of config. MediaWiki has lots of config deciding what DB/swift to talk to. If stale config is still running in prod due to network partitions (breaking scap and so on), we can still have problems. One option is to have config switches around "what DC is active".

The active DC name could be pulled from a file, managed by ecd eventually (instead of the less reliable scap). We could have apaches go read only if that file is out of contact with etcd.

Details

SubjectRepoBranchLines +/-
Use local resources in codfw for parsoid, url-downloader and mathoidoperations/mediawiki-configmaster+4 -4
Use ProductionServices for the jobqueue configurationoperations/mediawiki-configmaster+101 -180
Use wmfMasterDatacenter for picking the master redis configoperations/mediawiki-configmaster+1 -1
Add references to wmfServices for Cirrusearch.operations/mediawiki-configmaster+8 -13
Define service entries for InitialiseSettingsoperations/mediawiki-configmaster+103 -27
Reduce poolcounter configuration complexityoperations/mediawiki-configmaster+29 -42
Rationalize definition of service hostsoperations/mediawiki-configmaster+60 -9
Customize query in gerrit

Related Objects
Search...

Event Timeline

aaron created this task.Sep 30 2015, 5:40 PM
aaron claimed this task.
aaron raised the priority of this task from to Medium.
aaron updated the task description. (Show Details)
aaron added projects: Performance-Team, Sustainability, Epic.
aaron mentioned this in T114271: Audit *all* datastores to see how DR failover is handled.
aaron added subscribers: Krinkle, jcrespo, Glaisher and 13 others.
Krinkle moved this task from Inbox, needs triage to Backlog: Maintenance, non-prioritized on the Performance-Team board.Oct 5 2015, 6:15 PM
aaron renamed this task from Cleanup active-DC based MW code and make it more robust and easy to change to Cleanup active-DC based MW config code and make it more robust and easy to change.Oct 5 2015, 6:30 PM
aaron updated the task description. (Show Details)
aaron set Security to None.
fgiunchedi subscribed.Jan 20 2016, 4:40 PM
Joe added a comment.Jan 20 2016, 4:47 PM

Other things will probably need the same kind of switching:

First thing that comes to mind is Elasticsearch

Joe added a comment.Jan 20 2016, 4:47 PM

Also: various redis configs, I guess

Joe added projects: SRE, codfw-rollout-Jan-Mar-2016.Jan 20 2016, 4:48 PM
Aklapper added a project: codfw-rollout.Jan 20 2016, 6:28 PM
Joe mentioned this in T124671: Switchover of the application servers to codfw.Jan 25 2016, 4:52 PM
gerritbot added a comment.Jan 26 2016, 4:05 PM

Change 266509 had a related patch set uploaded (by Giuseppe Lavagetto):
Rationalize definition of service hosts

https://gerrit.wikimedia.org/r/266509

gerritbot added a project: Patch-For-Review.Jan 26 2016, 4:05 PM

Change 266510 had a related patch set uploaded (by Giuseppe Lavagetto):
Define Production service entries for InitialiseSettings

https://gerrit.wikimedia.org/r/266510

gerritbot added a comment.Jan 26 2016, 4:05 PM

Change 266511 had a related patch set uploaded (by Giuseppe Lavagetto):
Reduce poolcounter configuration complexity

https://gerrit.wikimedia.org/r/266511

gerritbot added a comment.Jan 26 2016, 4:05 PM

Change 266512 had a related patch set uploaded (by Giuseppe Lavagetto):
Add references to wmfServices for Cirrusearch.

https://gerrit.wikimedia.org/r/266512

gerritbot added a comment.Jan 26 2016, 4:05 PM

Change 266513 had a related patch set uploaded (by Giuseppe Lavagetto):
Use wmfMasterDatacenter for picking the master redis config

https://gerrit.wikimedia.org/r/266513

gerritbot added a comment.Feb 9 2016, 12:49 AM

Change 266509 merged by jenkins-bot:
Rationalize definition of service hosts

https://gerrit.wikimedia.org/r/266509

mark raised the priority of this task from Medium to High.Feb 10 2016, 3:52 PM
gerritbot added a comment.Feb 24 2016, 4:41 PM

Change 266510 merged by jenkins-bot:
Define service entries for InitialiseSettings

https://gerrit.wikimedia.org/r/266510

gerritbot added a comment.Feb 24 2016, 5:35 PM

Change 266511 merged by jenkins-bot:
Reduce poolcounter configuration complexity

https://gerrit.wikimedia.org/r/266511

jcrespo added a subtask: T124795: codfw is in read only according to mediawiki.Feb 24 2016, 5:54 PM
gerritbot added a comment.Mar 1 2016, 4:10 PM

Change 266512 merged by jenkins-bot:
Add references to wmfServices for Cirrusearch.

https://gerrit.wikimedia.org/r/266512

gerritbot added a comment.Mar 1 2016, 4:20 PM

Change 266513 merged by jenkins-bot:
Use wmfMasterDatacenter for picking the master redis config

https://gerrit.wikimedia.org/r/266513

Joe claimed this task.Mar 7 2016, 10:17 AM
Joe moved this task from Backlog to In Progress on the codfw-rollout-Jan-Mar-2016 board.
jcrespo closed subtask T124795: codfw is in read only according to mediawiki as Resolved.Mar 10 2016, 7:09 PM
gerritbot added a comment.Mar 24 2016, 1:05 PM

Change 279350 had a related patch set uploaded (by Giuseppe Lavagetto):
Use ProductionServices for the jobqueue configuration

https://gerrit.wikimedia.org/r/279350

gerritbot added a comment.Mar 24 2016, 1:14 PM

Change 279355 had a related patch set uploaded (by Giuseppe Lavagetto):
Use local resources in codfw for parsoid, url-downloader and mathoid

https://gerrit.wikimedia.org/r/279355

Krinkle unsubscribed.Mar 30 2016, 2:41 AM
gerritbot added a comment.Apr 7 2016, 3:13 PM

Change 279350 merged by jenkins-bot:
Use ProductionServices for the jobqueue configuration

https://gerrit.wikimedia.org/r/279350

gerritbot added a comment.Apr 7 2016, 3:28 PM

Change 279355 merged by jenkins-bot:
Use local resources in codfw for parsoid, url-downloader and mathoid

https://gerrit.wikimedia.org/r/279355

Joe closed this task as Resolved.Apr 11 2016, 3:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL