Page MenuHomePhabricator
Create Task
Maniphest T230800

Improve Rdbms/LoadBalance and its LoadMonitor logic
Closed, ResolvedPublic
Actions

Description

Follows investigation from {T180918}.

From https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/394430/

Rewrite LoadMonitor and related LoadBalancer logic

  • Use WANObjectCache::getWithSetCallback()
  • Increase the server state polling interval but add a mutex on shared cache updates via "lockTSE" as a throttle
  • Increase the moving average factor to make the weight scale values more responsive to problems
  • Account for DBError exceptions from getLag()
  • Add pingFailure() method to quickly react to failing servers rather than just waiting on the server state polls
  • Add backtraces to replication wait timeouts

Was initially reviewed by @Anomie, with feedback that has since been incorporated by @aaron. Requesting further review/merging from CPT.

Details

SubjectRepoBranchLines +/-
Make LoadMonitorMySQL better detect servers with connection problemsmediawiki/coremaster+815 -34
rdbms: refactor the caching logic in LoadMonitormediawiki/coremaster+231 -69
Customize query in gerrit

Event Timeline

Krinkle created this task.Aug 20 2019, 1:32 PM
Restricted Application added a project: Platform Engineering. · View Herald TranscriptAug 20 2019, 1:32 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Krinkle moved this task from Limbo to Perf recommendation on the Performance-Team (Radar) board.Aug 20 2019, 1:32 PM
gerritbot added a comment.Aug 20 2019, 1:33 PM

Change 394430 had a related patch set uploaded (by Krinkle; owner: Aaron Schulz):
[mediawiki/core@master] Rewrite LoadMonitor and related LoadBalancer logic

https://gerrit.wikimedia.org/r/394430

gerritbot added a project: Patch-For-Review.Aug 20 2019, 1:33 PM
Anomie edited projects, added Platform Team Workboards (Clinic Duty Team); removed Platform Engineering.Aug 20 2019, 2:17 PM
Anomie moved this task from Inbox to External Code Review Needed on the Platform Team Workboards (Clinic Duty Team) board.
daniel triaged this task as Medium priority.Aug 22 2019, 7:55 AM
Krinkle moved this task from Untriaged to Rdbms library on the MediaWiki-libs-Rdbms board.Aug 28 2019, 10:10 PM
WDoranWMF raised the priority of this task from Medium to High.Sep 11 2019, 4:42 PM
WDoranWMF subscribed.

Raising the priority for this task as it's blocking for performance.

• Marostegui subscribed.Nov 12 2019, 1:36 PM
• mobrovac moved this task from External Code Review Needed to External Code Review In Progress on the Platform Team Workboards (Clinic Duty Team) board.Dec 10 2019, 12:41 PM
• mobrovac added subscribers: • jcrespo, • mobrovac.

FYI, I already +1'ed the patch, but @jcrespo raised some questions on the PS that I think should be addressed before merging.

WDoranWMF moved this task from External Code Review In Progress to External Code Review Completed on the Platform Team Workboards (Clinic Duty Team) board.Mar 3 2020, 8:50 PM
gerritbot added a comment.Mar 9 2020, 8:38 PM

Change 577702 had a related patch set uploaded (by Krinkle; owner: Aaron Schulz):
[mediawiki/core@master] rdbms: refactor the caching logic in LoadMontior

https://gerrit.wikimedia.org/r/577702

gerritbot added a comment.Mar 10 2020, 2:04 AM

Change 577702 merged by jenkins-bot:
[mediawiki/core@master] rdbms: refactor the caching logic in LoadMonitor

https://gerrit.wikimedia.org/r/577702

ReleaseTaggerBot added a project: MW-1.35-notes (1.35.0-wmf.23; 2020-03-10).Mar 10 2020, 3:00 AM
Aklapper removed a subscriber: Anomie.Oct 16 2020, 5:01 PM
gerritbot added a comment.Nov 30 2020, 9:55 PM

Change 394430 abandoned by Aaron Schulz:
[mediawiki/core@master] Make LoadMonitorMySQL better detect servers with connection problems

Reason:
No use for WMF (goes with etcd external tools)

https://gerrit.wikimedia.org/r/394430

Krinkle closed this task as Resolved.Dec 3 2020, 3:05 AM
Krinkle assigned this task to aaron.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits