Page MenuHomePhabricator
Create Task
Maniphest T186436

Improve GUC database handling
Closed, ResolvedPublic
Actions

Description

Ref T182916: Database error: Unable to connect to s7.web.db.svc.eqiad.wmflabs

  • Ensure all PDO statement objects are dereferenced after use so that early close is actually possible.
  • Track connection count and log it.
  • Add method for closing a connection.
  • Add early close where possible
    • Use s1 for meta_p.wiki query (documented to be present on available on all db hosts)
    • Add explicit close at end of web request.
    • Add early close for CentralAuth db, only used for one query.
    • Sort all-wiki queries by slice so that we can open/close per batch and never have a conn open to more than one. Not done for now, given that the IP-based solution has already reduced connections from 9 to 1. At a later time when there is more than 1 minimum connection required, this can be revisited.
  • Figure out a way to make centralauth use s7, without hardcoding it. https://gerrit.wikimedia.org/r/408751

Details

SubjectRepoBranchLines +/-
Fix bug with overly eager connection closinglabs/tools/gucmaster+60 -7
Improve logging about closed connectionslabs/tools/gucmaster+21 -11
Re-use connections based on IP instead of hostnamelabs/tools/gucmaster+82 -18
Omit USE/$dbname for multi-wiki and 'centralauth' querieslabs/tools/gucmaster+3 -3
Close connections earlier (after getData, instead of after output)labs/tools/gucmaster+1 -0
Explicitly try to close connections before end of responselabs/tools/gucmaster+40 -2
Close 'centralauth' connection earlylabs/tools/gucmaster+5 -0
Add debug logging for connection countslabs/tools/gucmaster+10 -0
Implement App::closeDD() methodlabs/tools/gucmaster+119 -6
Consistently dereference PDO statement objects after uselabs/tools/gucmaster+22 -13
Customize query in gerrit

Related Objects
Search...

Event Timeline

Krinkle created this task.Feb 4 2018, 5:02 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 4 2018, 5:02 AM
Krinkle renamed this task from Improve database connection handling to Improve GUC database handling.Feb 4 2018, 5:02 AM
Krinkle triaged this task as High priority.
gerritbot subscribed.Feb 4 2018, 5:05 AM

Change 408025 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[labs/tools/guc@master] Consistently dereference PDO statement objects after use

https://gerrit.wikimedia.org/r/408025

gerritbot added a project: Patch-For-Review.Feb 4 2018, 5:05 AM

Change 408025 merged by jenkins-bot:
[labs/tools/guc@master] Consistently dereference PDO statement objects after use

https://gerrit.wikimedia.org/r/408025

Chicocvenancio subscribed.Feb 5 2018, 8:35 PM
gerritbot added a comment.Feb 7 2018, 12:00 AM

Change 408719 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[labs/tools/guc@master] Implement App::closeDD() method and use to close 'centralauth' conn early

https://gerrit.wikimedia.org/r/408719

gerritbot added a comment.Feb 7 2018, 12:09 AM

Change 408722 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[labs/tools/guc@master] Add debug logging for connection counts

https://gerrit.wikimedia.org/r/408722

gerritbot added a comment.Feb 7 2018, 12:15 AM

Change 408719 merged by jenkins-bot:
[labs/tools/guc@master] Implement App::closeDD() method

https://gerrit.wikimedia.org/r/408719

gerritbot added a comment.Feb 7 2018, 12:18 AM

Change 408722 merged by jenkins-bot:
[labs/tools/guc@master] Add debug logging for connection counts

https://gerrit.wikimedia.org/r/408722

Krinkle added a comment.Feb 7 2018, 12:20 AM

Current worst-case connection stats (e.g for a user that has non-zero edits on at least one wiki on each section)

Connections opened: 9
Highest connection count: 9
Finish

Let's start optimising :)

gerritbot added a comment.Feb 7 2018, 12:20 AM

Change 408724 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[labs/tools/guc@master] Close 'centralauth' connection early

https://gerrit.wikimedia.org/r/408724

gerritbot added a comment.Feb 7 2018, 12:21 AM

Change 408724 merged by jenkins-bot:
[labs/tools/guc@master] Close 'centralauth' connection early

https://gerrit.wikimedia.org/r/408724

gerritbot added a comment.Feb 7 2018, 12:33 AM

Change 408725 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[labs/tools/guc@master] Explicitly try to close connections before end of response

https://gerrit.wikimedia.org/r/408725

gerritbot added a comment.Feb 7 2018, 12:33 AM

Change 408726 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[labs/tools/guc@master] Close connections earlier (after getData, instead of after output)

https://gerrit.wikimedia.org/r/408726

gerritbot added a comment.Feb 7 2018, 12:39 AM

Change 408725 merged by jenkins-bot:
[labs/tools/guc@master] Explicitly try to close connections before end of response

https://gerrit.wikimedia.org/r/408725

gerritbot added a comment.Feb 7 2018, 2:00 AM

Change 408726 merged by Krinkle:
[labs/tools/guc@master] Close connections earlier (after getData, instead of after output)

https://gerrit.wikimedia.org/r/408726

Krinkle updated the task description. (Show Details)Feb 7 2018, 2:17 AM
Krinkle updated the task description. (Show Details)
gerritbot added a comment.Feb 7 2018, 2:52 AM

Change 408745 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[labs/tools/guc@master] Omit USE/$dbname for multi-wiki and 'centralauth' queries

https://gerrit.wikimedia.org/r/408745

gerritbot added a comment.Feb 7 2018, 2:53 AM

Change 408745 merged by jenkins-bot:
[labs/tools/guc@master] Omit USE/$dbname for multi-wiki and 'centralauth' queries

https://gerrit.wikimedia.org/r/408745

gerritbot added a comment.Feb 7 2018, 3:21 AM

Change 408751 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[labs/tools/guc@master] Re-use connections based on IP instead of hostname

https://gerrit.wikimedia.org/r/408751

gerritbot added a comment.Feb 7 2018, 3:24 AM

Change 408751 merged by jenkins-bot:
[labs/tools/guc@master] Re-use connections based on IP instead of hostname

https://gerrit.wikimedia.org/r/408751

Krinkle updated the task description. (Show Details)Feb 7 2018, 3:29 AM
gerritbot added a comment.Feb 7 2018, 3:35 AM

Change 408752 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[labs/tools/guc@master] Improve logging about closed connections

https://gerrit.wikimedia.org/r/408752

gerritbot added a comment.Feb 7 2018, 3:35 AM

Change 408752 merged by jenkins-bot:
[labs/tools/guc@master] Improve logging about closed connections

https://gerrit.wikimedia.org/r/408752

Krinkle removed a project: Patch-For-Review.Feb 7 2018, 3:36 AM
Krinkle updated the task description. (Show Details)
gerritbot added a comment.Feb 7 2018, 3:57 AM

Change 408755 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[labs/tools/guc@master] Fix bug with overly eager connection closing

https://gerrit.wikimedia.org/r/408755

gerritbot added a project: Patch-For-Review.Feb 7 2018, 3:57 AM

Change 408755 merged by Krinkle:
[labs/tools/guc@master] Fix bug with overly eager connection closing

https://gerrit.wikimedia.org/r/408755

Krinkle closed this task as Resolved.Feb 7 2018, 3:58 AM
Krinkle mentioned this in T182916: Database error: Unable to connect to s7.web.db.svc.eqiad.wmflabs.Feb 7 2018, 4:06 AM
Krinkle mentioned this in T186689: toolforge: xtools: exceed max database connections.Feb 8 2018, 4:27 AM
• 253523 renamed this task from Improve GUC database handling to ☬ʄທയஆടஷະ☬のລັບ☠ ☬ ʄທയஆടஷະ┅─ ★͜͡ ʄທയஆടஷະ┅─.Feb 12 2018, 10:28 AM
• 253523 reopened this task as Open.
• 253523 raised the priority of this task from High to Needs Triage.
• 253523 set the point value for this task to 408751.
Aklapper renamed this task from ☬ʄທയஆടஷະ☬のລັບ☠ ☬ ʄທയஆടஷະ┅─ ★͜͡ ʄທയஆടஷະ┅─ to Improve GUC database handling.Feb 12 2018, 1:25 PM
Aklapper closed this task as Resolved.
Aklapper triaged this task as High priority.
Aklapper removed the point value for this task.
Krinkle removed a project: Patch-For-Review.May 19 2018, 11:54 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL