Page MenuHomePhabricator
Create Task
Maniphest T321770

Entity dumpers should reload db config on the fly
Closed, ResolvedPublic
Actions

Description

Follow up to T300255: Wikidata entity dumper keeps connecting to depooled host for really long time and T298485: MW scripts should reload the database config

It is taking a bit after each depool for the dumper to actually drain the connections. The solution would be to add $lbFactory->autoReconfigure() to after each batch (DumpGenerator::generateDump() I think)

Details

SubjectRepoBranchLines +/-
Auto-reconfigure LBFactory in some more placesmediawiki/extensions/Wikibasemaster+6 -0
Auto-reconfigure LBFactory in DumpGeneratormediawiki/extensions/Wikibasemaster+58 -0
Add autoReconfigure() to ILBFactorymediawiki/coremaster+6 -0
Customize query in gerrit

Related Objects

Event Timeline

Ladsgroup created this task.Oct 27 2022, 8:06 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 27 2022, 8:06 AM
Lucas_Werkmeister_WMDE added a comment.Oct 27 2022, 12:58 PM

If we want to avoid adding DB-specific knowledge into the DumpGenerator, I think we could also put the “reconfigure” call into SqlEntityIdPager::fetchIds()… but the dump generator probably makes more sense.

ItamarWMDE added a project: wmde-wikidata-tech.Oct 28 2022, 8:44 AM
Maintenance_bot moved this task from Inbox to Sorted Team A on the [DEPRECATED] wdwb-tech board.Oct 28 2022, 9:15 AM
Michael added a comment.Nov 8 2022, 2:10 PM

Prio Notes:

  • Affects Production systems
  • Does not affect development efforts
  • Does not affect onboarding efforts
  • Might affect additional stakeholders (DBAs)
Michael moved this task from Incoming to [DOT] Prioritized on the wmde-wikidata-tech board.Nov 8 2022, 2:11 PM
Michael added a project: Wikidata Dev Team.
Michael moved this task from Incoming to [DOT] Tech Backlog on the Wikidata Dev Team board.Nov 9 2022, 10:56 AM
Lucas_Werkmeister_WMDE added a comment.Nov 16 2022, 10:06 AM

Note from a conversation with Amir the other day: Once we fix this, we can increase the dumpers’ runtime again (revert the change related to T300255).

Lucas_Werkmeister_WMDE claimed this task.Nov 16 2022, 3:52 PM
gerritbot added a comment.Nov 16 2022, 3:52 PM

Change 857670 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@master] Auto-reconfigure LBFactory in DumpGenerator

https://gerrit.wikimedia.org/r/857670

gerritbot added a project: Patch-For-Review.Nov 16 2022, 3:52 PM
Lucas_Werkmeister_WMDE edited projects, added Wikidata Dev Team (Sprint-∞); removed Wikidata Dev Team.Nov 16 2022, 4:19 PM
Lucas_Werkmeister_WMDE moved this task from Parents/Waiting to Peer Review on the Wikidata Dev Team (Sprint-∞) board.
Lucas_Werkmeister_WMDE moved this task from Peer Review to Doing on the Wikidata Dev Team (Sprint-∞) board.
Lucas_Werkmeister_WMDE added a subscriber: daniel.Nov 16 2022, 4:31 PM

@Ladsgroup and @daniel – I just stumbled across this paragraph in the LBFactory::autoReconfigure() (and reconfigure()) phpdoc:

@warning This must only be called in top level code such as the execute()
method of a maintenance script. Any database connection in use when this
method is called will become defunct.

This no longer applies, right? DBConnRefs automatically reconnect, and we don’t really use non-ref connections anymore?

gerritbot added a comment.Nov 16 2022, 4:53 PM

Change 857707 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/core@master] Add autoReconfigure() to ILBFactory

https://gerrit.wikimedia.org/r/857707

Lucas_Werkmeister_WMDE moved this task from Doing to Peer Review on the Wikidata Dev Team (Sprint-∞) board.Nov 16 2022, 5:18 PM
gerritbot added a comment.Nov 17 2022, 1:22 PM

Change 857707 merged by jenkins-bot:

[mediawiki/core@master] Add autoReconfigure() to ILBFactory

https://gerrit.wikimedia.org/r/857707

ReleaseTaggerBot added a project: MW-1.40-notes (1.40.0-wmf.12; 2022-11-28).Nov 17 2022, 2:00 PM
gerritbot added a comment.Nov 17 2022, 5:20 PM

Change 857670 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Auto-reconfigure LBFactory in DumpGenerator

https://gerrit.wikimedia.org/r/857670

Maintenance_bot removed a project: Patch-For-Review.Nov 17 2022, 5:30 PM
Lucas_Werkmeister_WMDE moved this task from Peer Review to Tech Verification on the Wikidata Dev Team (Sprint-∞) board.Nov 18 2022, 10:30 AM
gerritbot added a comment.Nov 25 2022, 4:52 PM

Change 860922 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@master] Auto-reconfigure LBFactory in some more places

https://gerrit.wikimedia.org/r/860922

gerritbot added a project: Patch-For-Review.Nov 25 2022, 4:52 PM
Lucas_Werkmeister_WMDE added a comment.Nov 25 2022, 4:55 PM
In T321770#8399946, @Lucas_Werkmeister_WMDE wrote:

@Ladsgroup and @daniel – I just stumbled across this paragraph in the LBFactory::autoReconfigure() (and reconfigure()) phpdoc:

@warning This must only be called in top level code such as the execute()
method of a maintenance script. Any database connection in use when this
method is called will become defunct.

This no longer applies, right? DBConnRefs automatically reconnect, and we don’t really use non-ref connections anymore?

Uploaded a Gerrit change to remove that comment.

gerritbot added a comment.Nov 25 2022, 5:41 PM

Change 860922 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Auto-reconfigure LBFactory in some more places

https://gerrit.wikimedia.org/r/860922

Maintenance_bot removed a project: Patch-For-Review.Nov 25 2022, 6:30 PM
Michael moved this task from [DOT] Prioritized to Ongoing on the wmde-wikidata-tech board.Dec 6 2022, 2:14 PM
Lucas_Werkmeister_WMDE closed this task as Resolved.Dec 6 2022, 2:16 PM
Lucas_Werkmeister_WMDE moved this task from Tech Verification to Our work done on the Wikidata Dev Team (Sprint-∞) board.

I’m going to assume this is done; DBAs, if you still see reconnects from long-running Wikibase scripts, please reopen.

karapayneWMDE moved this task from Our work done to 2022 Completed Tasks on the Wikidata Dev Team (Sprint-∞) board.Dec 20 2022, 8:05 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits