Page MenuHomePhabricator
Create Task
Maniphest T226167

audit public tables and make sure we dump them all
Closed, ResolvedPublic0 Estimated Story Points
Actions

Description

Purely internal tables that just connect one foreign key to another are not needed but let's double check that everything else is dumped. It's been awhile.

  • P8634 lists tables broken out into groups of: dumped already, known private, cache/temp, empty on WMF, for review, and one group from which the relevant fields are dumped as xml.
  • P8774 has a smaller set of tables that aren't on all wikis.

Details

SubjectRepoBranchLines +/-
add more public tables for xml/sql dumpsoperations/puppetproduction+65 -5
add more public sql tables to xml/sql dumpsoperations/puppetproduction+12 -0
add a few more public sql tables to default list to be dumpedoperations/dumpsmaster+12 -0
add a few more public sql tables to default list to be dumpedoperations/dumpsmaster+12 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedArielGlennT226167 audit public tables and make sure we dump them all
 ResolvedAddshoreT219175 [Mega] - Migrate data from wb_terms to new schema
 ResolvedAddshoreT219120 [Checkpoint 1] Create Schema, Migration plan and Doctrine DBAL connection
Resolved• alaa_wmdeT219142 Send the schema to DBAs and follow up with their review
 Resolved• alaa_wmdeT219143 Write DDL sql files for creating the schema
 Resolved• alaa_wmdeT220210 Create a script to generate lots of Items/Properties with lots of Terms
 ResolvedNoneT220230 Tailor a migration plan
 DeclinedNoneT220231 Migration Plan 2
 Declined• alaa_wmdeT219145 Migration Plan 1
 ResolvedLadsgroupT220480 Migration Plan 3
 ResolvedAddshoreT219121 [Checkpoint 2] Refactor Write & Migration code for Properties
 ResolvedLadsgroupT219294 Create interface for writing and deleting entity terms
 ResolvedJeroenDeDauwT219295 Create service for writing property terms
 DeclinedNoneT220169 Optimize term updating
 ResolvedNoneT219297 Adjust current write logic to write property terms to new schema
 ResolvedNoneT219894 Create maintenance script for migrating property terms to new schema
 ResolvedLadsgroupT219296 Update TermSqlStore to comply with EntityTermStoreWriter interface
 ResolvedNoneT220150 [Story] Clean up unused records
 InvalidNoneT221699 Introduce appropriate abstraction for clean up invocation in wikibase-term-store
 DeclinedNoneT221700 implement clean up logic in doctrine-term-store
 ResolvedLucas_Werkmeister_WMDET221701 implement clean up logic in MW DBAL store
 InvalidNoneT221703 add a job to wikibase to invoke clean up logic
 DeclinedNoneT221704 Update wikibase logic to queue a deferred update for cleanup after updating/delete terms in terms store
 ResolvedLadsgroupT221615 Make a test instance in cloud VPS to test new term store
 DeclinedNoneT219805 Investigate Doctrine DBAL usage possibility
 ResolvedJeroenDeDauwT220155 Implement Doctrine DBAL Driver Connection wrapper for MW db connection
 DeclinedNoneT221568 Improve MW connection access
 Resolved• alaa_wmdeT224269 Handle race-conditions between storage and cleanup logic
 ResolvedLucas_Werkmeister_WMDET225050 Update DatabaseItemTermStore and DatabasePropertyTermStore to use callback logic in DatabaseTermIdsAcquirer
 ResolvedLucas_Werkmeister_WMDET224504 Implement database-based PropertyTermStore
 ResolvedLucas_Werkmeister_WMDET224541 Update Wikibase entity saving logic to use PropertyTermStore
 ResolvedAddshoreT219122 [Checkpoint 4] Refactor Write & Migration code for Items
 ResolvedNoneT221564 Create maintenance script for populating new item term persistence
 InvalidNoneT224542 Update Wikibase entity saving logic to use ItemTermStore
 Resolved• alaa_wmdeT224527 Implement database-based ItemTermStore
 ResolvedLadsgroupT219123 Migrate to and read from new store for item terms
 InvalidNoneT221715 Implement ItemTermStore in mediawiki dbal lib
 DuplicateLadsgroupT225055 Switch `tmpItemTermsMigrationStages` to MIGRATION_WRITE_BOTH
 ResolvedLadsgroupT225056 Run Item Terms Rebuild script
 DuplicateLadsgroupT225057 Switch `tmpItemTermsMigrationStages` to MIGRATION_WRITE_NEW
 ResolvedLadsgroupT234329 Investigate, fix and reimport failing item terms into new store
 ResolvedLadsgroupT232393 Find out why rebuilding some items in new term store failed
 ResolvedLadsgroupT234948 New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; try restarting transaction
 ResolvedAddshoreT235265 Slow query on SpecialMostLinked creating lag on wikidata slaves
 ResolvedAddshoreT239470 Check the success of the initial terms migration (does it have holes)
 ResolvedAddshoreT239471 Sqoop wikidata terms tables into hadoop
 ResolvedLadsgroupT240235 Alter rebuildItemTerms.php to allow passing of a a list of int item ids to process
 ResolvedAddshoreT245740 LUA getLabelWithLang calls result in many db queries to terms storage with spikey and unpredictable patterns
 ResolvedLadsgroupT246005 Wikibase client description API module results in 15k selected rows with new term storage
 ResolvedLadsgroupT246159 Setting on read new for items causes increased worst response time and number of open connections
 ResolvedLadsgroupT219301 Migrate to and read from new store for property terms
 ResolvedJun 17 2019• alaa_wmdeT224403 Update Labs replication and views to include new schema tables
 Resolved• alaa_wmdeT225038 Add new schema tables as public views in labs
 ResolvedJun 18 2019LadsgroupT225039 Add normalized tables to production
 ResolvedNoneT225040 Check if filtering is needed in labs for new term store data
 ResolvedJun 23 2019• alaa_wmdeT225051 Switch `tmpPropertyTermsMigrationStage` to MIGRATION_WRITE_BOTH
 ResolvedJun 25 2019LadsgroupT225052 Run Property Terms Rebuild script
 Resolved• alaa_wmdeT225053 Switch `tmpPropertyTermsMigrationStage` to MIGRATION_WRITE_NEW
 Resolved• alaa_wmdeT226008 Lookup obtained via SingleEntitySourceServices::getPrefetchingTermLookup must obey EntitySource
 ResolvedBUG REPORT• alaa_wmdeT230119 New term store connects to the wrong host in clients
 ResolvedLadsgroupT225054 Switch `tmpPropertyTermsMigrationStage` to MIGRATION_NEW

Event Timeline

ArielGlenn triaged this task as Medium priority.Jun 20 2019, 10:58 AM
ArielGlenn created this task.
ArielGlenn updated the task description. (Show Details)Jun 20 2019, 12:28 PM
ArielGlenn moved this task from Backlog to Active on the Dumps-Generation board.
ArielGlenn added a comment.Jun 20 2019, 12:32 PM

Some useful links:

ArielGlenn added a subscriber: Reedy.Jul 8 2019, 8:41 AM

We might be able to dump user_former_groups, change_tag_def and babel without privacy implications. Adding @Reedy for comment. (Feel free to remove yourself and/or redirect if you're not the right person to ping.)

Reedy added a comment.Jul 8 2019, 9:32 AM

All three are replicated 1:1 to labs, so I think they're fine to just be dumped too (as someone with labs access can do the same thing)

https://github.com/wikimedia/puppet/blob/eea5e40f9388c3240cee3344f31edcbe71a023a3/modules/profile/templates/labs/db/views/maintain-views.yaml

ArielGlenn added a comment.Jul 9 2019, 6:27 PM

T103011 gives some nice background on the labs-replicated databases. so adding that here.

Reedy, thanks for the comment, I'll add a patch to dump those tables starting with the next run,

gerritbot added a comment.Jul 9 2019, 6:35 PM

Change 521565 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/dumps@master] add a few more public sql tables to default list to be dumped

https://gerrit.wikimedia.org/r/521565

gerritbot added a project: Patch-For-Review.Jul 9 2019, 6:35 PM
gerritbot added a comment.Jul 9 2019, 6:39 PM

Change 521566 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] add more public sql tables to xml/sql dumps

https://gerrit.wikimedia.org/r/521566

gerritbot added a comment.Jul 19 2019, 8:06 AM

Change 521565 merged by ArielGlenn:
[operations/dumps@master] add a few more public sql tables to default list to be dumped

https://gerrit.wikimedia.org/r/521565

gerritbot added a comment.Jul 19 2019, 8:15 AM

Change 524438 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/dumps@master] add a few more public sql tables to default list to be dumped

https://gerrit.wikimedia.org/r/524438

gerritbot added a comment.Jul 19 2019, 8:35 AM

Change 524438 merged by ArielGlenn:
[operations/dumps@master] add a few more public sql tables to default list to be dumped

https://gerrit.wikimedia.org/r/524438

gerritbot added a comment.Jul 19 2019, 8:42 AM

Change 521566 merged by ArielGlenn:
[operations/puppet@production] add more public sql tables to xml/sql dumps

https://gerrit.wikimedia.org/r/521566

ArielGlenn updated the task description. (Show Details)Jul 19 2019, 9:04 AM
gerritbot added a comment.Aug 2 2019, 10:42 AM

Change 527505 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] add more public tables for xml/sql dumps

https://gerrit.wikimedia.org/r/527505

ArielGlenn added a subscriber: aaron.Aug 5 2019, 12:30 PM

@aaron This revision rEFLR848ef073fa89036c40c440016a8092690ddcf56b for FlaggedRevs seems to indicate that flaggedrevs_stats and flaggedrevs_stats2 are no longer used. Do you know or can you point me to someone who could verify that this is the case? If they aren't used, I will add them to my 'don't ever dump these' list. Thanks!

aaron added a comment.Aug 9 2019, 5:54 AM

They were obsoleted by flaggedrevs_statistics.

ArielGlenn added a comment.Aug 9 2019, 3:03 PM

Thanks a lot! I've updated the patch above to remove those entries. Now just waiting on the wb_terms migration to get further along.

ArielGlenn moved this task from Active to Blocked/Stalled/Waiting for event on the Dumps-Generation board.Aug 12 2019, 9:22 AM
ArielGlenn added a comment.Nov 9 2019, 6:24 AM

wb_terms table migration is still ongoing; check on this again in mid-December.

ArielGlenn added subtasks: T219301: Migrate to and read from new store for property terms, T219175: [Mega] - Migrate data from wb_terms to new schema.Dec 20 2019, 12:09 PM
ArielGlenn added a comment.Dec 23 2019, 7:34 AM

Some rebuilds for the wb_terms migration are taking place; I'll check again in the third week of January to see if all of the cleanup is done. We might be able to push this out for the Feb 1 run if all goes well.

ArielGlenn mentioned this in T241169: Create database dump of new Wikibase term store.Dec 23 2019, 7:42 AM
ArielGlenn merged a task: T241169: Create database dump of new Wikibase term store.Dec 23 2019, 7:44 AM
ArielGlenn added a subscriber: Bugreporter.
ArielGlenn added a comment.Jan 31 2020, 9:33 AM

After short irc chat, new estimate for wbterms migration to complete is in 3-4 weeks. I'll update this task around then.

Addshore closed subtask T219301: Migrate to and read from new store for property terms as Resolved.Mar 6 2020, 2:00 PM
Addshore closed subtask T219175: [Mega] - Migrate data from wb_terms to new schema as Resolved.Mar 23 2020, 9:17 AM
ArielGlenn added a comment.Mar 23 2020, 9:31 AM

Once the wikidata dump completes, the patches can be merged and deployed. Yay!

gerritbot added a comment.Mar 31 2020, 6:42 PM

Change 527505 merged by ArielGlenn:
[operations/puppet@production] add more public tables for xml/sql dumps

https://gerrit.wikimedia.org/r/527505

ArielGlenn closed this task as Resolved.Apr 15 2020, 10:11 AM

This is deployed, dumps have run with it, so it can be closed at last.

ArielGlenn moved this task from Blocked/Stalled/Waiting for event to Done on the Dumps-Generation board.May 20 2020, 8:09 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits