Page MenuHomePhabricator

audit public tables and make sure we dump them all
Open, MediumPublic0 Estimate Story Points

Description

Purely internal tables that just connect one foreign key to another are not needed but let's double check that everything else is dumped. It's been awhile.

  • P8634 lists tables broken out into groups of: dumped already, known private, cache/temp, empty on WMF, for review, and one group from which the relevant fields are dumped as xml.
  • P8774 has a smaller set of tables that aren't on all wikis.

Details

Related Objects

StatusSubtypeAssignedTask
OpenArielGlenn
OpenNone
ResolvedAddshore
Resolved alaa_wmde
Resolved alaa_wmde
Resolved alaa_wmde
ResolvedNone
DeclinedNone
Declined alaa_wmde
ResolvedLadsgroup
ResolvedAddshore
ResolvedLadsgroup
ResolvedJeroenDeDauw
DeclinedNone
ResolvedNone
ResolvedNone
ResolvedLadsgroup
ResolvedNone
InvalidNone
DeclinedNone
ResolvedLucas_Werkmeister_WMDE
InvalidNone
DeclinedNone
ResolvedLadsgroup
DeclinedNone
ResolvedJeroenDeDauw
DeclinedNone
Resolved alaa_wmde
ResolvedLucas_Werkmeister_WMDE
ResolvedLucas_Werkmeister_WMDE
ResolvedLucas_Werkmeister_WMDE
ResolvedAddshore
ResolvedNone
InvalidNone
Resolved alaa_wmde
OpenLadsgroup
InvalidNone
OpenLadsgroup
ResolvedLadsgroup
OpenLadsgroup
ResolvedLadsgroup
ResolvedLadsgroup
ResolvedLadsgroup
OpenNone
ResolvedAddshore
ResolvedAddshore
ResolvedLadsgroup
OpenLadsgroup
ResolvedJun 17 2019 alaa_wmde
Resolved alaa_wmde
ResolvedJun 18 2019Ladsgroup
ResolvedNone
ResolvedJun 23 2019 alaa_wmde
ResolvedJun 25 2019Ladsgroup
Resolved alaa_wmde
Resolved alaa_wmde
ResolvedBUG REPORT alaa_wmde
StalledNone

Event Timeline

ArielGlenn triaged this task as Medium priority.Jun 20 2019, 10:58 AM
ArielGlenn created this task.
ArielGlenn updated the task description. (Show Details)Jun 20 2019, 12:28 PM
ArielGlenn moved this task from Backlog to Active on the Dumps-Generation board.

We might be able to dump user_former_groups, change_tag_def and babel without privacy implications. Adding @Reedy for comment. (Feel free to remove yourself and/or redirect if you're not the right person to ping.)

Reedy added a comment.Jul 8 2019, 9:32 AM

All three are replicated 1:1 to labs, so I think they're fine to just be dumped too (as someone with labs access can do the same thing)

https://github.com/wikimedia/puppet/blob/eea5e40f9388c3240cee3344f31edcbe71a023a3/modules/profile/templates/labs/db/views/maintain-views.yaml

T103011 gives some nice background on the labs-replicated databases. so adding that here.

Reedy, thanks for the comment, I'll add a patch to dump those tables starting with the next run,

Change 521565 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/dumps@master] add a few more public sql tables to default list to be dumped

https://gerrit.wikimedia.org/r/521565

Change 521566 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] add more public sql tables to xml/sql dumps

https://gerrit.wikimedia.org/r/521566

Change 521565 merged by ArielGlenn:
[operations/dumps@master] add a few more public sql tables to default list to be dumped

https://gerrit.wikimedia.org/r/521565

Change 524438 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/dumps@master] add a few more public sql tables to default list to be dumped

https://gerrit.wikimedia.org/r/524438

Change 524438 merged by ArielGlenn:
[operations/dumps@master] add a few more public sql tables to default list to be dumped

https://gerrit.wikimedia.org/r/524438

Change 521566 merged by ArielGlenn:
[operations/puppet@production] add more public sql tables to xml/sql dumps

https://gerrit.wikimedia.org/r/521566

ArielGlenn updated the task description. (Show Details)Jul 19 2019, 9:04 AM

Change 527505 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] add more public tables for xml/sql dumps

https://gerrit.wikimedia.org/r/527505

@aaron This revision rEFLR848ef073fa89036c40c440016a8092690ddcf56b for FlaggedRevs seems to indicate that flaggedrevs_stats and flaggedrevs_stats2 are no longer used. Do you know or can you point me to someone who could verify that this is the case? If they aren't used, I will add them to my 'don't ever dump these' list. Thanks!

aaron added a comment.Aug 9 2019, 5:54 AM

They were obsoleted by flaggedrevs_statistics.

Thanks a lot! I've updated the patch above to remove those entries. Now just waiting on the wb_terms migration to get further along.

wb_terms table migration is still ongoing; check on this again in mid-December.

Some rebuilds for the wb_terms migration are taking place; I'll check again in the third week of January to see if all of the cleanup is done. We might be able to push this out for the Feb 1 run if all goes well.

After short irc chat, new estimate for wbterms migration to complete is in 3-4 weeks. I'll update this task around then.