Page MenuHomePhabricator

Update Labs replication and views to include new schema tables
Closed, ResolvedPublicJun 17 2019

Description

This must happen before the first time we switch to reading anything from the new schema, otherwise tools will not be able to use the new schema tables replicated from production, and probably will break.

We determined in T225040 that no filtering for these tables is necessary, neither in replication nor in the views.

Please proceed with the sub-tasks in the following order:

  1. T225039 Add normalized tables to production
  2. T225038 Add new schema tables as public views in labs

Details

Due Date
Jun 17 2019, 10:00 PM

Related Objects

StatusSubtypeAssignedTask
ResolvedAddshore
ResolvedArielGlenn
ResolvedAddshore
ResolvedAddshore
Resolved alaa_wmde
Resolved alaa_wmde
Resolved alaa_wmde
ResolvedNone
DeclinedNone
Declined alaa_wmde
ResolvedLadsgroup
ResolvedAddshore
ResolvedLadsgroup
ResolvedJeroenDeDauw
DeclinedNone
ResolvedNone
ResolvedNone
ResolvedLadsgroup
ResolvedNone
InvalidNone
DeclinedNone
ResolvedLucas_Werkmeister_WMDE
InvalidNone
DeclinedNone
ResolvedLadsgroup
DeclinedNone
ResolvedJeroenDeDauw
DeclinedNone
Resolved alaa_wmde
ResolvedLucas_Werkmeister_WMDE
ResolvedLucas_Werkmeister_WMDE
ResolvedLucas_Werkmeister_WMDE
ResolvedAddshore
ResolvedNone
InvalidNone
Resolved alaa_wmde
ResolvedLadsgroup
InvalidNone
DuplicateLadsgroup
ResolvedLadsgroup
DuplicateLadsgroup
ResolvedLadsgroup
ResolvedLadsgroup
ResolvedLadsgroup
ResolvedAddshore
ResolvedAddshore
ResolvedAddshore
ResolvedLadsgroup
ResolvedAddshore
ResolvedLadsgroup
ResolvedLadsgroup
ResolvedLadsgroup
ResolvedJun 17 2019 alaa_wmde
Resolved alaa_wmde
ResolvedJun 18 2019Ladsgroup
ResolvedNone

Event Timeline

alaa_wmde set Due Date to Jun 17 2019, 10:00 PM.
alaa_wmde updated the task description. (Show Details)
Restricted Application changed the subtype of this task from "Task" to "Deadline". · View Herald TranscriptMay 27 2019, 8:27 AM

hi @Marostegui @Ladsgroup

regarding wb_terms new schema changes and migration plan (context T221764) .. I got informed by @Addshore that replication from production to labs (for tools) for the new tables might require some manual change so that those tables are included in replication.

Is there a documentation or a place where I can look to know what that change may look like?
Is this something we have to do (wb_terms team in wmde)?

For replication to labs, you need cloud services team (@Bstorm and @Andrew ) to update the views (which first needs a patch in puppet in modules/profile/templates/labs/db/views/maintain-views.yaml) then gets by cloud services team but I think the tables need to at least exist in production.

@alaa_wmde the new tables (and all its columns) will get replicated to labs unless specified otherwise, however, as @Ladsgroup points out, to actually be able to query data, you need to have the views created.
So first step would be to confirm if all the tables and all its columns should be replicated or if we need to filter things out?
This can be a good start:
https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Wiki_Replicas#Step_1:_sanitization
https://wikitech.wikimedia.org/wiki/Labsdb_redaction

Thanks for the elaborate response @Ladsgroup and @Marostegui

As far as I can see, wb_terms is listed in https://github.com/wikimedia/puppet/blob/production/modules/profile/templates/labs/db/views/maintain-views.yaml#L138, and there does not seem to be any column filtering setup on it in https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/role/files/mariadb/filtered_tables.txt.

This means our new schema tables will as well be configured to copy, without any column filtering, at least for the beginning.

I'll create a patch for the views for now, and will pass the filtering topic through the team to check before we go live

Yeah, wb_terms is replicated as is.

Lucas_Werkmeister_WMDE renamed this task from Update Labs replication to include new schema tables to Update Labs replication and views to include new schema tables.Jun 5 2019, 11:49 AM
Lucas_Werkmeister_WMDE updated the task description. (Show Details)
darthmon_wmde updated the task description. (Show Details)
darthmon_wmde updated the task description. (Show Details)