Page MenuHomePhabricator

Update Labs replication and views to include new schema tables
Closed, ResolvedPublicJun 17 2019

Description

This must happen before the first time we switch to reading anything from the new schema, otherwise tools will not be able to use the new schema tables replicated from production, and probably will break.

We determined in T225040 that no filtering for these tables is necessary, neither in replication nor in the views.

Please proceed with the sub-tasks in the following order:

  1. T225039 Add normalized tables to production
  2. T225038 Add new schema tables as public views in labs

Details

Due Date
Jun 17 2019, 10:00 PM

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 27 2019, 8:23 AM
alaa_wmde triaged this task as Normal priority.May 27 2019, 8:27 AM
alaa_wmde set Due Date to Jun 17 2019, 10:00 PM.
alaa_wmde updated the task description. (Show Details)
Restricted Application changed the subtype of this task from "Task" to "Deadline". · View Herald TranscriptMay 27 2019, 8:27 AM

hi @Marostegui @Ladsgroup

regarding wb_terms new schema changes and migration plan (context T221764) .. I got informed by @Addshore that replication from production to labs (for tools) for the new tables might require some manual change so that those tables are included in replication.

Is there a documentation or a place where I can look to know what that change may look like?
Is this something we have to do (wb_terms team in wmde)?

For replication to labs, you need cloud services team (@Bstorm and @Andrew ) to update the views (which first needs a patch in puppet in modules/profile/templates/labs/db/views/maintain-views.yaml) then gets by cloud services team but I think the tables need to at least exist in production.

@alaa_wmde the new tables (and all its columns) will get replicated to labs unless specified otherwise, however, as @Ladsgroup points out, to actually be able to query data, you need to have the views created.
So first step would be to confirm if all the tables and all its columns should be replicated or if we need to filter things out?
This can be a good start:
https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Wiki_Replicas#Step_1:_sanitization
https://wikitech.wikimedia.org/wiki/Labsdb_redaction

Thanks for the elaborate response @Ladsgroup and @Marostegui

As far as I can see, wb_terms is listed in https://github.com/wikimedia/puppet/blob/production/modules/profile/templates/labs/db/views/maintain-views.yaml#L138, and there does not seem to be any column filtering setup on it in https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/role/files/mariadb/filtered_tables.txt.

This means our new schema tables will as well be configured to copy, without any column filtering, at least for the beginning.

I'll create a patch for the views for now, and will pass the filtering topic through the team to check before we go live

Yeah, wb_terms is replicated as is.

Lucas_Werkmeister_WMDE renamed this task from Update Labs replication to include new schema tables to Update Labs replication and views to include new schema tables.Jun 5 2019, 11:49 AM
Lucas_Werkmeister_WMDE updated the task description. (Show Details)
Restricted Application added a project: Wikidata. · View Herald TranscriptJun 18 2019, 12:43 PM
darthmon_wmde updated the task description. (Show Details)
darthmon_wmde updated the task description. (Show Details)
alaa_wmde closed this task as Resolved.Jun 19 2019, 10:05 PM
alaa_wmde claimed this task.