The change to make some tags private will impact the sqoop job as a join is needed for every tag row touched in the public view.
To prevent this cost, we need to make the sqoop job gather the tables from production replicas.
When we do that, we also need to remove mw-private-* tags from the tags published publicly by the mediawiki-history job: T431524: Remove private tags from mediawiki-history public dumps.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • Dreamy_Jazz | T430772 DE4.1.3: Integrate personal information detection classifier into production | |||
| Resolved | MPostoronca-WMF | T431289 Update the special page displaying content policy to use the first version of the design | |||
| Resolved | ROdonoghue-WMF | T431288 Design: Create mocks for a special page to display flagged items | |||
| Resolved | • Dreamy_Jazz | T433367 Create initial special page for displaying content policy violations | |||
| Resolved | MPostoronca-WMF | T432931 Allow authorized users to add or remove the false positive tag | |||
| Resolved | MPostoronca-WMF | T431292 Tag revisions with private change tag for likely oversightable/revision deletable content | |||
| Resolved | Feature | • Dreamy_Jazz | T386456 Feature: Allow for private, rights-restricted tags | ||
| Declined | None | T431523 Update Sqoop to get `change_tag` and `change_tag_def` from production replicas |
We don't believe this change blocks enabling private tags; until we resolve this ticket, private tags will not be ingested into the Data Lake at all (see Slack discussion).
As this ticket is a performance optimization and relatively easy to do, let's wait until the next monthly Sqoop and see that the performance hit actually causes issues. When it is resolved, we will have the private tags in the Data Lake.
We also confirmed with PSI that the private tags can be viewed by WMF staff, and that's possibly desirable for analytics. So, as we do this ticket, we can keep the private tags available in the internal mediawiki_history, but redact them from the public dumps (T431524).
@GGoncalves-WMF we are moving this into Needs Clarification as we don't know when we need to do this. IIUC, nothing needs to happen until we decide we need to sqoop from private replicas, yes?
Correct, and I think the main factor for moving to private replicas is if the performance regression becomes intolerable.
I just looked this up and it only took 15 minutes to sqoop this from enwiki's cloud replica last run. So I think this will perform ok as-is. Of course we will migrate this and all other tables to sqoop from production, but later, perhaps when we migrate away from Sqoop.