x4 needs to be redacted and added to this host
Description
Details
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T196020 Consider adding ContentTranslation (CX) tables to wiki replicas | |||
| Open | Feature | None | T395072 Add "wikishared" database to wiki replicas | ||
| Open | fnegri | T428047 Upgrade clouddb* hosts to Debian Trixie | |||
| Unknown Object (Task) | |||||
| Resolved | Jclark-ctr | T409162 Q2:rack/setup/install clouddb1026-1033 | |||
| Stalled | None | T387419 Create wiki replicas views for globaljsonlinks tables | |||
| Stalled | None | T395881 Set up x1 replication to Wiki Replicas | |||
| Duplicate | Marostegui | T408692 Set up replication on new hosts clouddb102[2-5] | |||
| Resolved | Ladsgroup | T343131 Commons database is growing way too fast | |||
| Resolved | Zabe | T398709 FY2025-26 WE 6.4.1: Move links tables of commons to a dedicated cluster | |||
| Resolved | None | T437111 Split production writes for commons link tables | |||
| Open | Marostegui | T434048 Decommission clouddb1013-clouddb1020 | |||
| Resolved | Marostegui | T409557 Productionize new clouddb* hosts (clouddb1022-1033) | |||
| Resolved | Marostegui | T404715 Setup x4 section | |||
| Resolved | Marostegui | T437838 Add x4 to an-redacteddb1001 |
Event Timeline
Change #1341115 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] an-redacteddb1001.yaml: Add x4
Change #1341115 merged by Marostegui:
[operations/puppet@production] an-redacteddb1001.yaml: Add x4
Change #1341170 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] check_private_data_report: Add x4 on an-redacteddb1001
Change #1341170 merged by Marostegui:
[operations/puppet@production] check_private_data_report: Add x4 on an-redacteddb1001
@Milimetric an-redacteddb1001 is now running x4 on port 3364.
can you check if everything looks good from your side? I don't know if the views need to be recreated/created or whatnot.
The instance is up and has data, that's great. Only thing is, linktarget should be redacted (it currently has extra data, like stuff that doesn't exist in templatelinks [1], and I guess that happens when this view is run. How are you doing that for just a subset of views? Who does this these days, DPE-SRE?
[1] you can check quickly with (edit, sorry old version was wrong) select * from linktarget where not(exists( select 1 from templatelinks where tl_target_id = lt_id ) or exists( select 1 from pagelinks where pl_target_id = lt_id )) limit 1;
So this host was copied from a clouddb1024 that was already redacted and has views - I am not sure but I think views from clouddb* are different from an-redactted (and if anything, more restrictive?).
We don't own the views creation I think it is a mixed thing between DPE and WMCS, ping @taavi @fnegri @BTullis
For what is worth:
root@an-redacteddb1001:~# check_private_data.py -S /run/mysqld/mysqld.x4.sock -- Non-public databases that are present: -- Non-public tables that are present: -- Unfiltered columns that are present: root@an-redacteddb1001:~#
btw selecting from the view on clouddb1024 and an-redacttedb
root@clouddb1024.eqiad.wmnet[commonswiki_p]> select * from linktarget where not exists (select 1 from templatelinks where tl_target_id = lt_id) limit 1; +----------+--------------+----------+ | lt_id | lt_namespace | lt_title | +----------+--------------+----------+ | 10646543 | 0 | ! | +----------+--------------+----------+ 1 row in set (0.005 sec) root@an-redacteddb1001.eqiad.wmnet[commonswiki_p]> select * from linktarget where not exists (select 1 from templatelinks where tl_target_id = lt_id) limit 1; +----------+--------------+----------+ | lt_id | lt_namespace | lt_title | +----------+--------------+----------+ | 10646543 | 0 | ! | +----------+--------------+----------+ 1 row in set (0.009 sec)
I think views from clouddb* are different from an-redactted
There should be no diff, the same maintain-views script is used to update the views on both, based on the same maintain-views.yaml from Puppet.
So @Milimetric could you confirm if things are okay from your side or do we need to bring someone else to confirm that?
Thank you!
I am going to close this as fixed as I can select fine from the views and per T437838#12318836 the should be the same as clouddb* ones.
If something is missing please do reopen.
(Sorry I had a draft reply I never sent)
This still seems to me like it's not redacting that linktarget view correctly. But I don't think it matters for the following reasons:
- we sqoop linktarget from the production replicas in all cases
- the public doesn't have query access to this an-redacteddb host
So even if this isn't set up properly it's fine. Also, we will circle back to this when we migrate all sqoops to the private cluster.
Thanks @Milimetric - I guess the views for that table needs some adjusting which is WMCS's area of expertise. If you need that fixed at some point, create a ticket with them and if you can also CC me.
Thank you!
@Milimetric the view definition is in maintain-views.yaml and was last changed in January 2026:
linktarget: source: linktarget view: select lt_id, lt_namespace, lt_title where: exists( select 1 from templatelinks where tl_target_id = lt_id ) or exists( select 1 from pagelinks where pl_target_id = lt_id )
I double-checked that both the SQL view definition in an-redacteddb and the one in clouddb102[45] are up-to-date with the definition in the yaml:
fnegri@an-redacteddb1001:~$ sudo maintain-views --table linktarget --diff-mode --sections x4 2026-09-21 13:18:20,501 INFO Processing instance 'x4' 2026-09-21 13:18:20,502 INFO Processing db 'testcommonswiki' 2026-09-21 13:18:20,592 INFO Processing db 'commonswiki' maintain-views completed on 1 instance(s) and 2 database(s) DIFF MODE enabled, no changes were made. Views that would have been created: 0 Views that would have been updated: 0 Views that would have been deleted: 0 Views with SQL errors: 0 fnegri@clouddb1024:~$ sudo maintain-views --table linktarget --diff-mode --sections x4 2026-09-21 13:22:18,196 INFO Processing instance 'x4' 2026-09-21 13:22:18,196 INFO Processing db 'testcommonswiki' 2026-09-21 13:22:18,445 INFO Processing db 'commonswiki' maintain-views completed on 1 instance(s) and 2 database(s) DIFF MODE enabled, no changes were made. Views that would have been created: 0 Views that would have been updated: 0 Views that would have been deleted: 0 Views with SQL errors: 0 fnegri@clouddb1025:~$ sudo maintain-views --table linktarget --diff-mode --sections x4 2026-09-21 13:22:28,583 INFO Processing instance 'x4' 2026-09-21 13:22:28,585 INFO Processing db 'testcommonswiki' 2026-09-21 13:22:28,939 INFO Processing db 'commonswiki' maintain-views completed on 1 instance(s) and 2 database(s) DIFF MODE enabled, no changes were made. Views that would have been created: 0 Views that would have been updated: 0 Views that would have been deleted: 0 Views with SQL errors: 0