Sqooping from the production replicas would mean applying the same sanitization rules on our side. I see the filter here is:

where:
exists(
 select 1 from templatelinks
  where tl_target_id = lt_id 
) or
exists(
 select 1 from pagelinks
  where pl_target_id = lt_id
)

so just changing the config to sqoop from the linktarget table wouldn't allow us to add this filter. We'd also have to change the query to add this logic, and then we'd have duplicated that logic. @Ladsgroup can we get confirmation that this is necessary in this view? The table definition doesn't seem to betray any private information in any field, so it looks like maybe this was an attempt at just reducing the number of rows that are available? But it seems to have caused an even worse performance situation, so let's discuss.

Repeating from IRC for the sake of documentation:

it's for PII protection, imagine someone adds [[Real name of User:Foo is Foo Bar]] to wikis, now we have a linktarget row with that text so if removed, then in wikireplicas, it won't show up
but also, do the analytics sqoop need only absolutely public info? Isn't the whole thing private?

I guess the filtering in analytics can happen somewhere else. I need to automate the linktarget clean up as well.

may sound stupid but maybe just create a temp table (it's rather small) and sqoop that? Would that be doable? Leave that filtering to mysql basically

Alternatively, just move it to the private one and read from production.

Change 980936 had a related patch set uploaded (by Milimetric; author: Milimetric):

[operations/puppet@production] maintain-views: add note on linktarget sanitization

https://gerrit.wikimedia.org/r/980936

Change 980937 had a related patch set uploaded (by Milimetric; author: Milimetric):

[analytics/refinery@master] Sanitize the linktarget sqoop

https://gerrit.wikimedia.org/r/980937

Change 980936 merged by Ladsgroup:

[operations/puppet@production] maintain-views: add note on linktarget sanitization

https://gerrit.wikimedia.org/r/980936

Change 980939 had a related patch set uploaded (by Milimetric; author: Milimetric):

[operations/puppet@production] sqoop: move where we get the linktarget from

https://gerrit.wikimedia.org/r/980939

milimetric opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/555

Update where we look for the linktarget table

Change 980939 merged by Ladsgroup:

[operations/puppet@production] sqoop: move where we get the linktarget from

https://gerrit.wikimedia.org/r/980939

Change 981351 had a related patch set uploaded (by Milimetric; author: Milimetric):

[analytics/refinery@master] Move linktarget table to the private folder

https://gerrit.wikimedia.org/r/981351

Change 980937 merged by Milimetric:

[analytics/refinery@master] Sanitize the linktarget sqoop

https://gerrit.wikimedia.org/r/980937

Change 981351 merged by Milimetric:

[analytics/refinery@master] Move linktarget table to the private folder

https://gerrit.wikimedia.org/r/981351

milimetric merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/555

Update where we look for the linktarget table

The underlying cause: T343198#9391318

Fixing this as we speak.

Hi, If someone is around, please stop the ongoing sqoop so the schema changes could go through.