Page MenuHomePhabricator
Create Task
Maniphest T397923

Adapt Sqoop for categorylinks schema change
Closed, ResolvedPublic
Actions

Description

This will need to be updated to be compatible with the categorylinks schema changes.

Done is:

  • cl_target_id and cl_collation_id added to Sqoop's categorylinks table definition in python/refinery/sqoop.py
  • Columns cl_to and cl_collation dropped from Sqoop's categorylinks table definition
  • Table collation added to Sqoop
  • Table collation added to Sqoop's Puppet script invocation definition
  • Table categorylinks CREATE HQL script updated with two new columns added
  • Table categorylinks CREATE HQL script updated with two columns dropped
  • Table wmf_raw.wikimedia_categorylinks updated in Hive with two new columns (ALTER TABLE ADD COLUMN...)
  • Table wmf_raw.wikimedia_collation created in Hive
  • Table wmf_raw.wikimedia_collation added to mediawiki history load DAG
  • Table wmf_raw.wikimedia_collation added to bin/refinery-drop-mediawiki-snapshots script

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Fix wmf_raw.categorylinks column typesanalytics/refinerymaster+2 -2
Set cl_to and cl_collation columns in categorylinks to NULLanalytics/refinerymaster+4 -4
Add wmf_raw.mediawiki_collation table to snapshot maintenanceanalytics/refinerymaster+1 -0
Add collation to the list of sqooped tableoperations/puppetproduction+1 -1
Update Sqoop's schema for the categorylinks tableanalytics/refinerymaster+54 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

Zabe created this task.Jun 26 2025, 1:03 PM
phuedx removed a project: Test Kitchen.Jun 27 2025, 9:06 AM
Ottomata added subscribers: Milimetric, Ahoelzl, mforns, JAllemandou.Jul 15 2025, 4:00 PM
Ottomata triaged this task as High priority.Jul 16 2025, 4:26 PM
Ottomata added subscribers: xcollazo, Ottomata.Jul 16 2025, 4:28 PM

@xcollazo , Dan and I are grooming, and this needs to happen soon. We'd like to spread a bit of knowledge around, would you be willing to make this happen? @Milimetric can help.

Ottomata moved this task from Incoming (new tickets) to Q4 2025 April 1st - June 30th on the Data-Engineering board.Jul 16 2025, 4:28 PM
Ottomata edited projects, added Data-Engineering (Q4 2025 April 1st - June 30th); removed Data-Engineering.
Ahoelzl moved this task from Next Up to Urgent on the Data-Engineering (Q4 2025 April 1st - June 30th) board.Jul 16 2025, 6:18 PM
xcollazo claimed this task.Jul 21 2025, 2:08 PM
In T397923#11010124, @Ottomata wrote:

@xcollazo , Dan and I are grooming, and this needs to happen soon. We'd like to spread a bit of knowledge around, would you be willing to make this happen? @Milimetric can help.

No problem.

Ahoelzl reassigned this task from xcollazo to amastilovic.Jul 21 2025, 6:31 PM
Ottomata mentioned this in T399958: Inform Data-Engineering about removal of cuc_ip, cule_ip, and cupe_ip.Jul 23 2025, 4:08 PM
Ahoelzl edited projects, added Data-Engineering (Q1 FY25/26 July 1st - September 30th); removed Data-Engineering (Q4 2025 April 1st - June 30th).Jul 25 2025, 10:55 PM
Ahoelzl moved this task from Next Up to Urgent on the Data-Engineering (Q1 FY25/26 July 1st - September 30th) board.
gerritbot added a comment.Jul 30 2025, 9:46 PM

Change #1174563 had a related patch set uploaded (by Aleksandar Mastilovic; author: Aleksandar Mastilovic):

[analytics/refinery@master] Update Sqoop's schema for the categorylinks table

https://gerrit.wikimedia.org/r/1174563

gerritbot added a project: Patch-For-Review.Jul 30 2025, 9:46 PM
amastilovic moved this task from Urgent to In progress on the Data-Engineering (Q1 FY25/26 July 1st - September 30th) board.Aug 4 2025, 3:21 PM
amastilovic moved this task from In progress to In Review on the Data-Engineering (Q1 FY25/26 July 1st - September 30th) board.
gerritbot added a comment.Aug 5 2025, 5:31 PM

Change #1175924 had a related patch set uploaded (by Aleksandar Mastilovic; author: Aleksandar Mastilovic):

[operations/puppet@production] Add collation to the list of sqooped table

https://gerrit.wikimedia.org/r/1175924

gerritbot added a comment.Aug 6 2025, 3:19 PM

Change #1174563 merged by Aleksandar Mastilovic:

[analytics/refinery@master] Update Sqoop's schema for the categorylinks table

https://gerrit.wikimedia.org/r/1174563

gerritbot added a comment.Aug 7 2025, 3:03 PM

Change #1175924 merged by Btullis:

[operations/puppet@production] Add collation to the list of sqooped table

https://gerrit.wikimedia.org/r/1175924

Maintenance_bot removed a project: Patch-For-Review.Aug 7 2025, 3:31 PM
CodeReviewBot added a project: Patch-For-Review.Aug 7 2025, 5:53 PM

amastilovic opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1598

Add table collation to mediawiki history load DAG

JAllemandou added a comment.Aug 11 2025, 11:02 AM

I approved the MR above.
And actually while we're at it, we should review if the list here https://github.com/wikimedia/analytics-refinery/blob/master/bin/refinery-drop-mediawiki-snapshots#L85 matches the list of tables we sqoop!

amastilovic added a comment.Aug 11 2025, 2:46 PM
In T397923#11073329, @JAllemandou wrote:

I approved the MR above.
And actually while we're at it, we should review if the list here https://github.com/wikimedia/analytics-refinery/blob/master/bin/refinery-drop-mediawiki-snapshots#L85 matches the list of tables we sqoop!

Thank you! Re: the "drop mediawiki snapshots" thing, the new collation table is not on the list, but I'm not quite sure if it should be.

JAllemandou added a comment.Aug 11 2025, 3:01 PM
In T397923#11074113, @amastilovic wrote:

Re: the "drop mediawiki snapshots" thing, the new collation table is not on the list, but I'm not quite sure if it should be.

It should be indeed, as any other table we get through sqoop. We don't want to keep those snapshots forever :)

gerritbot added a comment.Aug 11 2025, 9:37 PM

Change #1177507 had a related patch set uploaded (by Aleksandar Mastilovic; author: Aleksandar Mastilovic):

[analytics/refinery@master] Add wmf_raw.mediawiki_collation table to snapshot maintenance

https://gerrit.wikimedia.org/r/1177507

gerritbot added a comment.Aug 12 2025, 4:05 PM

Change #1177507 merged by Aleksandar Mastilovic:

[analytics/refinery@master] Add wmf_raw.mediawiki_collation table to snapshot maintenance

https://gerrit.wikimedia.org/r/1177507

amastilovic closed this task as Resolved.Aug 12 2025, 9:04 PM
amastilovic updated the task description. (Show Details)
Ahoelzl moved this task from In Review to Done on the Data-Engineering (Q1 FY25/26 July 1st - September 30th) board.Aug 18 2025, 3:08 PM
Ottomata mentioned this in T402925: Drop cl_to and cl_collation from categorylinks in wmf production.Aug 27 2025, 4:10 PM
amastilovic reopened this task as In Progress.Aug 27 2025, 5:31 PM
amastilovic updated the task description. (Show Details)
gerritbot added a comment.Aug 28 2025, 4:15 PM

Change #1182877 had a related patch set uploaded (by Aleksandar Mastilovic; author: Aleksandar Mastilovic):

[analytics/refinery@master] Set cl_to and cl_collation columns in categorylinks to NULL

https://gerrit.wikimedia.org/r/1182877

gerritbot added a comment.Aug 28 2025, 4:24 PM

Change #1182877 merged by Aleksandar Mastilovic:

[analytics/refinery@master] Set cl_to and cl_collation columns in categorylinks to NULL

https://gerrit.wikimedia.org/r/1182877

amastilovic closed this task as Resolved.Aug 28 2025, 7:03 PM
amastilovic updated the task description. (Show Details)
BTullis closed subtask T403702: Errors from refinery-sqoop-whole-mediawiki.service - September 2025 - re: categorylinks as Resolved.Sep 4 2025, 4:10 PM
gerritbot added a comment.Sep 5 2025, 5:30 PM

Change #1185169 had a related patch set uploaded (by Aleksandar Mastilovic; author: Aleksandar Mastilovic):

[analytics/refinery@master] Fix wmf_raw.categorylinks column types

https://gerrit.wikimedia.org/r/1185169

gerritbot added a comment.Sep 8 2025, 3:07 PM

Change #1185169 merged by Aleksandar Mastilovic:

[analytics/refinery@master] Fix wmf_raw.categorylinks column types

https://gerrit.wikimedia.org/r/1185169

amastilovic mentioned this in T404735: Fix CommonsCategoryGraphBuilder to reflect latest changes to categorylinks table.Sep 16 2025, 4:01 PM
GGoncalves-WMF mentioned this in T416481: Adapt Sqoop for imagelinks schema changes.Feb 4 2026, 2:32 PM
fnegri mentioned this in T417492: Drop support for cl_to, cl_collation and il_to from wikireplicas.Feb 26 2026, 11:56 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits