Page MenuHomePhabricator
Create Task
Maniphest T399958

Inform Data-Engineering about removal of cuc_ip, cule_ip, and cupe_ip
Closed, ResolvedPublic
Actions

Description

Summary

cuc_ip, cule_ip, and cupe_ip columns are going to be dropped soon and so Data-Engineering owned code needs updating.

Background

Acceptance criteria

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Bring back the deprecated cuc_ip field in cu_changes tableanalytics/refinerymaster+3 -0
Stop using cuc_ip field in cu_changes, switch to cuc_ip_hexanalytics/refinerymaster+7 -6
Implement a GetHexDecodedIpUDF that decodes hex-encoded IPsanalytics/refinery/sourcemaster+145 -7
Update changelog.md to reflect the latest version 0.2.68analytics/refinery/sourcemaster+3 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Dreamy_Jazz created this task.Jul 18 2025, 1:48 PM
Restricted Application added a project: Trust and Safety Product Team. · View Herald TranscriptJul 18 2025, 1:48 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Milimetric triaged this task as High priority.Jul 23 2025, 4:07 PM
Milimetric moved this task from Incoming (new tickets) to Q4 2025 April 1st - June 30th on the Data-Engineering board.
Milimetric edited projects, added Data-Engineering (Q4 2025 April 1st - June 30th); removed Data-Engineering.
Milimetric subscribed.

To Do:

  • Update sqoop code
  • Update hql scripts mentioned above (look for more)
Ottomata added subscribers: amastilovic, Ahoelzl, Ottomata.Jul 23 2025, 4:08 PM

@amastilovic since you are looking into T397923: Adapt Sqoop for categorylinks schema change, could you take this too? cc @Ahoelzl

Dreamy_Jazz moved this task from Inbox to Tracking work by others on the Trust and Safety Product Team board.Jul 23 2025, 4:10 PM

Thanks for looking into this. Let me know if there is anything I can assist with.

amastilovic added a comment.Jul 23 2025, 5:35 PM
In T399958#11028559, @Ottomata wrote:

@amastilovic since you are looking into T397923: Adapt Sqoop for categorylinks schema change, could you take this too? cc @Ahoelzl

Sure!

amastilovic claimed this task.Jul 23 2025, 5:35 PM
Ahoelzl edited projects, added Data-Engineering (Q1 FY25/26 July 1st - September 30th); removed Data-Engineering (Q4 2025 April 1st - June 30th).Jul 25 2025, 10:54 PM
amastilovic added a comment.EditedJul 30 2025, 11:51 PM

OK so I've done some investigation work and here's what I think needs to be done:

  1. Modify python/refinery/sqoop.py to import cuc_ip_hex instead of cuc_ip into table mediawiki_private_cu_changes
  2. Implement a Hive UDF function formatHex that converts IPv4 and IPv6 addresses from hexadecimal representation into a regular one in base 10, just like https://gerrit.wikimedia.org/g/mediawiki/libs/IPUtils/+/33e6259971c914b826fc80bb21e68f53e0b7ccd6/src/IPUtils.php#396
  3. Modify hql/geoeditors/editors_daily_monthly.hql to use cuc_ip_hex instead of cuc_ip, with the help of the new Hive UDF formatHex function.

What do you think, @Ottomata @Milimetric ?

Ottomata added a comment.Jul 31 2025, 1:10 PM

I don't know the sqooped stuff well, but that sounds about right to me!

  1. Modify hql/geoeditors/editors_daily_monthly.hql to use cuc_ip_hex instead of cuc_ip, with the help of the new Hive UDF formatHex function.

I think your solution is better, since it keeps the mediawiki_private_cu_changes the same as upstream. But, another option might be to use the UDF to apply a transform when importing (does sqoop support that?) that save the decoded IP in the old cuc_ip field? Then editors_daily_monthly.hql would be backwards compatible?

Just putting that there in case you and Dan like it better, but I think what you have is probably best!

gerritbot added a comment.Aug 1 2025, 6:14 PM

Change #1175156 had a related patch set uploaded (by Aleksandar Mastilovic; author: Aleksandar Mastilovic):

[analytics/refinery/source@master] Implemend a GetHexDecodedIpUDF that decodes hex-encoded IPs

https://gerrit.wikimedia.org/r/1175156

gerritbot added a project: Patch-For-Review.Aug 1 2025, 6:14 PM
gerritbot added a comment.Aug 1 2025, 6:20 PM

Change #1175159 had a related patch set uploaded (by Aleksandar Mastilovic; author: Aleksandar Mastilovic):

[analytics/refinery@master] Stop using cuc_ip field in cu_changes, switch to cuc_ip_hex

https://gerrit.wikimedia.org/r/1175159

gerritbot added a comment.Aug 2 2025, 9:27 PM

Change #1175156 merged by jenkins-bot:

[analytics/refinery/source@master] Implement a GetHexDecodedIpUDF that decodes hex-encoded IPs

https://gerrit.wikimedia.org/r/1175156

amastilovic moved this task from Next Up to In progress on the Data-Engineering (Q1 FY25/26 July 1st - September 30th) board.Aug 4 2025, 3:21 PM
amastilovic moved this task from In progress to In Review on the Data-Engineering (Q1 FY25/26 July 1st - September 30th) board.
gerritbot added a comment.Aug 4 2025, 5:30 PM

Change #1175556 had a related patch set uploaded (by Aleksandar Mastilovic; author: Aleksandar Mastilovic):

[analytics/refinery/source@master] Update changelog.md to reflect the latest version 0.2.68

https://gerrit.wikimedia.org/r/1175556

gerritbot added a comment.Aug 5 2025, 2:20 PM

Change #1175556 merged by jenkins-bot:

[analytics/refinery/source@master] Update changelog.md to reflect the latest version 0.2.68

https://gerrit.wikimedia.org/r/1175556

gerritbot added a comment.Aug 5 2025, 3:44 PM

Change #1175159 merged by Aleksandar Mastilovic:

[analytics/refinery@master] Stop using cuc_ip field in cu_changes, switch to cuc_ip_hex

https://gerrit.wikimedia.org/r/1175159

gerritbot added a comment.Aug 5 2025, 4:13 PM

Change #1175918 had a related patch set uploaded (by Aleksandar Mastilovic; author: Aleksandar Mastilovic):

[analytics/refinery@master] Bring back the deprecated cuc_ip field in cu_changes table

https://gerrit.wikimedia.org/r/1175918

gerritbot added a comment.Aug 6 2025, 3:19 PM

Change #1175918 merged by Aleksandar Mastilovic:

[analytics/refinery@master] Bring back the deprecated cuc_ip field in cu_changes table

https://gerrit.wikimedia.org/r/1175918

Maintenance_bot removed a project: Patch-For-Review.Aug 6 2025, 3:31 PM
amastilovic closed this task as Resolved.Aug 11 2025, 11:15 PM
amastilovic updated the task description. (Show Details)
Ahoelzl moved this task from In Review to Done on the Data-Engineering (Q1 FY25/26 July 1st - September 30th) board.Aug 18 2025, 3:08 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits