Update sqoop for CheckUser table
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

None

Authored By

	Milimetric
	Jan 5 2023, 3:05 PM

Description

We will need to

get actor information from the actor table via cuc_actor
get comment information from the comment table via cuc_comment_id

By the time the source table is starting to deprecate the old cuc_user, cuc_user_text, cuc_comment fields, we may want to just move this from sqoop to Spark. But as a stop-gap, we will add some patches here that we can use if we're not ready and the columns are dropped.

Details

Subject	Repo	Branch	Lines +/-
Update cu_changes sqoop and hive table	analytics/refinery	master	+8 -1
Get comment from comment table	analytics/refinery	master	+3 -1
Get actor information from actor table	analytics/refinery	master	+4 -2

Customize query in gerrit

Related Objects
Search...

View Standalone Graph

This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.

Status	Assigned	Task
		· · ·
Resolved	Zabe	T233004 Update CheckUser for actor and comment table
Resolved	None	T326330 Update sqoop for CheckUser table
		· · ·

Event Timeline

Milimetric created this task.Jan 5 2023, 3:05 PM

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptJan 5 2023, 3:05 PM

Milimetric removed projects: MW-1.40-notes (1.40.0-wmf.18; 2023-01-09), MW-1.39-notes (1.39.0-wmf.23; 2022-08-01), MW-1.38-notes (1.38.0-wmf.26; 2022-03-14), Platform Team Workboards (Clinic Duty Team), Schema-change, CheckUser.Jan 5 2023, 3:05 PM

Milimetric removed subscribers: Rxy, Aklapper, Agabi10 and 12 others.

Change 875951 had a related patch set uploaded (by Milimetric; author: Milimetric):

[analytics/refinery@master] Get actor information from actor table

https://gerrit.wikimedia.org/r/875951

gerritbot added a project: Patch-For-Review.Jan 5 2023, 3:06 PM

Zabe removed Zabe as the assignee of this task.Jan 5 2023, 4:54 PM

Zabe subscribed.

• EChetty edited projects, added Data-Engineering-Planning; removed Data-Engineering.Jan 6 2023, 11:00 AM

• EChetty moved this task from Backlog to Pipelines on the Data-Engineering-Planning board.Jan 6 2023, 12:13 PM

• EChetty added a project: Data Pipelines.

• EChetty moved this task from Backlog to To be discussed /To be estimated on the Data Pipelines board.Jan 6 2023, 12:52 PM

• EChetty set the point value for this task to 3.Jan 17 2023, 11:41 AM

• EChetty moved this task from To be discussed /To be estimated to To be prioritised on the Data Pipelines board.Jan 17 2023, 11:46 AM

In 3 days or so the cuc_comment_id will be fully populated (it already is everywhere except wikidatawiki), thus you can also migrate to read from that instead of cuc_comment aswell.

Mayakp.wiki subscribed.Jan 17 2023, 1:35 PM

• EChetty moved this task from To be prioritised to Sprint 07 on the Data Pipelines board.Jan 17 2023, 5:00 PM

• EChetty edited projects, added Data Pipelines (Sprint 07); removed Data Pipelines.

• EChetty moved this task from Ready to Ready to Deploy on the Data Pipelines (Sprint 07) board.Jan 18 2023, 5:13 PM

In T326330#8530756, @Zabe wrote:

In 3 days or so the cuc_comment_id will be fully populated (it already is everywhere except wikidatawiki), thus you can also migrate to read from that instead of cuc_comment aswell.

cuc_comment_id is now fully populated on all wikis.

Change 883659 had a related patch set uploaded (by Milimetric; author: Milimetric):

[analytics/refinery@master] Get comment from comment table

https://gerrit.wikimedia.org/r/883659

Change 875951 merged by Joal:

[analytics/refinery@master] Get actor information from actor table

https://gerrit.wikimedia.org/r/875951

Change 883659 merged by Joal:

[analytics/refinery@master] Get comment from comment table

https://gerrit.wikimedia.org/r/883659

Change 883842 had a related patch set uploaded (by Joal; author: Joal):

[analytics/refinery@master] Update cu_changes sqoop and hive table