Update the external automatic translation Oozie job
Closed, DeclinedPublic
Actions

Assigned To

Authored By

	nshahquinn-wmf
	Jan 6 2021, 3:44 PM

Description

The external automatic translation Oozie job has not been able to run since December, since Analytics Engineering updated the URL of the Hive endpoint. The last data in neilpquinn.toledo_pageviews is for 2020-12-21. Source data is deleted after 90 days, so the last day before we start losing data is 2021-03-20.

We need to update and relaunch the job. When we do this, we should also move it to the analytics/wmf-product/jobs repo, and adapt it to take advantage of the new ownership and deployment process (T267940).

Details

Due Date: Mar 16 2021, 6:30 PM

Related Objects

Mentioned In: T277781: Hive table neilpquinn.toledo_pageviews missing almost all data
Mentioned Here: T254891: Transform finalized Content Translation schema into Event Platform format
T277781: Hive table neilpquinn.toledo_pageviews missing almost all data
T246250: Sunset external automatic translation dashboard
T267940: Set up automated deployment strategy for Product Analytics job files

Event Timeline

nshahquinn-wmf created this task.Jan 6 2021, 3:44 PM

Before we do this, we should confirm that this job is actually worth keeping. There is no organizational focus on the ExternalGuidance tool anymore, and we've decided there's no need to keep the automated report based on this job (T246250). There's no huge cost to keeping it, but if we can't imagine any future use, there's no need to pay even this moderate cost.

LGoto triaged this task as Medium priority.Jan 12 2021, 6:17 PM

LGoto edited projects, added Product-Analytics (Kanban); removed Product-Analytics.Feb 9 2021, 6:21 PM

LGoto moved this task from Next 2 weeks to Needs Investigation on the Product-Analytics (Kanban) board.

nshahquinn-wmf updated the task description. (Show Details)Mar 3 2021, 3:49 PM

I consulted @Pginer-WMF and @Arrbee, and they believe it's worth continuing to collect this data. I will need to do this soon as we are approaching the point of data loss.

nshahquinn-wmf moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.Mar 17 2021, 1:48 PM

nshahquinn-wmf mentioned this in T277781: Hive table neilpquinn.toledo_pageviews missing almost all data .Mar 18 2021, 5:10 PM

Mostly done now (even if there's less point because of T277781); raising to high because (more) data loss is imminent.

The job is running again, but the daily pageview counts generated by the new job are only 20% of the ones generated by the old job, even though I didn't make any substantive change to the query. I'll need to figure this out if we want to keep producing meaningful data.

I've discussed this with @Pginer-WMF: we do want to keep this job running properly, but I'll do it after T254891/T254891.

nshahquinn-wmf moved this task from Doing to Next 2 weeks on the Product-Analytics (Kanban) board.Mar 24 2021, 12:39 PM

nshahquinn-wmf moved this task from Next 2 weeks to Needs Investigation on the Product-Analytics (Kanban) board.May 5 2021, 12:04 PM

I've discussed with Pau, and instead of doing this, we can use the edit tag data to get a rough idea of the impact that the Google Translation entry point for Content Translation has.

Update the external automatic translation Oozie job Closed, DeclinedPublicActions

Description

Details

Related Objects

Event Timeline

Update the external automatic translation Oozie job
Closed, DeclinedPublic
Actions