Page MenuHomePhabricator

Migrate data-engineering jobs to mw-cron
Closed, ResolvedPublic

Description

Migrate Data-Engineering periodic mediawiki jobs from mwmaint to mw-cron on kubernetes.

Job nameCriticalityDone?
mediawiki_job_ImageSuggestions_SendNotificationsForUnillustratedWatchedTitles_CA.timerY
mediawiki_job_ImageSuggestions_SendNotificationsForUnillustratedWatchedTitles_FI.timerY
mediawiki_job_ImageSuggestions_SendNotificationsForUnillustratedWatchedTitles_HU.timerY
mediawiki_job_ImageSuggestions_SendNotificationsForUnillustratedWatchedTitles_ID.timerY
mediawiki_job_ImageSuggestions_SendNotificationsForUnillustratedWatchedTitles_NO.timery
mediawiki_job_ImageSuggestions_SendNotificationsForUnillustratedWatchedTitles_PT.timerY
mediawiki_job_ImageSuggestions_SendNotificationsForUnillustratedWatchedTitles_RU.timerY

Doc on the new platform

ServiceOps will handle migrating the jobs, but would appreciate input from Data-Engineering on:

  • jobs that should be watched more
  • jobs that are low criticality and could be migrated first
  • outdated jobs that can be removed
  • any potential gotchas in the way these jobs use MediaWiki

Event Timeline

I was looking at setting up T385709: Periodic job alerting, and realized you've already been onboarded to alertmanager and there are alert receivers for structured-data via email, and for data-engineering via mail or tasks.
What would you like the cronjob failure alerts to be routed through? I can also set-up a task receiver for structured-data.

... there are alert receivers for structured-data via email...

structured-data response:
I'm not sure how to interpret this - I don't think we receive any alerts for these jobs ATM (but we do for other things)
Anyway, it would be great if we could get email alerts at sd-alerts@lists.wikimedia.org
Thanks!

You're correct that you do not receive alerts from these jobs at the moment. I meant that there is a already a way for you to receive alerts in alertmanager, indeed to sd-alerts@lists.wikimedia.org.

I was just checking if you wanted alerts for the jobs *once they've been migrated* to go to that address, or if you wanted me to have the alert create phabricator tasks.

Am I correct in thinking that only Structured Data Engineering is interested in these alerts and not Data-Engineering ?

Just checking back in - would it be possible to get these jobs evaluated using the matrix in the ticket body please? If possible we'd like to get started on migrating them as soon as is possible. Thanks!

Change #1140671 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] mw::maintenance: migrate one image suggestions job to k8s

https://gerrit.wikimedia.org/r/1140671

Change #1140672 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] mw::maintenance: migrate all image suggestions jobs to k8s

https://gerrit.wikimedia.org/r/1140672

Change #1140671 merged by Hnowlan:

[operations/puppet@production] mw::maintenance: migrate one image suggestions job to k8s

https://gerrit.wikimedia.org/r/1140671

The cawiki image suggestions job has been migrated and ran this morning, exiting successfully. However, the job itself emits no output (seemingly in common with the existing metal jobs) - is there an easy way to verify that this job is behaving as expected (graphs, logs in other services, on-wiki artifacts)?

The cawiki image suggestions job has been migrated and ran this morning, exiting successfully. However, the job itself emits no output (seemingly in common with the existing metal jobs) - is there an easy way to verify that this job is behaving as expected (graphs, logs in other services, on-wiki artifacts)?

We will be migrating the remaining image suggestion jobs this week, please let us know if we should block this.

Change #1140672 merged by Hnowlan:

[operations/puppet@production] mw::maintenance: migrate all image suggestions jobs to k8s

https://gerrit.wikimedia.org/r/1140672

hnowlan claimed this task.
hnowlan updated the task description. (Show Details)