Page MenuHomePhabricator

Wikifactmine-pipeline daily import job backing up
Closed, DeclinedPublic

Description

As of this writing, 23 copies of the daily import job are actively running on the job grid:

$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
9972787 0.31694 cron-tools tools.wikifa r     06/07/2018 02:16:03 task@tools-exec-1425.tools.eqi     1
  23071 0.31619 cron-tools tools.wikifa r     06/08/2018 02:16:04 task@tools-exec-1417.tools.eqi     1
  74509 0.31544 cron-tools tools.wikifa r     06/09/2018 02:16:04 task@tools-exec-1417.tools.eqi     1
 125134 0.31469 cron-tools tools.wikifa r     06/10/2018 02:16:04 task@tools-exec-1427.tools.eqi     1
 176004 0.31394 cron-tools tools.wikifa r     06/11/2018 02:16:04 task@tools-exec-1403.eqiad.wmf     1
 227912 0.31319 cron-tools tools.wikifa r     06/12/2018 02:16:05 task@tools-exec-1432.tools.eqi     1
 279109 0.31244 cron-tools tools.wikifa r     06/13/2018 02:16:03 task@tools-exec-1409.eqiad.wmf     1
 330475 0.31169 cron-tools tools.wikifa r     06/14/2018 02:16:02 task@tools-exec-1416.tools.eqi     1
 381880 0.31094 cron-tools tools.wikifa r     06/15/2018 02:16:03 task@tools-exec-1437.tools.eqi     1
 433707 0.31019 cron-tools tools.wikifa r     06/16/2018 02:16:07 task@tools-exec-1420.tools.eqi     1
 485262 0.30944 cron-tools tools.wikifa r     06/17/2018 02:16:03 task@tools-exec-1432.tools.eqi     1
 536969 0.30869 cron-tools tools.wikifa r     06/18/2018 02:16:03 task@tools-exec-1419.tools.eqi     1
 588732 0.30794 cron-tools tools.wikifa r     06/19/2018 02:16:03 task@tools-exec-1431.tools.eqi     1
 640299 0.30719 cron-tools tools.wikifa r     06/20/2018 02:16:03 task@tools-exec-1417.tools.eqi     1
 690949 0.30644 cron-tools tools.wikifa r     06/21/2018 02:16:03 task@tools-exec-1442.tools.eqi     1
 740417 0.30569 cron-tools tools.wikifa r     06/22/2018 02:16:03 task@tools-exec-1415.tools.eqi     1
 790406 0.30494 cron-tools tools.wikifa r     06/23/2018 02:16:04 task@tools-exec-1405.eqiad.wmf     1
 840984 0.30419 cron-tools tools.wikifa r     06/24/2018 02:16:04 task@tools-exec-1406.eqiad.wmf     1
 891915 0.30344 cron-tools tools.wikifa r     06/25/2018 02:16:03 task@tools-exec-1441.tools.eqi     1
 941815 0.30269 cron-tools tools.wikifa r     06/26/2018 02:16:03 task@tools-exec-1436.tools.eqi     1
 991993 0.30194 cron-tools tools.wikifa r     06/27/2018 02:16:03 task@tools-exec-1440.tools.eqi     1
1042562 0.30119 cron-tools tools.wikifa r     06/28/2018 02:16:03 task@tools-exec-1405.eqiad.wmf     1
1092352 0.30044 cron-tools tools.wikifa r     06/29/2018 02:16:05 task@tools-exec-1427.tools.eqi     1

Event Timeline

Tarrow triaged this task as Medium priority.Jul 3 2018, 9:42 AM

Thanks, I've killed them.

Looks like WikiFactMine's elastic cluster has fallen over and that why the jobs are backing up.

Mentioned in SAL (#wikimedia-cloud) [2018-10-05T00:19:21Z] <bd808> Killed 42 duplicate cron-tools.wikifactmine-pipeline-1 jobs running on the grid (T198498)

@Tarrow as of 2018-10-05T00:14 there were 42 "cron-tools.wikifactmine-pipeline-1" job instance running with dates from 2018-07-26 to 2018-10-04. I have killed them, but I think you need to look into a) adding a -once flag for concurrency control, and b) deciding if this tool is doing useful work if its import pipeline can be broken for 40+ days without you or others noticing.

Yes, I think this may be the case. I'll look at at least stopping concurrent jobs and then contacting the Wikifactmine team and seeing if they still see benefit in running the job.

Didn't hear more back about this but the jobs were stopped in 2018