Page MenuHomePhabricator

Clean up grid jobs that are pegged to impossible resource values
Closed, ResolvedPublic

Description

It seems most of the jobs in the list are trying to run on Ubuntu Precise. This makes me wonder if there is something still submitting jobs for Precise. Determining that should be part of this task, but it is also likely that these are simply old cruft hanging around.

Jobs can not run because queue instance is not contained in its hard queue list
	7960872,	8249690,	8379533,	8465545,	8496220,	8545289,	8579489,	8610957,
	8651303,	8720723,	8748750,	8798318,	8837842,	8869569,	8953318,	8990425,
	9026621,	9073356,	9104037,	9126420,	9185721,	9219112,	9243121,	9334477,
	9424170,	9458910,	9540815,	9718125,	9748960,	9843406,	9907359,	9955137,
	196196,	262244,	507259,	520300,	624073,	3602838,	3799927,	3816501,
	3992335,	4018891,	4021038,	4040922,	4043042,	4091436,	4122897,	4144926,
	5130366,	5233561,	5259864,	5272839,	5282858,	5284719,	5375520,	5474863,
	5476921,	5486074,	5488051,	5514531,	5516476,	5522063,	5524073,	5570871,
	5581904,	5610263,	5618534,	5638325,	5650084,	5653432,	5697329,	5707411,
	5708637,	5709492,	5710716,	5716409,	5809021,	5810473,	6234830,	6294986

Jobs can not run because no host can satisfy the resource requirements
	7960872,	8249690,	8379533,	8465545,	8496220,	8545289,	8579489,	8610957,
	8651303,	8720723,	8748750,	8798318,	8837842,	8869569,	8953318,	8990425,
	9026621,	9073356,	9104037,	9126420,	9185721,	9219112,	9243121,	9334477,
	9424170,	9458910,	9540815,	9718125,	9748960,	9843406,	9907359,	9955137,
	196196,	262244,	507259,	520300,	624073,	3602838,	3799927,	3816501,
	3992335,	4018891,	4021038,	4040922,	4043042,	4091436,	4122897,	4144926,
	5130366,	5233561,	5259864,	5272839,	5282858,	5284719,	5375520,	5474863,
	5476921,	5486074,	5488051,	5514531,	5516476,	5522063,	5524073,	5570871,
	5581904,	5610263,	5618534,	5638325,	5650084,	5653432,	5697329,	5707411,
	5708637,	5709492,	5710716,	5716409,	5809021,	5810473,	6234830,	6294986

Event Timeline

A large number of these jobs belong to the drtrigonbot tool and seem to have been submitted between 2017-08-03 and 2018-05-24:

$ sudo become drtrigonbot
$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
9956245 0.37865 lighttpd-d tools.drtrig r     06/06/2018 18:33:18 webgrid-lighttpd@tools-webgrid     1
7960872 0.58000 mail.tools tools.drtrig qw    08/03/2017 00:30:36                                    1
8249690 0.57479 mail.tools tools.drtrig qw    08/10/2017 23:25:16                                    1
8379533 0.57244 mail.tools tools.drtrig qw    08/14/2017 13:41:40                                    1
8465545 0.57093 mail.tools tools.drtrig qw    08/16/2017 21:08:35                                    1
8496220 0.57040 mail.tools tools.drtrig qw    08/17/2017 16:39:48                                    1
8545289 0.56951 mail.tools tools.drtrig qw    08/19/2017 01:12:43                                    1
8579489 0.56890 mail.tools tools.drtrig qw    08/19/2017 23:27:32                                    1
8610957 0.56834 mail.tools tools.drtrig qw    08/20/2017 19:57:13                                    1
8651303 0.56768 mail.tools tools.drtrig qw    08/21/2017 20:18:16                                    1
8720723 0.56646 mail.tools tools.drtrig qw    08/23/2017 16:59:29                                    1
8748750 0.56597 mail.tools tools.drtrig qw    08/24/2017 11:08:23                                    1
8798318 0.56512 mail.tools tools.drtrig qw    08/25/2017 18:11:37                                    1
8837842 0.56440 mail.tools tools.drtrig qw    08/26/2017 20:39:55                                    1
8869569 0.56384 mail.tools tools.drtrig qw    08/27/2017 17:09:17                                    1
8953318 0.56239 mail.tools tools.drtrig qw    08/29/2017 22:31:31                                    1
8990425 0.56173 mail.tools tools.drtrig qw    08/30/2017 22:35:59                                    1
9026621 0.56111 mail.tools tools.drtrig qw    08/31/2017 21:27:29                                    1
9073356 0.56032 mail.tools tools.drtrig qw    09/02/2017 02:28:16                                    1
9104037 0.55976 mail.tools tools.drtrig qw    09/02/2017 22:40:32                                    1
9126420 0.55938 mail.tools tools.drtrig qw    09/03/2017 12:53:34                                    1
9185721 0.55836 mail.tools tools.drtrig qw    09/05/2017 02:03:19                                    1
9219112 0.55780 mail.tools tools.drtrig qw    09/05/2017 22:45:14                                    1
9243121 0.55739 mail.tools tools.drtrig qw    09/06/2017 13:40:52                                    1
9334477 0.55587 mail.tools tools.drtrig qw    09/08/2017 21:33:13                                    1
9424170 0.55438 mail.tools tools.drtrig qw    09/11/2017 04:06:33                                    1
9458910 0.55383 mail.tools tools.drtrig qw    09/12/2017 00:16:49                                    1
9540815 0.55256 mail.tools tools.drtrig qw    09/13/2017 23:00:37                                    1
9718125 0.54963 mail.tools tools.drtrig qw    09/18/2017 10:25:05                                    1
9748960 0.54912 mail.tools tools.drtrig qw    09/19/2017 05:10:11                                    1
9843406 0.54757 mail.tools tools.drtrig qw    09/21/2017 13:52:54                                    1
9907359 0.54656 mail.tools tools.drtrig qw    09/23/2017 03:04:50                                    1
9955137 0.54579 mail.tools tools.drtrig qw    09/24/2017 07:17:37                                    1
 196196 0.54185 mail.tools tools.drtrig qw    09/30/2017 07:58:13                                    1
 262244 0.54078 mail.tools tools.drtrig qw    10/01/2017 23:02:42                                    1
 507259 0.53670 mail.tools tools.drtrig qw    10/08/2017 04:45:53                                    1
 520300 0.53648 mail.tools tools.drtrig qw    10/08/2017 12:50:18                                    1
 624073 0.53476 mail.tools tools.drtrig qw    10/11/2017 04:01:18                                    1
3602838 0.41722 mail.tools tools.drtrig qw    04/08/2018 19:37:14                                    1
3799927 0.41459 mail.tools tools.drtrig qw    04/12/2018 19:58:10                                    1
3816501 0.41438 mail.tools tools.drtrig qw    04/13/2018 03:58:44                                    1
3992335 0.41207 mail.tools tools.drtrig qw    04/16/2018 16:42:48                                    1
4018891 0.41172 mail.tools tools.drtrig qw    04/17/2018 05:14:46                                    1
4021038 0.41170 mail.tools tools.drtrig qw    04/17/2018 06:14:45                                    1
4040922 0.41144 mail.tools tools.drtrig qw    04/17/2018 15:31:53                                    1
4043042 0.41142 mail.tools tools.drtrig qw    04/17/2018 16:31:52                                    1
4091436 0.41079 mail.tools tools.drtrig qw    04/18/2018 15:38:16                                    1
4122897 0.41038 mail.tools tools.drtrig qw    04/19/2018 06:23:24                                    1
4144926 0.41011 mail.tools tools.drtrig qw    04/19/2018 16:33:25                                    1
5130366 0.39976 mail.tools tools.drtrig qw    05/05/2018 12:00:00                                    1
5233561 0.39830 mail.tools tools.drtrig qw    05/07/2018 17:35:43                                    1
5259864 0.39793 mail.tools tools.drtrig qw    05/08/2018 07:16:46                                    1
5272839 0.39774 mail.tools tools.drtrig qw    05/08/2018 14:12:06                                    1
5282858 0.39759 mail.tools tools.drtrig qw    05/08/2018 19:38:03                                    1
5284719 0.39757 mail.tools tools.drtrig qw    05/08/2018 20:38:03                                    1
5375520 0.39625 mail.tools tools.drtrig qw    05/10/2018 20:43:35                                    1
5474863 0.39485 mail.tools tools.drtrig qw    05/13/2018 00:24:12                                    1
5476921 0.39482 mail.tools tools.drtrig qw    05/13/2018 01:24:11                                    1
5486074 0.39469 mail.tools tools.drtrig qw    05/13/2018 06:00:43                                    1
5488051 0.39467 mail.tools tools.drtrig qw    05/13/2018 07:00:39                                    1
5514531 0.39430 mail.tools tools.drtrig qw    05/13/2018 20:31:22                                    1
5516476 0.39427 mail.tools tools.drtrig qw    05/13/2018 21:31:22                                    1
5522063 0.39419 mail.tools tools.drtrig qw    05/14/2018 00:18:10                                    1
5524073 0.39417 mail.tools tools.drtrig qw    05/14/2018 01:18:10                                    1
5570871 0.39351 mail.tools tools.drtrig qw    05/15/2018 01:33:36                                    1
5581904 0.39336 mail.tools tools.drtrig qw    05/15/2018 07:03:36                                    1
5610263 0.39296 mail.tools tools.drtrig qw    05/15/2018 21:33:35                                    1
5618534 0.39285 mail.tools tools.drtrig qw    05/16/2018 01:26:17                                    1
5638325 0.39259 mail.tools tools.drtrig qw    05/16/2018 11:03:45                                    1
5650084 0.39243 mail.tools tools.drtrig qw    05/16/2018 17:00:24                                    1
5653432 0.39239 mail.tools tools.drtrig qw    05/16/2018 18:37:16                                    1
5697329 0.39182 mail.tools tools.drtrig qw    05/17/2018 15:28:26                                    1
5707411 0.39169 mail.tools tools.drtrig qw    05/17/2018 20:17:52                                    1
5708637 0.39167 mail.tools tools.drtrig qw    05/17/2018 20:55:43                                    1
5709492 0.39166 mail.tools tools.drtrig qw    05/17/2018 21:17:52                                    1
5710716 0.39164 mail.tools tools.drtrig qw    05/17/2018 21:55:43                                    1
5716409 0.39157 mail.tools tools.drtrig qw    05/18/2018 00:35:09                                    1
5809021 0.39035 mail.tools tools.drtrig qw    05/19/2018 21:23:37                                    1
5810473 0.39033 mail.tools tools.drtrig qw    05/19/2018 22:03:39                                    1
6234830 0.38797 mail.tools tools.drtrig qw    05/23/2018 12:45:38                                    1
6294986 0.38718 mail.tools tools.drtrig qw    05/24/2018 17:43:36                                    1

I cleaned these out of the queue:

$ qstat|grep ' qw '|awk '{print $1}'|xargs qdel
tools.drtrigonbot has deleted job 7960872
tools.drtrigonbot has deleted job 8249690
tools.drtrigonbot has deleted job 8379533
tools.drtrigonbot has deleted job 8465545
tools.drtrigonbot has deleted job 8496220
tools.drtrigonbot has deleted job 8545289
tools.drtrigonbot has deleted job 8579489
tools.drtrigonbot has deleted job 8610957
tools.drtrigonbot has deleted job 8651303
tools.drtrigonbot has deleted job 8720723
tools.drtrigonbot has deleted job 8748750
tools.drtrigonbot has deleted job 8798318
tools.drtrigonbot has deleted job 8837842
tools.drtrigonbot has deleted job 8869569
tools.drtrigonbot has deleted job 8953318
tools.drtrigonbot has deleted job 8990425
tools.drtrigonbot has deleted job 9026621
tools.drtrigonbot has deleted job 9073356
tools.drtrigonbot has deleted job 9104037
tools.drtrigonbot has deleted job 9126420
tools.drtrigonbot has deleted job 9185721
tools.drtrigonbot has deleted job 9219112
tools.drtrigonbot has deleted job 9243121
tools.drtrigonbot has deleted job 9334477
tools.drtrigonbot has deleted job 9424170
tools.drtrigonbot has deleted job 9458910
tools.drtrigonbot has deleted job 9540815
tools.drtrigonbot has deleted job 9718125
tools.drtrigonbot has deleted job 9748960
tools.drtrigonbot has deleted job 9843406
tools.drtrigonbot has deleted job 9907359
tools.drtrigonbot has deleted job 9955137
tools.drtrigonbot has deleted job 196196
tools.drtrigonbot has deleted job 262244
tools.drtrigonbot has deleted job 507259
tools.drtrigonbot has deleted job 520300
tools.drtrigonbot has deleted job 624073
tools.drtrigonbot has deleted job 3602838
tools.drtrigonbot has deleted job 3799927
tools.drtrigonbot has deleted job 3816501
tools.drtrigonbot has deleted job 3992335
tools.drtrigonbot has deleted job 4018891
tools.drtrigonbot has deleted job 4021038
tools.drtrigonbot has deleted job 4040922
tools.drtrigonbot has deleted job 4043042
tools.drtrigonbot has deleted job 4091436
tools.drtrigonbot has deleted job 4122897
tools.drtrigonbot has deleted job 4144926
tools.drtrigonbot has deleted job 5130366
tools.drtrigonbot has deleted job 5233561
tools.drtrigonbot has deleted job 5259864
tools.drtrigonbot has deleted job 5272839
tools.drtrigonbot has deleted job 5282858
tools.drtrigonbot has deleted job 5284719
tools.drtrigonbot has deleted job 5375520
tools.drtrigonbot has deleted job 5474863
tools.drtrigonbot has deleted job 5476921
tools.drtrigonbot has deleted job 5486074
tools.drtrigonbot has deleted job 5488051
tools.drtrigonbot has deleted job 5514531
tools.drtrigonbot has deleted job 5516476
tools.drtrigonbot has deleted job 5522063
tools.drtrigonbot has deleted job 5524073
tools.drtrigonbot has deleted job 5570871
tools.drtrigonbot has deleted job 5581904
tools.drtrigonbot has deleted job 5610263
tools.drtrigonbot has deleted job 5618534
tools.drtrigonbot has deleted job 5638325
tools.drtrigonbot has deleted job 5650084
tools.drtrigonbot has deleted job 5653432
tools.drtrigonbot has deleted job 5697329
tools.drtrigonbot has deleted job 5707411
tools.drtrigonbot has deleted job 5708637
tools.drtrigonbot has deleted job 5709492
tools.drtrigonbot has deleted job 5710716
tools.drtrigonbot has deleted job 5716409
tools.drtrigonbot has deleted job 5809021
tools.drtrigonbot has deleted job 5810473
tools.drtrigonbot has deleted job 6234830
tools.drtrigonbot has deleted job 6294986
$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
9956245 0.37865 lighttpd-d tools.drtrig r     06/06/2018 18:33:18 webgrid-lighttpd@tools-webgrid     1

Mentioned in SAL (#wikimedia-cloud) [2018-10-04T23:45:07Z] <bd808> Deleted queued grid jobs with invalid queue requirements (release=precise) T206272

After the cleanup in T206272#4644112, the only job output left in sudo qstat -j output is 394500 which was submitted by the botwikiawk tool on 2018-06-15. This job is in error state rather than missing a matching queue. This seems to have been a typoed test job (something like jsub echo,hello). I deleted it as well.

Mentioned in SAL (#wikimedia-cloud) [2018-10-05T00:03:34Z] <bd808> Deleted error state grid job 394500 (T206272)

bd808 claimed this task.

Just as I finished the cleanups above a new batch of jobs have shown up in qstat -j output, but these look like they should clear eventually. The are currently held by the scheduler because there is no exec node with enough free ram to execute them (5G-8G asked for). I think these will clear as other jobs finish and free resources on the grid.