Maniphest T197976

Dibot running many copies of same job on job grid
Closed, ResolvedPublic
Actions

Assigned To

zhuyifei1999

Authored By

	bd808
	Jun 22 2018, 6:18 PM

Project Tags

Referenced Files

None

Subscribers

Description

The report at https://tools.wmflabs.org/grid-jobs/tool/dibot currently shows many duplicate jobs running for dibot:

Job 	Total seen 	Active 	Last seen (exit)
filemoves_replacer 	84 	31 	Currently running
inc_check 	129 	33 	Currently running
inc_image 	4 	2 	Currently running
inc_main 	7 	5 	Currently running
inc_mritog 	13 	5 	Currently running
inc_redirect_deleter 	7 	2 	Currently running
inc_remindbot 	3 	0 	2018-06-22 03:26
lighttpd-dibot 	1 	1 	Currently running
nullbot 	13 	4 	Currently running
pats-gadget 	84 	30 	Currently running
removeout 	1 	0 	2018-06-19 23:58
statbot 	3 	1 	Currently running

The crontab for this tool includes the -once flag for all of these jobs except the pats-gadget and filemoves_replacer jobs. In theory this flag should have prevented multiple jobs with the same name from starting. In practice it obviously did not.

There are two issues to address here:

Stopping extra jobs that are running to free up grid engine capacity for all tools
Understanding what could have made jsub -once fail like this

This could possibly be related to T194380: Identify bots using AES128-SHA maintainers running on toolforge and T195834: mono-based bot hangs after mono version upgrade as these jobs all include a -v MONO_TLS_PROVIDER=btls flag in the crontab.

Related Objects

Mentioned Here: T194380: Identify bots using AES128-SHA maintainers running on toolforge
T195834: mono-based bot hangs after mono version upgrade

Event Timeline

bd808 created this task.Jun 22 2018, 6:18 PM

Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptJun 22 2018, 6:18 PM

@Dmitry89 can I stop the duplicated jobs safely and expect that when the next cron triggered run starts things will pick up from where they left off? Also, can the concurrent jobs for pats-gadget and filemoves_replacer be limited using -once as well?

In T195834#4315223, @zhuyifei1999 wrote:

20:15:37 <Nemo_bis> This host is too busy with mono bots anyway ;) Reminded me to check back. Using T195834#4241876, there are 110 such processes from dibot and one from mbh (@MaxBioHazard yours is tools.mbh /mnt/nfs/labstore-secondary-tools-project/mbh/mono48/bin/mono-sgen /data/project/mbh/bots/retired_counter.exe).

Looking at dibot's crontab, it now contains -once and -mem 4G, so I'm assuming the maintainer added the args after the re-enable, and forgot to kill the affected jobs. I'll kill them.