Using the method documented (by @aborrero, thanks!) at https://wikitech.wikimedia.org/wiki/Help:Toolforge/Mono#Debugging_a_Mono_tool, here are the C and Mono stack traces of all threads on a different host: P7172 (raw output)

On PID 4006, break monoeg_g_calloc then c, it does not break after 10+ seconds. Yes it is very likely that it is calloc spinlocking due to insufficient memory.

But why do I feel the assembly means while (1) {} P7174

It seems to me there is a high likelihood this is just T150099 striking again.

Yep, confirming insufficent memory after comparing the disassembled code and C source code.

monoeg_g_calloc maps to [[https://github.com/mono/mono/blob/9be68f8952ea0e1aad582bfe2f47bad71aee2cc7/mono/eglib/gmem.c#L109|g_calloc]]:

gpointer g_calloc (gsize n, gsize x)
{
	gpointer ptr;
	if (!x || !n)
		return 0;
		ptr = G_CALLOC_INTERNAL (n, x);
	if (ptr)
		return ptr;
	g_error ("Could not allocate %i (%i * %i) bytes", x*n, n, x);
}

[[https://github.com/mono/mono/blob/d8af77551de1f59803a46946786e0f9a5eb5cb9d/mono/eglib/glib.h#L608|g_error]] is defined macro:

#define g_error(...) do { g_log (G_LOG_DOMAIN, G_LOG_LEVEL_ERROR, __VA_ARGS__); for (;;); } while (0)

This corresponds to both the assembly above:

   0x00000000006de5db <+91>:	callq  0x6de910 <monoeg_g_log>
=> 0x00000000006de5e0 <+96>: jmp 0x6de5e0 <monoeg_g_calloc+96>

The error should have been Could not allocate ... bytes.

Thanks @Chicocvenancio for linking to that ticket (and reminding me of it).

@Kf8 since grid engine calculates memory usage by allocated size (mmap) instead of resident size, feel free to supply a high value for -mem

The use of g_erroron calloc failure started at least 8 years ago, and g_error having infinite loop also started 8 years ago, with earliest version applied as 2.8. No idea why this specific program like 3.2.8 but not 5.12.

Let's see if this is fixed by suppling a higher -mem. If so, good (hopefully it's that we used to have thinner libraries). If not, there's a memory leak and we can debug further.

Waiting for @Kf8 to confirm whether increasing the memory works or not.

No issues since I've increased the memory limit. My guess is that the new mono runtime requires a little bit more memory than the previous version, that's why 900m wasn't enough anymore. Thanks to everybody involved!

P.S. I see a lot of mono-sgen processes on grid machines consuming CPU. Somebody should probably have look at them.

In T195834#4241616, @Kf8 wrote:

P.S. I see a lot of mono-sgen processes on grid machines consuming CPU. Somebody should probably have look at them.

Basically the same issue. PID 15885 on tools-exec-1441:

(gdb) bt
#0  0x00000000006de5e0 in monoeg_g_calloc (n=n@entry=1, x=1656) at gmem.c:117
#1  0x00000000006de5fd in monoeg_malloc0 (x=<optimized out>) at gmem.c:121
#2  0x00000000006d54b4 in mono_thread_info_attach () at mono-threads.c:649
#3  0x0000000000615942 in start_wrapper (data=0x2b5e6c0050e0) at threads.c:1127
#4  0x00002b5e492d3184 in start_thread (arg=0x2b5e52893700) at pthread_create.c:312
#5  0x00002b5e497fd03d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thinking of replacing that infinite loop with abortion.

Can someone mail the cloud mailing list about this since it affects quite a lot of tools, or shall I try to figure out a list of affected tools?

Affected tools: P7180 (raw output)

zhuyifei1999@zhuyifei1999-ThinkPad-X260:~/linux$ cat T195834 | sed '/Not found/ { N; d }' | grep 'tools\.' | awk '{ print $1 }' | sort | uniq -c
    138 tools.dibot
     23 tools.mbh

CC @Dmitry89 @MaxBioHazard

Mentioned in SAL (#wikimedia-cloud) [2018-05-30T05:20:59Z] <zhuyifei1999_> qdel all grid jobs except inc_remindbot & lighttpd-dibot because they are just stuck in an infinite busy loop and consume grid CPU doing nothing T195834

I notified @Dmitry89 on-wiki.

FWIW: there's a GitHub issue about this already reported: https://github.com/mono/mono/issues/7603

Mentioned in SAL (#wikimedia-cloud) [2018-05-30T10:45:10Z] <zhuyifei1999_> installing mono-runtime-dbg on tools-bastion-05 to produce debugging information; was previously installed on tools-exec-1413 & 1441. Might be a good idea to uninstall them once we can close T195834

Mentioned in SAL (#wikimedia-cloud) [2018-06-03T10:19:24Z] <zhuyifei1999_> Grid is full. qdel'ed all jobs belonging to tools.dibot except lighttpd, and tools.mbh that has a job name starting 'comm_delin', 'delfilexcl' T195834

@Dmitry89 @MaxBioHazard Please fix your tools/bots. Add -once, increase -mem, autokill on infinte loop, or disable the entries in crontab. Filling the entire grid engine is unacceptable and I will have to do the last option if that remains the case. Of course I'd really prefer the tool/bot maintainers themselves 'fixing' it without sacrificing functionality.

All of my bots is still using my own version of mono, compiled into my personal folder, 4.8.0, not updated Toolforge version 5.12.0.

Is flag "-once" means that Toolforge will not try to re-run the task, if it was unsuccessfully run in first time?

I changed my crontab, now it is:

0 5 * * * jsub -N transnamespace_moves -mem 4G -quiet -once -v MONO_TLS_PROVIDER=btls /data/project/mbh/mono48/bin/mono /data/project/mbh/bots/trans.ns.moves.exe
50 23 * * * jsub -N orphan_files -mem 4G -once -quiet -v MONO_TLS_PROVIDER=btls /data/project/mbh/mono48/bin/mono /data/project/mbh/bots/orphane.files.exe
55 23 * * * jsub -N files_without_license -once -mem 4G -quiet -v MONO_TLS_PROVIDER=btls /data/project/mbh/mono48/bin/mono /data/project/mbh/bots/nolicensed_files.exe
0 5 * * * jsub -N transnamespace_redirects -once -mem 4G -quiet -v MONO_TLS_PROVIDER=btls /data/project/mbh/mono48/bin/mono /data/project/mbh/bots/illegal_redirects.exe
0 0 * * * jsub -N remove_curr_events -once -mem 4G -quiet -v MONO_TLS_PROVIDER=btls /data/project/mbh/mono48/bin/mono /data/project/mbh/bots/currevents_remove.exe
0 * * * * jsub -N exclude_deleted_files -once -mem 4G -quiet -v MONO_TLS_PROVIDER=btls /data/project/mbh/mono48/bin/mono /data/project/mbh/bots/exclude_deleted_files.exe
0 0 * * * jsub -N nonfree_files_in_nonmain_ns -once -mem 4G -quiet -v MONO_TLS_PROVIDER=btls /data/project/mbh/mono48/bin/mono /data/project/mbh/bots/nffiles_in_nmns.exe
0 0 1 * * jsub -N pats_awarding -once -mem 4G -quiet -v MONO_TLS_PROVIDER=btls /data/project/mbh/mono48/bin/mono /data/project/mbh/bots/pats_awarding.exe
0 * * * * jsub -N comm_delinker -once -mem 4G -quiet -v MONO_TLS_PROVIDER=btls /data/project/mbh/mono48/bin/mono /data/project/mbh/bots/mbh_delinker.exe
0 21 * * * jsub -N retired_users -once -mem 4G -quiet -v MONO_TLS_PROVIDER=btls -quiet /data/project/mbh/mono48/bin/mono /data/project/mbh/bots/retired_counter.exe
0 0 * * * jsub -N flag_removing_arch -mem 4G -once -quiet -v MONO_TLS_PROVIDER=btls /data/project/mbh/mono48/bin/mono /data/project/mbh/bots/zsf_archiving.exe
0 0 1 * * jsub -N adminstats -once -mem 4G -quiet -v MONO_TLS_PROVIDER=btls /data/project/mbh/mono48/bin/mono /data/project/mbh/bots/adminstats.exe
0 0 * * * jsub -N category_moves -quiet -once -mem 4G -v MONO_TLS_PROVIDER=btls /data/project/mbh/mono48/bin/mono /data/project/mbh/bots/catmoves.exe

Is it enough?

In T195834#4252158, @MaxBioHazard wrote:

Is flag "-once" means that Toolforge will not try to re-run the task, if it was unsuccessfully run in first time?

The -once flag is for keeping duplicate jobs from starting via cron. It tells jsub compare the job name (-N ...) with the currently running job list for the tool. If there is a match it will log to the job's .err log file and not attempt to start another copy. This flag is added automatically if you use qcronsub or jstart rather than jsub to start your job.

So yes, it should keep your crontab from launching many copies of the same job. It won't fix whatever infinite loop/soft crash condition the job is experiencing that was keeping the duplicates from exiting. You will need to figure that out separately.

In T195834#4252158, @MaxBioHazard wrote:

Is it enough?

Looks good to me. Thanks. :) Let's hope upstream fixes the infinite loop ASAP. The bug affects 4.8.0 as well apparently.

Mentioned in SAL (#wikimedia-cloud) [2018-06-05T07:39:27Z] <zhuyifei1999_> qdel-ed all jobs except lighttpd of dibot and disabled crontab (sed 's/^/#/') T195834

20:15:37 <Nemo_bis> This host is too busy with mono bots anyway ;) Reminded me to check back. Using T195834#4241876, there are 110 such processes from dibot and one from mbh (@MaxBioHazard yours is tools.mbh /mnt/nfs/labstore-secondary-tools-project/mbh/mono48/bin/mono-sgen /data/project/mbh/bots/retired_counter.exe).

Looking at dibot's crontab, it now contains -once and -mem 4G, so I'm assuming the maintainer added the args after the re-enable, and forgot to kill the affected jobs. I'll kill them.

Mentioned in SAL (#wikimedia-cloud) [2018-06-26T12:33:30Z] <zhuyifei1999_> killed all jobs besides lighttpd T195834

P7294