Page MenuHomePhabricator

wikitasks mono bot crashes when running on grid engine with OutOfMemoryException
Open, Needs TriagePublicBUG REPORT

Description

On my Toolforge account wikitasks every day bot wp_cyrlat is executed.
It is launched with jsub -once -j y -mem 1536m... options and uses mono from Toolforge (v5.12.0.226 right now).
Some of its runs ends with OutOfMemoryException, which looks strange since memory usage of my bot is somewhere around 50 megabytes.
Do anyone know what can be the reason of such problem (my program, Mono, Linux, other Toolforge software) or, maybe, can fix it?
Log of its output is attached to this report.


Latest launches are made with modified GZipUnpack function, which prints used memory size, packed data size and unpacked data size.

public static string GZipUnpack(byte[] data)
{
    Console.Write($"[{GC.GetTotalMemory(false)/1024/1024}/{data.Length}/");
    Console.Out.Flush();
    using (var msi = new MemoryStream(data))
    {
        using (var gzs = new GZipStream(msi, CompressionMode.Decompress))
        {
            using (var mso = new MemoryStream())
            {
                gzs.CopyTo(mso);
                var msoa = mso.ToArray();
                Console.Write(msoa.Length + "]");
                Console.Out.Flush();
                return Encoding.UTF8.GetString(msoa);
            }
        }
    }
}

So [11/60383/ means that 11 megabytes of RAM was used and unpacking of 59 kilobytes request was failed.
(Please refrain from comments on my coding style, technical decisions or selected language)

Event Timeline

Vort created this task.Oct 24 2020, 5:18 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 24 2020, 5:18 AM
bd808 renamed this task from Bot on Toolforge crashes with OutOfMemoryException to wikitasks mono bot crashes grid engine with OutOfMemoryException.Oct 26 2020, 4:02 PM
bd808 added a project: Tools.
bd808 added a subscriber: bd808.Oct 26 2020, 4:48 PM

@Vort have you tried increasing the memory limit for the jsub command? I don't have much experience with mono in Toolforge, but I do know that the java JVM needs a much larger vmem limit than the -mem 1536m value that you are showing for your bot. I would suggest trying to run your bot with -mem 4g to see if having a lot more allowed memory helps. If it does, you could then try a bisection search to find a lower, but still large enough, memory limit to use longer term.

T219351: Java jobs run the Stretch grid seem to require a very large memory reservation could be sort of related.

Other than allocating more ram, I would suggest investigating what objects your code is keeping in the heap which are not able to be cleaned up by the mono garbage collector.

bd808 renamed this task from wikitasks mono bot crashes grid engine with OutOfMemoryException to wikitasks mono bot crashes when running on grid engine with OutOfMemoryException.Oct 26 2020, 4:49 PM
Vort added a comment.Oct 26 2020, 5:11 PM

Thanks. I will try 4g. But it is, of course, not a solution.
If 50 MB program requires 4 GB, then 500 MB program will just fail.
So it is better to find a real cause.

The thing is that my program stores almost no data in memory!
Look at line #151. It is the largest object, which contains ~500 strings (article titles).
Other objects are temporary. And since at 1 iteration ~600 KB of data is processed, program simply can't use 1.5 GB.

bd808 added a comment.Oct 26 2020, 5:30 PM

Other objects are temporary. And since at 1 iteration ~600 KB of data is processed, program simply can't use 1.5 GB.

"Temporary" objects are actually quite commonly the cause of heap bloat in garbage collected languages. Again I am not a mono expert, but in my distant past I was a java expert (circa 2000-2007). The sgen garbage collector in mono is similar to java's generational garbage collector. Both runtimes try to make best guesses about how to size the various "generations" or "edens" which are used to control where heap objects are allocated in order to find a balance between runtime performance (minimizing GC pauses) and heap size (minimizing allocated ram footprint). These guesses are just guesses however and often need a lot of research to tune the gc for different workloads. https://www.mono-project.com/docs/advanced/garbage-collector/sgen/ has some information on tuning the mono garbage collector. The "MONO_GC_PARAMS" and "MONO_GC_DEBUG" sections of man mono may be helpful as well.

Vort added a comment.Oct 26 2020, 5:44 PM

Look at attached wp_cyrlat.out.
.[11/61730/273265].[14/59280/265817].[20/61519/276691].[10/59246/261464].[13/62278/273618].[19/61048/271265].[26/52771/256674].[10/63377/277635].[16/61853/273123].[23/59548/278272].[8/64609/282540].[10/61280/265858]
means that program had these states: ..., 11 MB, 14 MB, 20 MB, GC(?), 10, 13, 19, 26, GC(?), 10, 16, 23, GC(?), 8, 10, ...
So temporary objects was cleared every 3-4 iterations.

qstat also confirms such behaviour. I never saw consumption higher than 80 MB. Differences from debug data can come from additional memory requirements for storing of system library code.

-mem to grid is VMS, which may or may not be proportional to RSS.

If you can send SIGSTOP to the process (to pause it) right before OutOfMemoryException, I can dump the VM mapping table and we can check if something wild is happening.

Vort added a comment.EditedOct 27 2020, 7:57 AM

@zhuyifei1999:
I have tried lower memory settings, 256m:
qstat -j 1637509:
...
hard resource_list: h_vmem=262144k
...
usage 1: cpu=00:13:37, mem=41.73038 GB s, io=0.00016 GB, vmem=55.043M, maxvmem=55.043M
...

And now it automagically set itself to the glitch state.
It is not OutOfMemoryException, but looks like the cause is the same.

Output is just
[2020.10.27 07:49:17] Task started
Authenticating...

I don't know how to pause it on the grid.
If you can, then, please, do it, I will left it in hung state.

If another test is needed, then let me know.
I think that 256m test is good, but I can also run 512m etc.

Vort added a comment.Oct 27 2020, 8:19 AM

Activity of this bot have almost no side effects, so it can be launched at any time and interrupted also at any time.
(It updates table on ruwiki, but if table is already updated, it will do nothing).
Only requirement is to have auth.txt file next to it with OAuth keys of bot account (ConsumerToken, ConsumerSecret, AccessToken, AccessSecret) (lines #22-25 of MwApi.cs).

I don't know how to pause it on the grid.

I did a googling and https://stackoverflow.com/a/35809358 shows a way to send signals to self in mono. Rather than sending SIGABRT, send SIGSTOP and that will programmatically pause the process from within.

Vort added a comment.EditedOct 27 2020, 9:27 AM

Ok, I can try this. But, of course, not before exception, but after it.
There are nothing special about the time when it happens.
So there are two possibilities: to pause it at random time, or after exception.
Results should not differ.

But first of all, I want to know what you (or I) will do with Job #1637509?
Is it useful for debugging?

I also can try to make minimized program, which triggers hang (or exception, depends on luck) with 256m (without using all of it, of course).

But first of all, I want to know what you (or I) will do with Job #1637509?
Is it useful for debugging?

As I understand, 1637509 is unexpected incomplete output, correct? Its VSZ is at capacity:

10:55:12 0 ✓ zhuyifei1999@tools-sgeexec-0906: ~$ ps uf -u tools.wikitasks
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
tools.w+ 13605  0.0  0.0  11144  2772 ?        Ss   07:49   0:00 /bin/bash /mnt/nfs/labstore-secondary-tools-project/wikitasks/wrap.sh wp_cyrlat
tools.w+ 13609  199  0.6 255996 53592 ?        Sl   07:49 1805:05  \_ mono WikiTasks.exe

I guess I can look into this one instead. The VM mapping table:

111:12:53 0 ✓ zhuyifei1999@tools-sgeexec-0906: ~$ sudo cat /proc/13609/maps
24112c000-4113c000 rwxp 00000000 00:00 0
341a34000-41ac4000 rwxp 00000000 00:00 0
42b362a4ca000-2b362a4ed000 r-xp 00000000 fe:03 1068233 /lib/x86_64-linux-gnu/ld-2.24.so
52b362a4ed000-2b362a4ee000 r--p 00000000 fe:03 31272 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
62b362a4ee000-2b362a4f5000 r--s 00000000 fe:03 964014 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
72b362a4f5000-2b362a4f6000 r--p 00000000 fe:03 1187051 /usr/lib/locale/chr_US/LC_MEASUREMENT
82b362a4f6000-2b362a4f7000 r--p 00000000 fe:03 1187057 /usr/lib/locale/chr_US/LC_TELEPHONE
92b362a4f7000-2b362a4f8000 r--p 00000000 fe:03 31271 /usr/lib/locale/en_US.utf8/LC_ADDRESS
102b362a4f8000-2b362a4f9000 r--p 00000000 fe:03 1187055 /usr/lib/locale/chr_US/LC_NAME
112b362a4f9000-2b362a4fa000 r--p 00000000 fe:03 1187056 /usr/lib/locale/chr_US/LC_PAPER
122b362a4fa000-2b362a4fb000 r--p 00000000 fe:03 1187340 /usr/lib/locale/en_AG/LC_MESSAGES/SYS_LC_MESSAGES
132b362a4fb000-2b362a4fc000 r--p 00000000 fe:03 1187054 /usr/lib/locale/chr_US/LC_MONETARY
142b362a4fc000-2b362a4fd000 r--p 00000000 fe:03 31274 /usr/lib/locale/en_US.utf8/LC_TIME
152b362a4fd000-2b362a4fe000 r--p 00000000 fe:03 1056157 /usr/lib/locale/aa_ET/LC_NUMERIC
162b362a4fe000-2b362a4ff000 rw-s 00000000 00:13 165322760 /dev/shm/mono.13609
172b362a4ff000-2b362a508000 rw-p 00000000 00:00 0
182b362a508000-2b362a50e000 rw-p 00000000 00:00 0
192b362a50e000-2b362a63e000 r--p 00000000 fe:03 1056124 /usr/lib/locale/aa_DJ.utf8/LC_COLLATE
202b362a63e000-2b362a68f000 r--p 00000000 fe:03 1056125 /usr/lib/locale/aa_DJ.utf8/LC_CTYPE
212b362a68f000-2b362a694000 rw-p 00000000 00:00 0
222b362a694000-2b362a699000 r--p 00000000 00:31 91167686 /mnt/nfs/labstore-secondary-tools-project/wikitasks/wp_cyrlat/WikiTasks.exe
232b362a699000-2b362a6ed000 rw-p 00000000 00:00 0
242b362a6ed000-2b362a6ee000 r--p 00023000 fe:03 1068233 /lib/x86_64-linux-gnu/ld-2.24.so
252b362a6ee000-2b362a6ef000 rw-p 00024000 fe:03 1068233 /lib/x86_64-linux-gnu/ld-2.24.so
262b362a6ef000-2b362a6f0000 rw-p 00000000 00:00 0
272b362a6f0000-2b362a7f3000 r-xp 00000000 fe:03 1068241 /lib/x86_64-linux-gnu/libm-2.24.so
282b362a7f3000-2b362a9f2000 ---p 00103000 fe:03 1068241 /lib/x86_64-linux-gnu/libm-2.24.so
292b362a9f2000-2b362a9f3000 r--p 00102000 fe:03 1068241 /lib/x86_64-linux-gnu/libm-2.24.so
302b362a9f3000-2b362a9f4000 rw-p 00103000 fe:03 1068241 /lib/x86_64-linux-gnu/libm-2.24.so
312b362a9f4000-2b362a9fb000 r-xp 00000000 fe:03 1068254 /lib/x86_64-linux-gnu/librt-2.24.so
322b362a9fb000-2b362abfa000 ---p 00007000 fe:03 1068254 /lib/x86_64-linux-gnu/librt-2.24.so
332b362abfa000-2b362abfb000 r--p 00006000 fe:03 1068254 /lib/x86_64-linux-gnu/librt-2.24.so
342b362abfb000-2b362abfc000 rw-p 00007000 fe:03 1068254 /lib/x86_64-linux-gnu/librt-2.24.so
352b362abfc000-2b362abff000 r-xp 00000000 fe:03 1068240 /lib/x86_64-linux-gnu/libdl-2.24.so
362b362abff000-2b362adfe000 ---p 00003000 fe:03 1068240 /lib/x86_64-linux-gnu/libdl-2.24.so
372b362adfe000-2b362adff000 r--p 00002000 fe:03 1068240 /lib/x86_64-linux-gnu/libdl-2.24.so
382b362adff000-2b362ae00000 rw-p 00003000 fe:03 1068240 /lib/x86_64-linux-gnu/libdl-2.24.so
392b362ae00000-2b362ae18000 r-xp 00000000 fe:03 1068252 /lib/x86_64-linux-gnu/libpthread-2.24.so
402b362ae18000-2b362b017000 ---p 00018000 fe:03 1068252 /lib/x86_64-linux-gnu/libpthread-2.24.so
412b362b017000-2b362b018000 r--p 00017000 fe:03 1068252 /lib/x86_64-linux-gnu/libpthread-2.24.so
422b362b018000-2b362b019000 rw-p 00018000 fe:03 1068252 /lib/x86_64-linux-gnu/libpthread-2.24.so
432b362b019000-2b362b01d000 rw-p 00000000 00:00 0
442b362b01d000-2b362b033000 r-xp 00000000 fe:03 1048593 /lib/x86_64-linux-gnu/libgcc_s.so.1
452b362b033000-2b362b232000 ---p 00016000 fe:03 1048593 /lib/x86_64-linux-gnu/libgcc_s.so.1
462b362b232000-2b362b233000 r--p 00015000 fe:03 1048593 /lib/x86_64-linux-gnu/libgcc_s.so.1
472b362b233000-2b362b234000 rw-p 00016000 fe:03 1048593 /lib/x86_64-linux-gnu/libgcc_s.so.1
482b362b234000-2b362b3c9000 r-xp 00000000 fe:03 1068237 /lib/x86_64-linux-gnu/libc-2.24.so
492b362b3c9000-2b362b5c9000 ---p 00195000 fe:03 1068237 /lib/x86_64-linux-gnu/libc-2.24.so
502b362b5c9000-2b362b5cd000 r--p 00195000 fe:03 1068237 /lib/x86_64-linux-gnu/libc-2.24.so
512b362b5cd000-2b362b5cf000 rw-p 00199000 fe:03 1068237 /lib/x86_64-linux-gnu/libc-2.24.so
522b362b5cf000-2b362b5d3000 rw-p 00000000 00:00 0
532b362b5d3000-2b362b5d4000 rw-p 00000000 00:00 0
542b362b5d4000-2b362b633000 ---p 00000000 00:00 0
552b362b633000-2b362b6e6000 rw-p 00000000 00:00 0
562b362b6e6000-2b362b703000 r--p 00000000 fe:03 12644 /usr/lib/mono/gac/System.Net.Http/4.0.0.0__b03f5f7f11d50a3a/System.Net.Http.dll
572b362b703000-2b362b7fb000 r--p 00000000 fe:03 548549 /usr/lib/mono/gac/System.Core/4.0.0.0__b77a5c561934e089/System.Core.dll
582b362b7fb000-2b362b800000 rw-p 00000000 00:00 0
592b362b800000-2b362bc00000 rw-p 00000000 00:00 0
602b362bc00000-2b362bc01000 ---p 00000000 00:00 0
612b362bc01000-2b362ce01000 rw-p 00000000 00:00 0
622b362ce01000-2b362d1cc000 r--p 00000000 fe:03 286041 /usr/lib/mono/4.5/mscorlib.dll
632b362d1cc000-2b362d734000 r-xp 00000000 fe:03 429878 /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
642b362d734000-2b362d933000 ---p 00568000 fe:03 429878 /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
652b362d933000-2b362d934000 r--p 00567000 fe:03 429878 /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
662b362d934000-2b362d935000 rw-p 00568000 fe:03 429878 /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
672b362d935000-2b362d95b000 rw-p 00000000 00:00 0
682b362d95b000-2b362da5b000 rw-p 00000000 00:00 0
692b362da5b000-2b362da5c000 ---p 00000000 00:00 0
702b362da5c000-2b362da5d000 rw-p 00000000 00:00 0
712b362da5d000-2b362da65000 ---p 00000000 00:00 0
722b362da65000-2b362dc5c000 rw-p 00000000 00:00 0
732b362dc5c000-2b362df07000 r--p 00000000 fe:03 283131 /usr/lib/mono/gac/System/4.0.0.0__b77a5c561934e089/System.dll
742b362df07000-2b362df25000 r--p 00000000 fe:03 283124 /usr/lib/mono/gac/System.Configuration/4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
752b362df25000-2b362e227000 r--p 00000000 fe:03 282191 /usr/lib/mono/gac/System.Xml/4.0.0.0__b77a5c561934e089/System.Xml.dll
762b362e227000-2b362e273000 r--p 00000000 fe:03 283117 /usr/lib/mono/gac/Mono.Security/4.0.0.0__0738eb9f132ed756/Mono.Security.dll
772b362e273000-2b362e300000 rw-p 00000000 00:00 0
782b362e300000-2b362e700000 rw-p 00000000 00:00 0
792b362e700000-2b362e701000 ---p 00000000 00:00 0
802b362e701000-2b362e702000 rw-p 00000000 00:00 0
812b362e702000-2b362e70a000 ---p 00000000 00:00 0
822b362e70a000-2b362ea01000 rw-p 00000000 00:00 0
832b362ea01000-2b362ea02000 ---p 00000000 00:00 0
842b362ea02000-2b362ea03000 rw-p 00000000 00:00 0
852b362ea03000-2b362ea0b000 ---p 00000000 00:00 0
862b362ea0b000-2b362ec02000 rw-p 00000000 00:00 0
872b362ec02000-2b362ec03000 ---p 00000000 00:00 0
882b362ec03000-2b362ec04000 rw-p 00000000 00:00 0
892b362ec04000-2b362ec0c000 ---p 00000000 00:00 0
902b362ec0c000-2b362ee03000 rw-p 00000000 00:00 0
912b362ee03000-2b362ee04000 ---p 00000000 00:00 0
922b362ee04000-2b362ee05000 rw-p 00000000 00:00 0
932b362ee05000-2b362ee0d000 ---p 00000000 00:00 0
942b362ee0d000-2b362f1f5000 rw-p 00000000 00:00 0
952b362f1f5000-2b362f2ce000 rw-p 00000000 00:00 0
962b362f2ce000-2b362f3d5000 rw-p 00000000 00:00 0
972b362f3d5000-2b362f3db000 rw-p 00000000 00:00 0
982b362f3db000-2b362f45d000 rw-p 00000000 00:00 0
992b362f45d000-2b362f462000 rw-p 00000000 00:00 0
1002b362f462000-2b362f4bf000 rw-p 00000000 00:00 0
1012b362f4bf000-2b362f645000 rw-p 00000000 00:00 0
1022b362f645000-2b362f64e000 rw-p 00000000 00:00 0
1032b362f64e000-2b362f7e5000 rw-p 00000000 00:00 0
1042b362f7e5000-2b362f7ef000 r-xp 00000000 fe:03 1068247 /lib/x86_64-linux-gnu/libnss_files-2.24.so
1052b362f7ef000-2b362f9ef000 ---p 0000a000 fe:03 1068247 /lib/x86_64-linux-gnu/libnss_files-2.24.so
1062b362f9ef000-2b362f9f0000 r--p 0000a000 fe:03 1068247 /lib/x86_64-linux-gnu/libnss_files-2.24.so
1072b362f9f0000-2b362f9f1000 rw-p 0000b000 fe:03 1068247 /lib/x86_64-linux-gnu/libnss_files-2.24.so
1082b362f9f1000-2b362f9f7000 rw-p 00000000 00:00 0
1092b362f9f7000-2b362f9fc000 r-xp 00000000 fe:03 1068246 /lib/x86_64-linux-gnu/libnss_dns-2.24.so
1102b362f9fc000-2b362fbfb000 ---p 00005000 fe:03 1068246 /lib/x86_64-linux-gnu/libnss_dns-2.24.so
1112b362fbfb000-2b362fbfc000 r--p 00004000 fe:03 1068246 /lib/x86_64-linux-gnu/libnss_dns-2.24.so
1122b362fbfc000-2b362fbfd000 rw-p 00005000 fe:03 1068246 /lib/x86_64-linux-gnu/libnss_dns-2.24.so
1132b362fbfd000-2b362fc11000 r-xp 00000000 fe:03 1068253 /lib/x86_64-linux-gnu/libresolv-2.24.so
1142b362fc11000-2b362fe10000 ---p 00014000 fe:03 1068253 /lib/x86_64-linux-gnu/libresolv-2.24.so
1152b362fe10000-2b362fe11000 r--p 00013000 fe:03 1068253 /lib/x86_64-linux-gnu/libresolv-2.24.so
1162b362fe11000-2b362fe12000 rw-p 00014000 fe:03 1068253 /lib/x86_64-linux-gnu/libresolv-2.24.so
1172b362fe12000-2b362fe14000 rw-p 00000000 00:00 0
1182b362fe14000-2b362fe24000 rw-p 00000000 00:00 0
1192b362fe24000-2b362fe43000 rw-p 00000000 00:00 0
1202b362fe43000-2b3630000000 rw-p 00000000 00:00 0
1212b3630000000-2b3630021000 rw-p 00000000 00:00 0
1222b3630021000-2b3634000000 ---p 00000000 00:00 0
1232b3634000000-2b3634021000 rw-p 00000000 00:00 0
1242b3634021000-2b3638000000 ---p 00000000 00:00 0
1252b3638000000-2b3638006000 r-xp 00000000 fe:03 1056417 /lib/x86_64-linux-gnu/libnss_db-2.2.3.so
1262b3638006000-2b3638205000 ---p 00006000 fe:03 1056417 /lib/x86_64-linux-gnu/libnss_db-2.2.3.so
1272b3638205000-2b3638206000 r--p 00005000 fe:03 1056417 /lib/x86_64-linux-gnu/libnss_db-2.2.3.so
1282b3638206000-2b3638207000 rw-p 00006000 fe:03 1056417 /lib/x86_64-linux-gnu/libnss_db-2.2.3.so
1292b3638207000-2b36383bb000 r-xp 00000000 fe:03 917757 /usr/lib/x86_64-linux-gnu/libdb-5.3.so
1302b36383bb000-2b36385bb000 ---p 001b4000 fe:03 917757 /usr/lib/x86_64-linux-gnu/libdb-5.3.so
1312b36385bb000-2b36385c2000 r--p 001b4000 fe:03 917757 /usr/lib/x86_64-linux-gnu/libdb-5.3.so
1322b36385c2000-2b36385c5000 rw-p 001bb000 fe:03 917757 /usr/lib/x86_64-linux-gnu/libdb-5.3.so
1332b36385c5000-2b36387a9000 rw-p 00000000 00:00 0
1342b36387a9000-2b36387aa000 ---p 00000000 00:00 0
1352b36387aa000-2b36387ab000 rw-p 00000000 00:00 0
1362b36387ab000-2b36387b3000 ---p 00000000 00:00 0
1372b36387b3000-2b3638a0d000 rw-p 00000000 00:00 0
1382b3638a0d000-2b3638a0e000 ---p 00000000 00:00 0
1392b3638a0e000-2b3638a0f000 rw-p 00000000 00:00 0
1402b3638a0f000-2b3638a17000 ---p 00000000 00:00 0
1412b3638a17000-2b3638cbd000 rw-p 00000000 00:00 0
1422b3638cbd000-2b3638cbe000 ---p 00000000 00:00 0
1432b3638cbe000-2b3638cbf000 rw-p 00000000 00:00 0
1442b3638cbf000-2b3638cc7000 ---p 00000000 00:00 0
1452b3638cc7000-2b3639084000 rw-p 00000000 00:00 0
1462b3639084000-2b3639250000 rw-p 00000000 00:00 0
1472b3639251000-2b36392be000 rw-p 00000000 00:00 0
1482b36392be000-2b36393a4000 rw-p 00000000 00:00 0
1492b36393a4000-2b36393ef000 rw-p 00000000 00:00 0
1502b36393ef000-2b36394fd000 rw-p 00000000 00:00 0
1512b36394fd000-2b3639529000 rw-p 00000000 00:00 0
1522b3639529000-2b3639554000 rw-p 00000000 00:00 0
1532b3639554000-2b36395a1000 rw-p 00000000 00:00 0
1542b36395a1000-2b363963b000 rw-p 00000000 00:00 0
1552b363963b000-2b36396d0000 rw-p 00000000 00:00 0
1562b36396d0000-2b363975a000 rw-p 00000000 00:00 0
1572b363975a000-2b3639835000 rw-p 00000000 00:00 0
1582b3639835000-2b363994b000 rw-p 00000000 00:00 0
15955f54ff25000-55f550352000 r-xp 00000000 fe:03 963742 /usr/bin/mono-sgen
16055f550552000-55f550559000 r--p 0042d000 fe:03 963742 /usr/bin/mono-sgen
16155f550559000-55f55055d000 rw-p 00434000 fe:03 963742 /usr/bin/mono-sgen
16255f55055d000-55f550574000 rw-p 00000000 00:00 0
16355f55158f000-55f5515f2000 rw-p 00000000 00:00 0 [heap]
16455f5515f3000-55f5515fb000 ---p 00000000 00:00 0
1657ffd12743000-7ffd12764000 rw-p 00000000 00:00 0 [stack]
1667ffd127b3000-7ffd127b5000 r--p 00000000 00:00 0 [vvar]
1677ffd127b5000-7ffd127b7000 r-xp 00000000 00:00 0 [vdso]
168ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

Sorted by size of each memory map:

111:13:30 0 ✓ zhuyifei1999@tools-sgeexec-0906: ~$ sudo cat /proc/13609/maps | awk '{ split($1, a, "-"); printf "%s %08x %s\n", a[1], strtonum("0x"a[2]) - strtonum("0x"a[1]), $6 }' | sort -k 2
22b362a6ef000 00001000
32b362b5d3000 00001000
42b362bc00000 00001000
52b362da5b000 00001000
62b362da5c000 00001000
72b362e700000 00001000
82b362e701000 00001000
92b362ea01000 00001000
102b362ea02000 00001000
112b362ec02000 00001000
122b362ec03000 00001000
132b362ee03000 00001000
142b362ee04000 00001000
152b36387a9000 00001000
162b36387aa000 00001000
172b3638a0d000 00001000
182b3638a0e000 00001000
192b3638cbd000 00001000
202b3638cbe000 00001000
212b362a4fe000 00001000 /dev/shm/mono.13609
222b362a6ed000 00001000 /lib/x86_64-linux-gnu/ld-2.24.so
232b362a6ee000 00001000 /lib/x86_64-linux-gnu/ld-2.24.so
242b362adfe000 00001000 /lib/x86_64-linux-gnu/libdl-2.24.so
252b362adff000 00001000 /lib/x86_64-linux-gnu/libdl-2.24.so
262b362b232000 00001000 /lib/x86_64-linux-gnu/libgcc_s.so.1
272b362b233000 00001000 /lib/x86_64-linux-gnu/libgcc_s.so.1
282b362a9f2000 00001000 /lib/x86_64-linux-gnu/libm-2.24.so
292b362a9f3000 00001000 /lib/x86_64-linux-gnu/libm-2.24.so
302b3638205000 00001000 /lib/x86_64-linux-gnu/libnss_db-2.2.3.so
312b3638206000 00001000 /lib/x86_64-linux-gnu/libnss_db-2.2.3.so
322b362fbfb000 00001000 /lib/x86_64-linux-gnu/libnss_dns-2.24.so
332b362fbfc000 00001000 /lib/x86_64-linux-gnu/libnss_dns-2.24.so
342b362f9ef000 00001000 /lib/x86_64-linux-gnu/libnss_files-2.24.so
352b362f9f0000 00001000 /lib/x86_64-linux-gnu/libnss_files-2.24.so
362b362b017000 00001000 /lib/x86_64-linux-gnu/libpthread-2.24.so
372b362b018000 00001000 /lib/x86_64-linux-gnu/libpthread-2.24.so
382b362fe10000 00001000 /lib/x86_64-linux-gnu/libresolv-2.24.so
392b362fe11000 00001000 /lib/x86_64-linux-gnu/libresolv-2.24.so
402b362abfa000 00001000 /lib/x86_64-linux-gnu/librt-2.24.so
412b362abfb000 00001000 /lib/x86_64-linux-gnu/librt-2.24.so
422b362a4fd000 00001000 /usr/lib/locale/aa_ET/LC_NUMERIC
432b362a4f5000 00001000 /usr/lib/locale/chr_US/LC_MEASUREMENT
442b362a4fb000 00001000 /usr/lib/locale/chr_US/LC_MONETARY
452b362a4f8000 00001000 /usr/lib/locale/chr_US/LC_NAME
462b362a4f9000 00001000 /usr/lib/locale/chr_US/LC_PAPER
472b362a4f6000 00001000 /usr/lib/locale/chr_US/LC_TELEPHONE
482b362a4fa000 00001000 /usr/lib/locale/en_AG/LC_MESSAGES/SYS_LC_MESSAGES
492b362a4f7000 00001000 /usr/lib/locale/en_US.utf8/LC_ADDRESS
502b362a4ed000 00001000 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
512b362a4fc000 00001000 /usr/lib/locale/en_US.utf8/LC_TIME
522b362d933000 00001000 /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
532b362d934000 00001000 /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
54ffffffffff600000 00001000 [vsyscall]
552b362fe12000 00002000
562b362b5cd000 00002000 /lib/x86_64-linux-gnu/libc-2.24.so
577ffd127b5000 00002000 [vdso]
587ffd127b3000 00002000 [vvar]
592b362abfc000 00003000 /lib/x86_64-linux-gnu/libdl-2.24.so
602b36385c2000 00003000 /usr/lib/x86_64-linux-gnu/libdb-5.3.so
612b362b019000 00004000
622b362b5cf000 00004000
632b362b5c9000 00004000 /lib/x86_64-linux-gnu/libc-2.24.so
6455f550559000 00004000 /usr/bin/mono-sgen
652b362a68f000 00005000
662b362b7fb000 00005000
672b362f45d000 00005000
682b362f9f7000 00005000 /lib/x86_64-linux-gnu/libnss_dns-2.24.so
692b362a694000 00005000 /mnt/nfs/labstore-secondary-tools-project/wikitasks/wp_cyrlat/WikiTasks.exe
702b362a508000 00006000
712b362f3d5000 00006000
722b362f9f1000 00006000
732b3638000000 00006000 /lib/x86_64-linux-gnu/libnss_db-2.2.3.so
742b362a9f4000 00007000 /lib/x86_64-linux-gnu/librt-2.24.so
7555f550552000 00007000 /usr/bin/mono-sgen
762b362a4ee000 00007000 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
772b36385bb000 00007000 /usr/lib/x86_64-linux-gnu/libdb-5.3.so
782b362da5d000 00008000
792b362e702000 00008000
802b362ea03000 00008000
812b362ec04000 00008000
822b362ee05000 00008000
832b36387ab000 00008000
842b3638a0f000 00008000
852b3638cbf000 00008000
8655f5515f3000 00008000
872b362a4ff000 00009000
882b362f645000 00009000
892b362f7e5000 0000a000 /lib/x86_64-linux-gnu/libnss_files-2.24.so
902b362fe14000 00010000
914112c000 00010000
922b362fbfd000 00014000 /lib/x86_64-linux-gnu/libresolv-2.24.so
932b362b01d000 00016000 /lib/x86_64-linux-gnu/libgcc_s.so.1
9455f55055d000 00017000
952b362ae00000 00018000 /lib/x86_64-linux-gnu/libpthread-2.24.so
962b362b6e6000 0001d000 /usr/lib/mono/gac/System.Net.Http/4.0.0.0__b03f5f7f11d50a3a/System.Net.Http.dll
972b362df07000 0001e000 /usr/lib/mono/gac/System.Configuration/4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
982b362fe24000 0001f000
992b3630000000 00021000
1002b3634000000 00021000
1017ffd12743000 00021000 [stack]
1022b362a4ca000 00023000 /lib/x86_64-linux-gnu/ld-2.24.so
1032b362d935000 00026000
1042b3639529000 0002b000
1052b36394fd000 0002c000
1062b36393a4000 0004b000
1072b362e227000 0004c000 /usr/lib/mono/gac/Mono.Security/4.0.0.0__0738eb9f132ed756/Mono.Security.dll
1082b3639554000 0004d000
1092b362a63e000 00051000 /usr/lib/locale/aa_DJ.utf8/LC_CTYPE
1102b362a699000 00054000
1112b362f462000 0005d000
1122b362b5d4000 0005f000
11355f55158f000 00063000 [heap]
1142b3639251000 0006d000
1152b362f3db000 00082000
1162b36396d0000 0008a000
1172b362e273000 0008d000
11841a34000 00090000
1192b363963b000 00095000
1202b36395a1000 0009a000
1212b362b633000 000b3000
1222b362f1f5000 000d9000
1232b363975a000 000db000
1242b36392be000 000e6000
1252b362b703000 000f8000 /usr/lib/mono/gac/System.Core/4.0.0.0__b77a5c561934e089/System.Core.dll
1262b362d95b000 00100000
1272b362a6f0000 00103000 /lib/x86_64-linux-gnu/libm-2.24.so
1282b362f2ce000 00107000
1292b36393ef000 0010e000
1302b3639835000 00116000
1312b362a50e000 00130000 /usr/lib/locale/aa_DJ.utf8/LC_COLLATE
1322b362f4bf000 00186000
1332b362b234000 00195000 /lib/x86_64-linux-gnu/libc-2.24.so
1342b362f64e000 00197000
1352b3638207000 001b4000 /usr/lib/x86_64-linux-gnu/libdb-5.3.so
1362b362fe43000 001bd000
1372b3639084000 001cc000
1382b36385c5000 001e4000
1392b362da65000 001f7000
1402b362ea0b000 001f7000
1412b362ec0c000 001f7000
1422b362abff000 001ff000 /lib/x86_64-linux-gnu/libdl-2.24.so
1432b362b033000 001ff000 /lib/x86_64-linux-gnu/libgcc_s.so.1
1442b362a7f3000 001ff000 /lib/x86_64-linux-gnu/libm-2.24.so
1452b3638006000 001ff000 /lib/x86_64-linux-gnu/libnss_db-2.2.3.so
1462b362f9fc000 001ff000 /lib/x86_64-linux-gnu/libnss_dns-2.24.so
1472b362ae18000 001ff000 /lib/x86_64-linux-gnu/libpthread-2.24.so
1482b362fc11000 001ff000 /lib/x86_64-linux-gnu/libresolv-2.24.so
1492b362a9fb000 001ff000 /lib/x86_64-linux-gnu/librt-2.24.so
1502b362d734000 001ff000 /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
1512b362b3c9000 00200000 /lib/x86_64-linux-gnu/libc-2.24.so
1522b362f7ef000 00200000 /lib/x86_64-linux-gnu/libnss_files-2.24.so
1532b36383bb000 00200000 /usr/lib/x86_64-linux-gnu/libdb-5.3.so
1542b36387b3000 0025a000
1552b3638a17000 002a6000
1562b362dc5c000 002ab000 /usr/lib/mono/gac/System/4.0.0.0__b77a5c561934e089/System.dll
1572b362e70a000 002f7000
1582b362df25000 00302000 /usr/lib/mono/gac/System.Xml/4.0.0.0__b77a5c561934e089/System.Xml.dll
1592b3638cc7000 003bd000
1602b362ce01000 003cb000 /usr/lib/mono/4.5/mscorlib.dll
1612b362ee0d000 003e8000
1622b362b800000 00400000
1632b362e300000 00400000
16455f54ff25000 0042d000 /usr/bin/mono-sgen
1652b362d1cc000 00568000 /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
1662b362bc01000 01200000
1672b3630021000 03fdf000
1682b3634021000 03fdf000

The sum of the maps does indicate the exhaustion:

11:14:43 0 ✓ zhuyifei1999@tools-sgeexec-0906: ~$ sudo cat /proc/13609/maps | awk '{ split($1, a, "-"); sum += strtonum("0x"a[2]) - strtonum("0x"a[1]) } END { print sum }'
262144000

The largest are one 18MiB read-writable map at address 0x2b362bc01000, and two non-readable, non-writable 63.9MiB maps at addresses 0x2b3630021000 and 0x2b3634021000. The last two non-readable, non-writable maps accounts for 51.1% of total VMS use, which begs the question, what are they for?

I'll do a core dump of this process to check its memory contents.

Saved:

11:21:24 0 ✓ zhuyifei1999@tools-sgeexec-0906: ~$ sudo gdb -p 13609 -batch -ex 'generate-core-file ~tools.wikitasks/T266377.core'
[New LWP 13610]
[New LWP 13611]
[New LWP 13612]
[New LWP 13613]
[New LWP 13614]
[New LWP 13615]
[New LWP 13616]
[New LWP 13617]
[New LWP 13618]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00002b362ae0d17f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
warning: target file /proc/13609/cmdline contained unexpected null characters
Saved corefile /data/project/wikitasks/T266377.core
11:29:53 0 ✓ zhuyifei1999@tools-sgeexec-0906: ~$ ls -l /data/project/wikitasks/T266377.core
-rw-r--r-- 1 root tools.wikitasks 208923592 Oct 27 23:29 /data/project/wikitasks/T266377.core
11:30:16 0 ✓ zhuyifei1999@tools-sgeexec-0906: ~$ sudo chmod 640 /data/project/wikitasks/T266377.core
11:30:41 0 ✓ zhuyifei1999@tools-sgeexec-0906: ~$ ls -l /data/project/wikitasks/T266377.core
-rw-r----- 1 root tools.wikitasks 208923592 Oct 27 23:29 /data/project/wikitasks/T266377.core

Coredump layout:

111:30:43 0 ✓ zhuyifei1999@tools-sgeexec-0906: ~$ sudo readelf -eW /data/project/wikitasks/T266377.core
2ELF Header:
3 Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
4 Class: ELF64
5 Data: 2's complement, little endian
6 Version: 1 (current)
7 OS/ABI: UNIX - System V
8 ABI Version: 0
9 Type: CORE (Core file)
10 Machine: Advanced Micro Devices X86-64
11 Version: 0x1
12 Entry point address: 0x0
13 Start of program headers: 64 (bytes into file)
14 Start of section headers: 208915784 (bytes into file)
15 Flags: 0x0
16 Size of this header: 64 (bytes)
17 Size of program headers: 56 (bytes)
18 Number of program headers: 120
19 Size of section headers: 64 (bytes)
20 Number of section headers: 122
21 Section header string table index: 121
22
23Section Headers:
24 [Nr] Name Type Address Off Size ES Flg Lk Inf Al
25 [ 0] NULL 0000000000000000 000000 000000 00 0 0 0
26 [ 1] note0 NOTE 0000000000000000 001a80 0052b0 00 A 0 0 1
27 [ 2] load PROGBITS 000000004112c000 006d30 010000 00 WAX 0 0 1
28 [ 3] load PROGBITS 0000000041a34000 016d30 090000 00 WAX 0 0 1
29 [ 4] load PROGBITS 00002b362a4ca000 0a6d30 023000 00 AX 0 0 1
30 [ 5] load PROGBITS 00002b362a4ff000 0c9d30 009000 00 WA 0 0 1
31 [ 6] load PROGBITS 00002b362a508000 0d2d30 006000 00 WA 0 0 1
32 [ 7] load PROGBITS 00002b362a68f000 0d8d30 005000 00 WA 0 0 1
33 [ 8] load PROGBITS 00002b362a699000 0ddd30 054000 00 WA 0 0 1
34 [ 9] load PROGBITS 00002b362a6ed000 131d30 001000 00 A 0 0 1
35 [10] load PROGBITS 00002b362a6ee000 132d30 001000 00 WA 0 0 1
36 [11] load PROGBITS 00002b362a6ef000 133d30 001000 00 WA 0 0 1
37 [12] load PROGBITS 00002b362a9f2000 134d30 001000 00 A 0 0 1
38 [13] load PROGBITS 00002b362a9f3000 135d30 001000 00 WA 0 0 1
39 [14] load PROGBITS 00002b362abfa000 136d30 001000 00 A 0 0 1
40 [15] load PROGBITS 00002b362abfb000 137d30 001000 00 WA 0 0 1
41 [16] load PROGBITS 00002b362adfe000 138d30 001000 00 A 0 0 1
42 [17] load PROGBITS 00002b362adff000 139d30 001000 00 WA 0 0 1
43 [18] load PROGBITS 00002b362b017000 13ad30 001000 00 A 0 0 1
44 [19] load PROGBITS 00002b362b018000 13bd30 001000 00 WA 0 0 1
45 [20] load PROGBITS 00002b362b019000 13cd30 004000 00 WA 0 0 1
46 [21] load PROGBITS 00002b362b232000 140d30 001000 00 A 0 0 1
47 [22] load PROGBITS 00002b362b233000 141d30 001000 00 WA 0 0 1
48 [23] load PROGBITS 00002b362b5c9000 142d30 004000 00 A 0 0 1
49 [24] load PROGBITS 00002b362b5cd000 146d30 002000 00 WA 0 0 1
50 [25] load PROGBITS 00002b362b5cf000 148d30 004000 00 WA 0 0 1
51 [26] load PROGBITS 00002b362b5d3000 14cd30 001000 00 WA 0 0 1
52 [27] load PROGBITS 00002b362b5d4000 14dd30 05f000 00 A 0 0 1
53 [28] load PROGBITS 00002b362b633000 1acd30 0b3000 00 WA 0 0 1
54 [29] load PROGBITS 00002b362b7fb000 25fd30 005000 00 WA 0 0 1
55 [30] load PROGBITS 00002b362b800000 264d30 400000 00 WA 0 0 1
56 [31] load PROGBITS 00002b362bc00000 664d30 001000 00 A 0 0 1
57 [32] load PROGBITS 00002b362bc01000 665d30 1200000 00 WA 0 0 1
58 [33] load PROGBITS 00002b362d933000 1865d30 001000 00 A 0 0 1
59 [34] load PROGBITS 00002b362d934000 1866d30 001000 00 WA 0 0 1
60 [35] load PROGBITS 00002b362d935000 1867d30 026000 00 WA 0 0 1
61 [36] load PROGBITS 00002b362d95b000 188dd30 100000 00 WA 0 0 1
62 [37] load PROGBITS 00002b362da5b000 198dd30 001000 00 A 0 0 1
63 [38] load PROGBITS 00002b362da5c000 198ed30 001000 00 WA 0 0 1
64 [39] load PROGBITS 00002b362da5d000 198fd30 008000 00 A 0 0 1
65 [40] load PROGBITS 00002b362da65000 1997d30 1f7000 00 WA 0 0 1
66 [41] load PROGBITS 00002b362e273000 1b8ed30 08d000 00 WA 0 0 1
67 [42] load PROGBITS 00002b362e300000 1c1bd30 400000 00 WA 0 0 1
68 [43] load PROGBITS 00002b362e700000 201bd30 001000 00 A 0 0 1
69 [44] load PROGBITS 00002b362e701000 201cd30 001000 00 WA 0 0 1
70 [45] load PROGBITS 00002b362e702000 201dd30 008000 00 A 0 0 1
71 [46] load PROGBITS 00002b362e70a000 2025d30 2f7000 00 WA 0 0 1
72 [47] load PROGBITS 00002b362ea01000 231cd30 001000 00 A 0 0 1
73 [48] load PROGBITS 00002b362ea02000 231dd30 001000 00 WA 0 0 1
74 [49] load PROGBITS 00002b362ea03000 231ed30 008000 00 A 0 0 1
75 [50] load PROGBITS 00002b362ea0b000 2326d30 1f7000 00 WA 0 0 1
76 [51] load PROGBITS 00002b362ec02000 251dd30 001000 00 A 0 0 1
77 [52] load PROGBITS 00002b362ec03000 251ed30 001000 00 WA 0 0 1
78 [53] load PROGBITS 00002b362ec04000 251fd30 008000 00 A 0 0 1
79 [54] load PROGBITS 00002b362ec0c000 2527d30 1f7000 00 WA 0 0 1
80 [55] load PROGBITS 00002b362ee03000 271ed30 001000 00 A 0 0 1
81 [56] load PROGBITS 00002b362ee04000 271fd30 001000 00 WA 0 0 1
82 [57] load PROGBITS 00002b362ee05000 2720d30 008000 00 A 0 0 1
83 [58] load PROGBITS 00002b362ee0d000 2728d30 3e8000 00 WA 0 0 1
84 [59] load PROGBITS 00002b362f1f5000 2b10d30 0d9000 00 WA 0 0 1
85 [60] load PROGBITS 00002b362f2ce000 2be9d30 107000 00 WA 0 0 1
86 [61] load PROGBITS 00002b362f3d5000 2cf0d30 006000 00 WA 0 0 1
87 [62] load PROGBITS 00002b362f3db000 2cf6d30 082000 00 WA 0 0 1
88 [63] load PROGBITS 00002b362f45d000 2d78d30 005000 00 WA 0 0 1
89 [64] load PROGBITS 00002b362f462000 2d7dd30 05d000 00 WA 0 0 1
90 [65] load PROGBITS 00002b362f4bf000 2ddad30 186000 00 WA 0 0 1
91 [66] load PROGBITS 00002b362f645000 2f60d30 009000 00 WA 0 0 1
92 [67] load PROGBITS 00002b362f64e000 2f69d30 197000 00 WA 0 0 1
93 [68] load PROGBITS 00002b362f9ef000 3100d30 001000 00 A 0 0 1
94 [69] load PROGBITS 00002b362f9f0000 3101d30 001000 00 WA 0 0 1
95 [70] load PROGBITS 00002b362f9f1000 3102d30 006000 00 WA 0 0 1
96 [71] load PROGBITS 00002b362fbfb000 3108d30 001000 00 A 0 0 1
97 [72] load PROGBITS 00002b362fbfc000 3109d30 001000 00 WA 0 0 1
98 [73] load PROGBITS 00002b362fe10000 310ad30 001000 00 A 0 0 1
99 [74] load PROGBITS 00002b362fe11000 310bd30 001000 00 WA 0 0 1
100 [75] load PROGBITS 00002b362fe12000 310cd30 002000 00 WA 0 0 1
101 [76] load PROGBITS 00002b362fe14000 310ed30 010000 00 WA 0 0 1
102 [77] load PROGBITS 00002b362fe24000 311ed30 01f000 00 WA 0 0 1
103 [78] load PROGBITS 00002b362fe43000 313dd30 1bd000 00 WA 0 0 1
104 [79] load PROGBITS 00002b3630000000 32fad30 021000 00 WA 0 0 1
105 [80] load PROGBITS 00002b3630021000 331bd30 3fdf000 00 A 0 0 1
106 [81] load PROGBITS 00002b3634000000 72fad30 021000 00 WA 0 0 1
107 [82] load PROGBITS 00002b3634021000 731bd30 3fdf000 00 A 0 0 1
108 [83] load PROGBITS 00002b3638205000 b2fad30 001000 00 A 0 0 1
109 [84] load PROGBITS 00002b3638206000 b2fbd30 001000 00 WA 0 0 1
110 [85] load PROGBITS 00002b36385bb000 b2fcd30 007000 00 A 0 0 1
111 [86] load PROGBITS 00002b36385c2000 b303d30 003000 00 WA 0 0 1
112 [87] load PROGBITS 00002b36385c5000 b306d30 1e4000 00 WA 0 0 1
113 [88] load PROGBITS 00002b36387a9000 b4ead30 001000 00 A 0 0 1
114 [89] load PROGBITS 00002b36387aa000 b4ebd30 001000 00 WA 0 0 1
115 [90] load PROGBITS 00002b36387ab000 b4ecd30 008000 00 A 0 0 1
116 [91] load PROGBITS 00002b36387b3000 b4f4d30 25a000 00 WA 0 0 1
117 [92] load PROGBITS 00002b3638a0d000 b74ed30 001000 00 A 0 0 1
118 [93] load PROGBITS 00002b3638a0e000 b74fd30 001000 00 WA 0 0 1
119 [94] load PROGBITS 00002b3638a0f000 b750d30 008000 00 A 0 0 1
120 [95] load PROGBITS 00002b3638a17000 b758d30 2a6000 00 WA 0 0 1
121 [96] load PROGBITS 00002b3638cbd000 b9fed30 001000 00 A 0 0 1
122 [97] load PROGBITS 00002b3638cbe000 b9ffd30 001000 00 WA 0 0 1
123 [98] load PROGBITS 00002b3638cbf000 ba00d30 008000 00 A 0 0 1
124 [99] load PROGBITS 00002b3638cc7000 ba08d30 3bd000 00 WA 0 0 1
125 [100] load PROGBITS 00002b3639084000 bdc5d30 1cc000 00 WA 0 0 1
126 [101] load PROGBITS 00002b3639251000 bf91d30 06d000 00 WA 0 0 1
127 [102] load PROGBITS 00002b36392be000 bffed30 0e6000 00 WA 0 0 1
128 [103] load PROGBITS 00002b36393a4000 c0e4d30 04b000 00 WA 0 0 1
129 [104] load PROGBITS 00002b36393ef000 c12fd30 10e000 00 WA 0 0 1
130 [105] load PROGBITS 00002b36394fd000 c23dd30 02c000 00 WA 0 0 1
131 [106] load PROGBITS 00002b3639529000 c269d30 02b000 00 WA 0 0 1
132 [107] load PROGBITS 00002b3639554000 c294d30 04d000 00 WA 0 0 1
133 [108] load PROGBITS 00002b36395a1000 c2e1d30 09a000 00 WA 0 0 1
134 [109] load PROGBITS 00002b363963b000 c37bd30 095000 00 WA 0 0 1
135 [110] load PROGBITS 00002b36396d0000 c410d30 08a000 00 WA 0 0 1
136 [111] load PROGBITS 00002b363975a000 c49ad30 0db000 00 WA 0 0 1
137 [112] load PROGBITS 00002b3639835000 c575d30 116000 00 WA 0 0 1
138 [113] load PROGBITS 000055f550552000 c68bd30 007000 00 A 0 0 1
139 [114] load PROGBITS 000055f550559000 c692d30 004000 00 WA 0 0 1
140 [115] load PROGBITS 000055f55055d000 c696d30 017000 00 WA 0 0 1
141 [116] load PROGBITS 000055f55158f000 c6add30 063000 00 WA 0 0 1
142 [117] load PROGBITS 000055f5515f3000 c710d30 008000 00 A 0 0 1
143 [118] load PROGBITS 00007ffd12743000 c718d30 021000 00 WA 0 0 1
144 [119] load PROGBITS 00007ffd127b5000 c739d30 002000 00 AX 0 0 1
145 [120] load PROGBITS ffffffffff600000 c73bd30 001000 00 AX 0 0 1
146 [121] .shstrtab STRTAB 0000000000000000 c73cd30 000016 00 0 0 1
147Key to Flags:
148 W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
149 L (link order), O (extra OS processing required), G (group), T (TLS),
150 C (compressed), x (unknown), o (OS specific), E (exclude),
151 l (large), p (processor specific)
152
153Program Headers:
154 Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
155 NOTE 0x001a80 0x0000000000000000 0x0000000000000000 0x0052b0 0x000000 R 0x1
156 LOAD 0x006d30 0x000000004112c000 0x0000000000000000 0x010000 0x010000 RWE 0x1
157 LOAD 0x016d30 0x0000000041a34000 0x0000000000000000 0x090000 0x090000 RWE 0x1
158 LOAD 0x0a6d30 0x00002b362a4ca000 0x0000000000000000 0x023000 0x023000 R E 0x1
159 LOAD 0x0c9d30 0x00002b362a4ff000 0x0000000000000000 0x009000 0x009000 RW 0x1
160 LOAD 0x0d2d30 0x00002b362a508000 0x0000000000000000 0x006000 0x006000 RW 0x1
161 LOAD 0x0d8d30 0x00002b362a68f000 0x0000000000000000 0x005000 0x005000 RW 0x1
162 LOAD 0x0ddd30 0x00002b362a699000 0x0000000000000000 0x054000 0x054000 RW 0x1
163 LOAD 0x131d30 0x00002b362a6ed000 0x0000000000000000 0x001000 0x001000 R 0x1
164 LOAD 0x132d30 0x00002b362a6ee000 0x0000000000000000 0x001000 0x001000 RW 0x1
165 LOAD 0x133d30 0x00002b362a6ef000 0x0000000000000000 0x001000 0x001000 RW 0x1
166 LOAD 0x134d30 0x00002b362a9f2000 0x0000000000000000 0x001000 0x001000 R 0x1
167 LOAD 0x135d30 0x00002b362a9f3000 0x0000000000000000 0x001000 0x001000 RW 0x1
168 LOAD 0x136d30 0x00002b362abfa000 0x0000000000000000 0x001000 0x001000 R 0x1
169 LOAD 0x137d30 0x00002b362abfb000 0x0000000000000000 0x001000 0x001000 RW 0x1
170 LOAD 0x138d30 0x00002b362adfe000 0x0000000000000000 0x001000 0x001000 R 0x1
171 LOAD 0x139d30 0x00002b362adff000 0x0000000000000000 0x001000 0x001000 RW 0x1
172 LOAD 0x13ad30 0x00002b362b017000 0x0000000000000000 0x001000 0x001000 R 0x1
173 LOAD 0x13bd30 0x00002b362b018000 0x0000000000000000 0x001000 0x001000 RW 0x1
174 LOAD 0x13cd30 0x00002b362b019000 0x0000000000000000 0x004000 0x004000 RW 0x1
175 LOAD 0x140d30 0x00002b362b232000 0x0000000000000000 0x001000 0x001000 R 0x1
176 LOAD 0x141d30 0x00002b362b233000 0x0000000000000000 0x001000 0x001000 RW 0x1
177 LOAD 0x142d30 0x00002b362b5c9000 0x0000000000000000 0x004000 0x004000 R 0x1
178 LOAD 0x146d30 0x00002b362b5cd000 0x0000000000000000 0x002000 0x002000 RW 0x1
179 LOAD 0x148d30 0x00002b362b5cf000 0x0000000000000000 0x004000 0x004000 RW 0x1
180 LOAD 0x14cd30 0x00002b362b5d3000 0x0000000000000000 0x001000 0x001000 RW 0x1
181 LOAD 0x14dd30 0x00002b362b5d4000 0x0000000000000000 0x05f000 0x05f000 R 0x1
182 LOAD 0x1acd30 0x00002b362b633000 0x0000000000000000 0x0b3000 0x0b3000 RW 0x1
183 LOAD 0x25fd30 0x00002b362b7fb000 0x0000000000000000 0x005000 0x005000 RW 0x1
184 LOAD 0x264d30 0x00002b362b800000 0x0000000000000000 0x400000 0x400000 RW 0x1
185 LOAD 0x664d30 0x00002b362bc00000 0x0000000000000000 0x001000 0x001000 R 0x1
186 LOAD 0x665d30 0x00002b362bc01000 0x0000000000000000 0x1200000 0x1200000 RW 0x1
187 LOAD 0x1865d30 0x00002b362d933000 0x0000000000000000 0x001000 0x001000 R 0x1
188 LOAD 0x1866d30 0x00002b362d934000 0x0000000000000000 0x001000 0x001000 RW 0x1
189 LOAD 0x1867d30 0x00002b362d935000 0x0000000000000000 0x026000 0x026000 RW 0x1
190 LOAD 0x188dd30 0x00002b362d95b000 0x0000000000000000 0x100000 0x100000 RW 0x1
191 LOAD 0x198dd30 0x00002b362da5b000 0x0000000000000000 0x001000 0x001000 R 0x1
192 LOAD 0x198ed30 0x00002b362da5c000 0x0000000000000000 0x001000 0x001000 RW 0x1
193 LOAD 0x198fd30 0x00002b362da5d000 0x0000000000000000 0x008000 0x008000 R 0x1
194 LOAD 0x1997d30 0x00002b362da65000 0x0000000000000000 0x1f7000 0x1f7000 RW 0x1
195 LOAD 0x1b8ed30 0x00002b362e273000 0x0000000000000000 0x08d000 0x08d000 RW 0x1
196 LOAD 0x1c1bd30 0x00002b362e300000 0x0000000000000000 0x400000 0x400000 RW 0x1
197 LOAD 0x201bd30 0x00002b362e700000 0x0000000000000000 0x001000 0x001000 R 0x1
198 LOAD 0x201cd30 0x00002b362e701000 0x0000000000000000 0x001000 0x001000 RW 0x1
199 LOAD 0x201dd30 0x00002b362e702000 0x0000000000000000 0x008000 0x008000 R 0x1
200 LOAD 0x2025d30 0x00002b362e70a000 0x0000000000000000 0x2f7000 0x2f7000 RW 0x1
201 LOAD 0x231cd30 0x00002b362ea01000 0x0000000000000000 0x001000 0x001000 R 0x1
202 LOAD 0x231dd30 0x00002b362ea02000 0x0000000000000000 0x001000 0x001000 RW 0x1
203 LOAD 0x231ed30 0x00002b362ea03000 0x0000000000000000 0x008000 0x008000 R 0x1
204 LOAD 0x2326d30 0x00002b362ea0b000 0x0000000000000000 0x1f7000 0x1f7000 RW 0x1
205 LOAD 0x251dd30 0x00002b362ec02000 0x0000000000000000 0x001000 0x001000 R 0x1
206 LOAD 0x251ed30 0x00002b362ec03000 0x0000000000000000 0x001000 0x001000 RW 0x1
207 LOAD 0x251fd30 0x00002b362ec04000 0x0000000000000000 0x008000 0x008000 R 0x1
208 LOAD 0x2527d30 0x00002b362ec0c000 0x0000000000000000 0x1f7000 0x1f7000 RW 0x1
209 LOAD 0x271ed30 0x00002b362ee03000 0x0000000000000000 0x001000 0x001000 R 0x1
210 LOAD 0x271fd30 0x00002b362ee04000 0x0000000000000000 0x001000 0x001000 RW 0x1
211 LOAD 0x2720d30 0x00002b362ee05000 0x0000000000000000 0x008000 0x008000 R 0x1
212 LOAD 0x2728d30 0x00002b362ee0d000 0x0000000000000000 0x3e8000 0x3e8000 RW 0x1
213 LOAD 0x2b10d30 0x00002b362f1f5000 0x0000000000000000 0x0d9000 0x0d9000 RW 0x1
214 LOAD 0x2be9d30 0x00002b362f2ce000 0x0000000000000000 0x107000 0x107000 RW 0x1
215 LOAD 0x2cf0d30 0x00002b362f3d5000 0x0000000000000000 0x006000 0x006000 RW 0x1
216 LOAD 0x2cf6d30 0x00002b362f3db000 0x0000000000000000 0x082000 0x082000 RW 0x1
217 LOAD 0x2d78d30 0x00002b362f45d000 0x0000000000000000 0x005000 0x005000 RW 0x1
218 LOAD 0x2d7dd30 0x00002b362f462000 0x0000000000000000 0x05d000 0x05d000 RW 0x1
219 LOAD 0x2ddad30 0x00002b362f4bf000 0x0000000000000000 0x186000 0x186000 RW 0x1
220 LOAD 0x2f60d30 0x00002b362f645000 0x0000000000000000 0x009000 0x009000 RW 0x1
221 LOAD 0x2f69d30 0x00002b362f64e000 0x0000000000000000 0x197000 0x197000 RW 0x1
222 LOAD 0x3100d30 0x00002b362f9ef000 0x0000000000000000 0x001000 0x001000 R 0x1
223 LOAD 0x3101d30 0x00002b362f9f0000 0x0000000000000000 0x001000 0x001000 RW 0x1
224 LOAD 0x3102d30 0x00002b362f9f1000 0x0000000000000000 0x006000 0x006000 RW 0x1
225 LOAD 0x3108d30 0x00002b362fbfb000 0x0000000000000000 0x001000 0x001000 R 0x1
226 LOAD 0x3109d30 0x00002b362fbfc000 0x0000000000000000 0x001000 0x001000 RW 0x1
227 LOAD 0x310ad30 0x00002b362fe10000 0x0000000000000000 0x001000 0x001000 R 0x1
228 LOAD 0x310bd30 0x00002b362fe11000 0x0000000000000000 0x001000 0x001000 RW 0x1
229 LOAD 0x310cd30 0x00002b362fe12000 0x0000000000000000 0x002000 0x002000 RW 0x1
230 LOAD 0x310ed30 0x00002b362fe14000 0x0000000000000000 0x010000 0x010000 RW 0x1
231 LOAD 0x311ed30 0x00002b362fe24000 0x0000000000000000 0x01f000 0x01f000 RW 0x1
232 LOAD 0x313dd30 0x00002b362fe43000 0x0000000000000000 0x1bd000 0x1bd000 RW 0x1
233 LOAD 0x32fad30 0x00002b3630000000 0x0000000000000000 0x021000 0x021000 RW 0x1
234 LOAD 0x331bd30 0x00002b3630021000 0x0000000000000000 0x3fdf000 0x3fdf000 R 0x1
235 LOAD 0x72fad30 0x00002b3634000000 0x0000000000000000 0x021000 0x021000 RW 0x1
236 LOAD 0x731bd30 0x00002b3634021000 0x0000000000000000 0x3fdf000 0x3fdf000 R 0x1
237 LOAD 0xb2fad30 0x00002b3638205000 0x0000000000000000 0x001000 0x001000 R 0x1
238 LOAD 0xb2fbd30 0x00002b3638206000 0x0000000000000000 0x001000 0x001000 RW 0x1
239 LOAD 0xb2fcd30 0x00002b36385bb000 0x0000000000000000 0x007000 0x007000 R 0x1
240 LOAD 0xb303d30 0x00002b36385c2000 0x0000000000000000 0x003000 0x003000 RW 0x1
241 LOAD 0xb306d30 0x00002b36385c5000 0x0000000000000000 0x1e4000 0x1e4000 RW 0x1
242 LOAD 0xb4ead30 0x00002b36387a9000 0x0000000000000000 0x001000 0x001000 R 0x1
243 LOAD 0xb4ebd30 0x00002b36387aa000 0x0000000000000000 0x001000 0x001000 RW 0x1
244 LOAD 0xb4ecd30 0x00002b36387ab000 0x0000000000000000 0x008000 0x008000 R 0x1
245 LOAD 0xb4f4d30 0x00002b36387b3000 0x0000000000000000 0x25a000 0x25a000 RW 0x1
246 LOAD 0xb74ed30 0x00002b3638a0d000 0x0000000000000000 0x001000 0x001000 R 0x1
247 LOAD 0xb74fd30 0x00002b3638a0e000 0x0000000000000000 0x001000 0x001000 RW 0x1
248 LOAD 0xb750d30 0x00002b3638a0f000 0x0000000000000000 0x008000 0x008000 R 0x1
249 LOAD 0xb758d30 0x00002b3638a17000 0x0000000000000000 0x2a6000 0x2a6000 RW 0x1
250 LOAD 0xb9fed30 0x00002b3638cbd000 0x0000000000000000 0x001000 0x001000 R 0x1
251 LOAD 0xb9ffd30 0x00002b3638cbe000 0x0000000000000000 0x001000 0x001000 RW 0x1
252 LOAD 0xba00d30 0x00002b3638cbf000 0x0000000000000000 0x008000 0x008000 R 0x1
253 LOAD 0xba08d30 0x00002b3638cc7000 0x0000000000000000 0x3bd000 0x3bd000 RW 0x1
254 LOAD 0xbdc5d30 0x00002b3639084000 0x0000000000000000 0x1cc000 0x1cc000 RW 0x1
255 LOAD 0xbf91d30 0x00002b3639251000 0x0000000000000000 0x06d000 0x06d000 RW 0x1
256 LOAD 0xbffed30 0x00002b36392be000 0x0000000000000000 0x0e6000 0x0e6000 RW 0x1
257 LOAD 0xc0e4d30 0x00002b36393a4000 0x0000000000000000 0x04b000 0x04b000 RW 0x1
258 LOAD 0xc12fd30 0x00002b36393ef000 0x0000000000000000 0x10e000 0x10e000 RW 0x1
259 LOAD 0xc23dd30 0x00002b36394fd000 0x0000000000000000 0x02c000 0x02c000 RW 0x1
260 LOAD 0xc269d30 0x00002b3639529000 0x0000000000000000 0x02b000 0x02b000 RW 0x1
261 LOAD 0xc294d30 0x00002b3639554000 0x0000000000000000 0x04d000 0x04d000 RW 0x1
262 LOAD 0xc2e1d30 0x00002b36395a1000 0x0000000000000000 0x09a000 0x09a000 RW 0x1
263 LOAD 0xc37bd30 0x00002b363963b000 0x0000000000000000 0x095000 0x095000 RW 0x1
264 LOAD 0xc410d30 0x00002b36396d0000 0x0000000000000000 0x08a000 0x08a000 RW 0x1
265 LOAD 0xc49ad30 0x00002b363975a000 0x0000000000000000 0x0db000 0x0db000 RW 0x1
266 LOAD 0xc575d30 0x00002b3639835000 0x0000000000000000 0x116000 0x116000 RW 0x1
267 LOAD 0xc68bd30 0x000055f550552000 0x0000000000000000 0x007000 0x007000 R 0x1
268 LOAD 0xc692d30 0x000055f550559000 0x0000000000000000 0x004000 0x004000 RW 0x1
269 LOAD 0xc696d30 0x000055f55055d000 0x0000000000000000 0x017000 0x017000 RW 0x1
270 LOAD 0xc6add30 0x000055f55158f000 0x0000000000000000 0x063000 0x063000 RW 0x1
271 LOAD 0xc710d30 0x000055f5515f3000 0x0000000000000000 0x008000 0x008000 R 0x1
272 LOAD 0xc718d30 0x00007ffd12743000 0x0000000000000000 0x021000 0x021000 RW 0x1
273 LOAD 0xc739d30 0x00007ffd127b5000 0x0000000000000000 0x002000 0x002000 R E 0x1
274 LOAD 0xc73bd30 0xffffffffff600000 0x0000000000000000 0x001000 0x001000 R E 0x1
275
276 Section to Segment mapping:
277 Segment Sections...
278 00
279 01 load
280 02 load
281 03 load
282 04 load
283 05 load
284 06 load
285 07 load
286 08 load
287 09 load
288 10 load
289 11 load
290 12 load
291 13 load
292 14 load
293 15 load
294 16 load
295 17 load
296 18 load
297 19 load
298 20 load
299 21 load
300 22 load
301 23 load
302 24 load
303 25 load
304 26 load
305 27 load
306 28 load
307 29 load
308 30 load
309 31 load
310 32 load
311 33 load
312 34 load
313 35 load
314 36 load
315 37 load
316 38 load
317 39 load
318 40 load
319 41 load
320 42 load
321 43 load
322 44 load
323 45 load
324 46 load
325 47 load
326 48 load
327 49 load
328 50 load
329 51 load
330 52 load
331 53 load
332 54 load
333 55 load
334 56 load
335 57 load
336 58 load
337 59 load
338 60 load
339 61 load
340 62 load
341 63 load
342 64 load
343 65 load
344 66 load
345 67 load
346 68 load
347 69 load
348 70 load
349 71 load
350 72 load
351 73 load
352 74 load
353 75 load
354 76 load
355 77 load
356 78 load
357 79 load
358 80 load
359 81 load
360 82 load
361 83 load
362 84 load
363 85 load
364 86 load
365 87 load
366 88 load
367 89 load
368 90 load
369 91 load
370 92 load
371 93 load
372 94 load
373 95 load
374 96 load
375 97 load
376 98 load
377 99 load
378 100 load
379 101 load
380 102 load
381 103 load
382 104 load
383 105 load
384 106 load
385 107 load
386 108 load
387 109 load
388 110 load
389 111 load
390 112 load
391 113 load
392 114 load
393 115 load
394 116 load
395 117 load
396 118 load
397 119 load

Also paused the process just in case any adversary is going to perform any ASLR attacks on it since I pasted the memory map above:

11:31:30 0 ✓ zhuyifei1999@tools-sgeexec-0906: ~$ sudo kill -STOP 13609

The large maps are basically zeros as I scroll through with xxd. Both of the maps have an 132KiB read-writable map immediately preceding it but I don't find any readable strings inside, hence I can't be sure what they are actually for or what allocated them by looking at the core dump.

@Vort can I have your permission to execute the program in gdb, where I will trap all the memory mapping requests and if they are above 50MiB I will create a stack trace of the point that did the mmap?

As for the hang, you are hitting T195834:

111:59:50 1 ✗ zhuyifei1999@tools-sgeexec-0906: ~$ sudo gdb -p 13609 -batch -ex 'thread apply all bt'
2[New LWP 13610]
3[New LWP 13611]
4[New LWP 13612]
5[New LWP 13613]
6[New LWP 13614]
7[New LWP 13615]
8[New LWP 13616]
9[New LWP 13617]
10[New LWP 13618]
11[Thread debugging using libthread_db enabled]
12Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
130x00002b362ae0d17f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
14
15Thread 10 (Thread 0x2b3638ebd700 (LWP 13618)):
16#0 0x000055f5502255a8 in monoeg_g_calloc ()
17#1 0x000055f55022246e in monoeg_g_hash_table_new ()
18#2 0x000055f5501560a3 in ?? ()
19#3 0x000055f55015650d in ?? ()
20#4 0x00002b362ae074a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
21#5 0x00002b362b31cd0f in clone () from /lib/x86_64-linux-gnu/libc.so.6
22
23Thread 9 (Thread 0x2b3638c0d700 (LWP 13617)):
24#0 0x000055f5502255a8 in monoeg_g_calloc ()
25#1 0x000055f55022988d in monoeg_g_list_prepend ()
26#2 0x000055f550127332 in ?? ()
27#3 0x000055f550190caa in ?? ()
28#4 0x000055f55012720a in ?? ()
29#5 0x000055f55010e7c1 in ?? ()
30#6 0x000055f5500d0e19 in ?? ()
31#7 0x000055f5500d19f9 in ?? ()
32#8 0x000055f5500cbdbf in ?? ()
33#9 0x000055f5500ccf4f in ?? ()
34#10 0x000055f5500cf30d in ?? ()
35#11 0x000055f54fff37e6 in ?? ()
36#12 0x000000004112d513 in ?? ()
37#13 0x00002b36389db015 in ?? ()
38#14 0x5861cd56f7e29300 in ?? ()
39#15 0x00002b3638c0ac00 in ?? ()
40#16 0x00002b3638c0aa38 in ?? ()
41#17 0x00002b36389d8020 in ?? ()
42#18 0x00002b36398313f0 in ?? ()
43#19 0x00002b362b8dead8 in ?? ()
44#20 0x00002b362b8deb48 in ?? ()
45#21 0x00002b362b8de698 in ?? ()
46#22 0x00002b3638c0ac00 in ?? ()
47#23 0x00002b3638c0ac80 in ?? ()
48#24 0x0000000000000000 in ?? ()
49
50Thread 8 (Thread 0x2b36389a9700 (LWP 13616)):
51#0 0x00002b362b3138bd in poll () from /lib/x86_64-linux-gnu/libc.so.6
52#1 0x000055f550159485 in ?? ()
53#2 0x000055f55015a49a in ?? ()
54#3 0x000055f550156505 in ?? ()
55#4 0x00002b362ae074a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
56#5 0x00002b362b31cd0f in clone () from /lib/x86_64-linux-gnu/libc.so.6
57
58Thread 7 (Thread 0x2b362f003700 (LWP 13615)):
59#0 0x00002b362ae0ff7c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
60#1 0x00002b362ae09c26 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
61#2 0x000055f550155fd0 in ?? ()
62#3 0x000055f55015650d in ?? ()
63#4 0x00002b362ae074a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
64#5 0x00002b362b31cd0f in clone () from /lib/x86_64-linux-gnu/libc.so.6
65
66Thread 6 (Thread 0x2b362ee02700 (LWP 13614)):
67#0 0x00002b362ae0ff7c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
68#1 0x00002b362ae09c26 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
69#2 0x000055f550155fd0 in ?? ()
70#3 0x000055f55015650d in ?? ()
71#4 0x00002b362ae074a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
72#5 0x00002b362b31cd0f in clone () from /lib/x86_64-linux-gnu/libc.so.6
73
74Thread 5 (Thread 0x2b362ec01700 (LWP 13613)):
75#0 0x00002b362ae0ff7c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
76#1 0x00002b362ae09c26 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
77#2 0x000055f550155fd0 in ?? ()
78#3 0x000055f55015650d in ?? ()
79#4 0x00002b362ae074a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
80#5 0x00002b362b31cd0f in clone () from /lib/x86_64-linux-gnu/libc.so.6
81
82Thread 4 (Thread 0x2b362e900700 (LWP 13612)):
83#0 0x00002b362ae0d17f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
84#1 0x000055f5502127cd in ?? ()
85#2 0x000055f55016d255 in ?? ()
86#3 0x000055f55016e53f in ?? ()
87#4 0x000055f55016ecb5 in ?? ()
88#5 0x000055f5501541d7 in ?? ()
89#6 0x0000000041a8693d in ?? ()
90#7 0x00002b362b8d9d18 in ?? ()
91#8 0xffffffffffffffff in ?? ()
92#9 0xffffffffffffffff in ?? ()
93#10 0x0000000000000000 in ?? ()
94
95Thread 3 (Thread 0x2b362dc5b700 (LWP 13611)):
96#0 0x00002b362ae0f556 in do_futex_wait.constprop () from /lib/x86_64-linux-gnu/libpthread.so.0
97#1 0x00002b362ae0f604 in __new_sem_wait_slow.constprop.0 () from /lib/x86_64-linux-gnu/libpthread.so.0
98#2 0x000055f55019e5fa in ?? ()
99#3 0x000055f550156505 in ?? ()
100#4 0x00002b362ae074a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
101#5 0x00002b362b31cd0f in clone () from /lib/x86_64-linux-gnu/libc.so.6
102
103Thread 2 (Thread 0x2b362be00700 (LWP 13610)):
104#0 0x00002b362ae0d17f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
105#1 0x000055f550205bcb in ?? ()
106#2 0x00002b362ae074a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
107#3 0x00002b362b31cd0f in clone () from /lib/x86_64-linux-gnu/libc.so.6
108
109Thread 1 (Thread 0x2b362a50b680 (LWP 13609)):
110#0 0x00002b362ae0d17f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
111#1 0x000055f5502127cd in ?? ()
112#2 0x000055f55016d255 in ?? ()
113#3 0x000055f55016e53f in ?? ()
114#4 0x000055f5501a7630 in ?? ()
115#5 0x0000000041aa3f0e in ?? ()
116#6 0x00007ffd12762570 in ?? ()
117#7 0x0000000000000000 in ?? ()

Vort added a comment.Oct 28 2020, 6:59 AM

As I understand, 1637509 is unexpected incomplete output, correct?

Yes, it is doing something that is not intended.

can I have your permission to execute the program in gdb, where I will trap all the memory mapping requests and if they are above 50MiB I will create a stack trace of the point that did the mmap?

Yes, of course. Just don't publish bot auth keys.
But before it I want to try catching exception with 512m setting.
I will post additional message when I get results.

As for the hang, you are hitting T195834:

It happens because of out if memory condition + some bug in Mono.
But why out if memory condition happens? That is interesting question too.

Vort added a comment.EditedOct 28 2020, 7:26 AM

While trying to get exception, I have found that exactly the same hang also can happen with 512m.
Will try few more times.

+1 try: the same hang.
+1 try: big chunk of errors:

(please wrap it into spoiler, I don't know how)

[2020.10.28 07:31:20] Task started
Authenticating...Could not allocate 8192 bytes
Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper remoting-invoke-with-check) System.Net.WebResponseStream.InitReadAsync (System.Threading.CancellationToken) [0x00032] in <fc308f916aec4e4283e0c1d4b761760a>:0
  at System.Net.WebOperation/<Run>d__57.MoveNext () [0x00179] in <fc308f916aec4e4283e0c1d4b761760a>:0
  at System.Runtime.CompilerServices.AsyncMethodBuilderCore/MoveNextRunner.InvokeMoveNext (object) [0x00000] in <71d8ad678db34313b7f718a414dfcb25>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <71d8ad678db34313b7f718a414dfcb25>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <71d8ad678db34313b7f718a414dfcb25>:0
  at System.Runtime.CompilerServices.AsyncMethodBuilderCore/MoveNextRunner.Run () [0x00024] in <71d8ad678db34313b7f718a414dfcb25>:0
  at System.Threading.Tasks.AwaitTaskContinuation.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <71d8ad678db34313b7f718a414dfcb25>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <71d8ad678db34313b7f718a414dfcb25>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <71d8ad678db34313b7f718a414dfcb25>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0001e] in <71d8ad678db34313b7f718a414dfcb25>:0
/proc/self/maps:
40db0000-40dc0000 rwxp 00000000 00:00 0 
4172f000-417ef000 rwxp 00000000 00:00 0 
2b94bb44f000-2b94bb472000 r-xp 00000000 fe:03 1048724                    /lib/x86_64-linux-gnu/ld-2.24.so
2b94bb472000-2b94bb473000 r--p 00000000 fe:03 25257                      /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
2b94bb473000-2b94bb47a000 r--s 00000000 fe:03 954009                     /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
2b94bb47a000-2b94bb47b000 r--p 00000000 fe:03 1187520                    /usr/lib/locale/chr_US/LC_MEASUREMENT
2b94bb47b000-2b94bb47c000 r--p 00000000 fe:03 1187526                    /usr/lib/locale/chr_US/LC_TELEPHONE
2b94bb47c000-2b94bb47d000 r--p 00000000 fe:03 25256                      /usr/lib/locale/en_US.utf8/LC_ADDRESS
2b94bb47d000-2b94bb47e000 r--p 00000000 fe:03 1187524                    /usr/lib/locale/chr_US/LC_NAME
2b94bb47e000-2b94bb47f000 r--p 00000000 fe:03 1187525                    /usr/lib/locale/chr_US/LC_PAPER
2b94bb47f000-2b94bb480000 r--p 00000000 fe:03 1187809                    /usr/lib/locale/en_AG/LC_MESSAGES/SYS_LC_MESSAGES
2b94bb480000-2b94bb481000 r--p 00000000 fe:03 1187523                    /usr/lib/locale/chr_US/LC_MONETARY
2b94bb481000-2b94bb482000 r--p 00000000 fe:03 25259                      /usr/lib/locale/en_US.utf8/LC_TIME
2b94bb482000-2b94bb483000 r--p 00000000 fe:03 1055837                    /usr/lib/locale/aa_ET/LC_NUMERIC
2b94bb483000-2b94bb484000 rw-s 00000000 00:13 58471219                   /dev/shm/mono.13899
2b94bb484000-2b94bb48d000 rw-p 00000000 00:00 0 
2b94bb48d000-2b94bb493000 rw-p 00000000 00:00 0 
2b94bb493000-2b94bb5c3000 r--p 00000000 fe:03 1055801                    /usr/lib/locale/aa_DJ.utf8/LC_COLLATE
2b94bb5c3000-2b94bb614000 r--p 00000000 fe:03 1055802                    /usr/lib/locale/aa_DJ.utf8/LC_CTYPE
2b94bb614000-2b94bb618000 rw-p 00000000 00:00 0 
2b94bb618000-2b94bb61d000 r--p 00000000 00:2d 91168018                   /mnt/nfs/labstore-secondary-tools-project/wikitasks/wp_cyrlat/WikiTasks.exe
2b94bb61d000-2b94bb672000 rw-p 00000000 00:00 0 
2b94bb672000-2b94bb673000 r--p 00023000 fe:03 1048724                    /lib/x86_64-linux-gnu/ld-2.24.so
2b94bb673000-2b94bb674000 rw-p 00024000 fe:03 1048724                    /lib/x86_64-linux-gnu/ld-2.24.so
2b94bb674000-2b94bb675000 rw-p 00000000 00:00 0 
2b94bb675000-2b94bb778000 r-xp 00000000 fe:03 1049053                    /lib/x86_64-linux-gnu/libm-2.24.so
2b94bb778000-2b94bb977000 ---p 00103000 fe:03 1049053                    /lib/x86_64-linux-gnu/libm-2.24.so
2b94bb977000-2b94bb978000 r--p 00102000 fe:03 1049053                    /lib/x86_64-linux-gnu/libm-2.24.so
2b94bb978000-2b94bb979000 rw-p 00103000 fe:03 1049053                    /lib/x86_64-linux-gnu/libm-2.24.so
2b94bb979000-2b94bb980000 r-xp 00000000 fe:03 1049071                    /lib/x86_64-linux-gnu/librt-2.24.so
2b94bb980000-2b94bbb7f000 ---p 00007000 fe:03 1049071                    /lib/x86_64-linux-gnu/librt-2.24.so
2b94bbb7f000-2b94bbb80000 r--p 00006000 fe:03 1049071                    /lib/x86_64-linux-gnu/librt-2.24.so
2b94bbb80000-2b94bbb81000 rw-p 00007000 fe:03 1049071                    /lib/x86_64-linux-gnu/librt-2.24.so
2b94bbb81000-2b94bbb84000 r-xp 00000000 fe:03 1048975                    /lib/x86_64-linux-gnu/libdl-2.24.so
2b94bbb84000-2b94bbd83000 ---p 00003000 fe:03 1048975                    /lib/x86_64-linux-gnu/libdl-2.24.so
2b94bbd83000-2b94bbd84000 r--p 00002000 fe:03 1048975                    /lib/x86_64-linux-gnu/libdl-2.24.so
2b94bbd84000-2b94bbd85000 rw-p 00003000 fe:03 1048975                    /lib/x86_64-linux-gnu/libdl-2.24.so
2b94bbd85000-2b94bbd9d000 r-xp 00000000 fe:03 1049068                    /lib/x86_64-linux-gnu/libpthread-2.24.so
2b94bbd9d000-2b94bbf9c000 ---p 00018000 fe:03 1049068                    /lib/x86_64-linux-gnu/libpthread-2.24.so
2b94bbf9c000-2b94bbf9d000 r--p 00017000 fe:03 1049068                    /lib/x86_64-linux-gnu/libpthread-2.24.so
2b94bbf9d000-2b94bbf9e000 rw-p 00018000 fe:03 1049068                    /lib/x86_64-linux-gnu/libpthread-2.24.so
2b94bbf9e000-2b94bbfa2000 rw-p 00000000 00:00 0 
2b94bbfa2000-2b94bbfb8000 r-xp 00000000 fe:03 1048593                    /lib/x86_64-linux-gnu/libgcc_s.so.1
2b94bbfb8000-2b94bc1b7000 ---p 00016000 fe:03 1048593                    /lib/x86_64-linux-gnu/libgcc_s.so.1
2b94bc1b7000-2b94bc1b8000 r--p 00015000 fe:03 1048593                    /lib/x86_64-linux-gnu/libgcc_s.so.1
2b94bc1b8000-2b94bc1b9000 rw-p 00016000 fe:03 1048593                    /lib/x86_64-linux-gnu/libgcc_s.so.1
2b94bc1b9000-2b94bc34e000 r-xp 00000000 fe:03 1048969                    /lib/x86_64-linux-gnu/libc-2.24.so
2b94bc34e000-2b94bc54e000 ---p 00195000 fe:03 1048969                    /lib/x86_64-linux-gnu/libc-2.24.so
2b94bc54e000-2b94bc552000 r--p 00195000 fe:03 1048969                    /lib/x86_64-linux-gnu/libc-2.24.so
2b94bc552000-2b94bc554000 rw-p 00199000 fe:03 1048969                    /lib/x86_64-linux-gnu/libc-2.24.so
2b94bc554000-2b94bc558000 rw-p 00000000 00:00 0 
2b94bc558000-2b94bc559000 rw-p 00000000 00:00 0 
2b94bc559000-2b94bc5b8000 ---p 00000000 00:00 0 
2b94bc5b8000-2b94bc5b9000 ---p 00000000 00:00 0 
2b94bc5b9000-2b94bc800000 rw-p 00000000 00:00 0 
2b94bc800000-2b94bdc00000 rw-p 00000000 00:00 0 
2b94bdc00000-2b94bdfcb000 r--p 00000000 fe:03 957258                     /usr/lib/mono/4.5/mscorlib.dll
2b94bdfcb000-2b94be533000 r-xp 00000000 fe:03 956760                     /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
2b94be533000-2b94be732000 ---p 00568000 fe:03 956760                     /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
2b94be732000-2b94be733000 r--p 00567000 fe:03 956760                     /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
2b94be733000-2b94be734000 rw-p 00568000 fe:03 956760                     /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
2b94be734000-2b94be75a000 rw-p 00000000 00:00 0 
2b94be75a000-2b94be75c000 rw-p 00000000 00:00 0 
2b94be75c000-2b94be7dc000 rw-p 00000000 00:00 0 
2b94be7dc000-2b94be7f9000 r--p 00000000 fe:03 957038                     /usr/lib/mono/gac/System.Net.Http/4.0.0.0__b03f5f7f11d50a3a/System.Net.Http.dll
2b94be7f9000-2b94be809000 rw-p 00000000 00:00 0 
2b94be809000-2b94be909000 rw-p 00000000 00:00 0 
2b94be909000-2b94be90a000 ---p 00000000 00:00 0 
2b94be90a000-2b94be90b000 rw-p 00000000 00:00 0 
2b94be90b000-2b94be913000 ---p 00000000 00:00 0 
2b94be913000-2b94beb0a000 rw-p 00000000 00:00 0 
2b94beb0a000-2b94bedb5000 r--p 00000000 fe:03 1072921                    /usr/lib/mono/gac/System/4.0.0.0__b77a5c561934e089/System.dll
2b94bedb5000-2b94beead000 r--p 00000000 fe:03 955585                     /usr/lib/mono/gac/System.Core/4.0.0.0__b77a5c561934e089/System.Core.dll
2b94beead000-2b94beecb000 r--p 00000000 fe:03 955568                     /usr/lib/mono/gac/System.Configuration/4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
2b94beecb000-2b94bf1cd000 r--p 00000000 fe:03 955581                     /usr/lib/mono/gac/System.Xml/4.0.0.0__b77a5c561934e089/System.Xml.dll
2b94bf1cd000-2b94bf1de000 rw-p 00000000 00:00 0 
2b94bf1de000-2b94bf200000 rw-p 00000000 00:00 0 
2b94bf200000-2b94bf500000 rw-p 00000000 00:00 0 
2b94bf500000-2b94bf54c000 r--p 00000000 fe:03 957244                     /usr/lib/mono/gac/Mono.Security/4.0.0.0__0738eb9f132ed756/Mono.Security.dll
2b94bf54c000-2b94bf64c000 rw-p 00000000 00:00 0 
2b94bf64c000-2b94bf64d000 ---p 00000000 00:00 0 
2b94bf64d000-2b94bf64e000 rw-p 00000000 00:00 0 
2b94bf64e000-2b94bf656000 ---p 00000000 00:00 0 
2b94bf656000-2b94bf94d000 rw-p 00000000 00:00 0 
2b94bf94d000-2b94bf94e000 ---p 00000000 00:00 0 
2b94bf94e000-2b94bf94f000 rw-p 00000000 00:00 0 
2b94bf94f000-2b94bf957000 ---p 00000000 00:00 0 
2b94bf957000-2b94bfb4e000 rw-p 00000000 00:00 0 
2b94bfb4e000-2b94bfb4f000 ---p 00000000 00:00 0 
2b94bfb4f000-2b94bfb50000 rw-p 00000000 00:00 0 
2b94bfb50000-2b94bfb58000 ---p 00000000 00:00 0 
2b94bfb58000-2b94bfd4f000 rw-p 00000000 00:00 0 
2b94bfd4f000-2b94bfd50000 ---p 00000000 00:00 0 
2b94bfd50000-2b94bfd51000 rw-p 00000000 00:00 0 
2b94bfd51000-2b94bfd59000 ---p 00000000 00:00 0 
2b94bfd59000-2b94c0000000 rw-p 00000000 00:00 0 
2b94c0000000-2b94c0021000 rw-p 00000000 00:00 0 
2b94c0021000-2b94c4000000 ---p 00000000 00:00 0 
2b94c4000000-2b94c4021000 rw-p 00000000 00:00 0 
2b94c4021000-2b94c8000000 ---p 00000000 00:00 0 
2b94c8000000-2b94c8021000 rw-p 00000000 00:00 0 
2b94c8021000-2b94cc000000 ---p 00000000 00:00 0 
2b94cc000000-2b94cc120000 rw-p 00000000 00:00 0 
2b94cc120000-2b94d0000000 ---p 00000000 00:00 0 
2b94d0000000-2b94d00a5000 rw-p 00000000 00:00 0 
2b94d00a5000-2b94d4000000 ---p 00000000 00:00 0 
2b94d4000000-2b94d400a000 r-xp 00000000 fe:03 1049063                    /lib/x86_64-linux-gnu/libnss_files-2.24.so
2b94d400a000-2b94d420a000 ---p 0000a000 fe:03 1049063                    /lib/x86_64-linux-gnu/libnss_files-2.24.so
2b94d420a000-2b94d420b000 r--p 0000a000 fe:03 1049063                    /lib/x86_64-linux-gnu/libnss_files-2.24.so
2b94d420b000-2b94d420c000 rw-p 0000b000 fe:03 1049063                    /lib/x86_64-linux-gnu/libnss_files-2.24.so
2b94d420c000-2b94d4212000 rw-p 00000000 00:00 0 
2b94d4212000-2b94d4217000 r-xp 00000000 fe:03 1049062                    /lib/x86_64-linux-gnu/libnss_dns-2.24.so
2b94d4217000-2b94d4416000 ---p 00005000 fe:03 1049062                    /lib/x86_64-linux-gnu/libnss_dns-2.24.so
2b94d4416000-2b94d4417000 r--p 00004000 fe:03 1049062                    /lib/x86_64-linux-gnu/libnss_dns-2.24.so
2b94d4417000-2b94d4418000 rw-p 00005000 fe:03 1049062                    /lib/x86_64-linux-gnu/libnss_dns-2.24.so
2b94d4418000-2b94d442c000 r-xp 00000000 fe:03 1049069                    /lib/x86_64-linux-gnu/libresolv-2.24.so
2b94d442c000-2b94d462b000 ---p 00014000 fe:03 1049069                    /lib/x86_64-linux-gnu/libresolv-2.24.so
2b94d462b000-2b94d462c000 r--p 00013000 fe:03 1049069                    /lib/x86_64-linux-gnu/libresolv-2.24.so
2b94d462c000-2b94d462d000 rw-p 00014000 fe:03 1049069                    /lib/x86_64-linux-gnu/libresolv-2.24.so
2b94d462d000-2b94d462f000 rw-p 00000000 00:00 0 
2b94d462f000-2b94d4635000 r-xp 00000000 fe:03 1056417                    /lib/x86_64-linux-gnu/libnss_db-2.2.3.so
2b94d4635000-2b94d4834000 ---p 00006000 fe:03 1056417                    /lib/x86_64-linux-gnu/libnss_db-2.2.3.so
2b94d4834000-2b94d4835000 r--p 00005000 fe:03 1056417                    /lib/x86_64-linux-gnu/libnss_db-2.2.3.so
2b94d4835000-2b94d4836000 rw-p 00006000 fe:03 1056417                    /lib/x86_64-linux-gnu/libnss_db-2.2.3.so
2b94d4836000-2b94d49ea000 r-xp 00000000 fe:03 917757                     /usr/lib/x86_64-linux-gnu/libdb-5.3.so
2b94d49ea000-2b94d4bea000 ---p 001b4000 fe:03 917757                     /usr/lib/x86_64-linux-gnu/libdb-5.3.so
2b94d4bea000-2b94d4bf1000 r--p 001b4000 fe:03 917757                     /usr/lib/x86_64-linux-gnu/libdb-5.3.so
2b94d4bf1000-2b94d4bf4000 rw-p 001bb000 fe:03 917757                     /usr/lib/x86_64-linux-gnu/libdb-5.3.so
2b94d4bf4000-2b94d4bf5000 ---p 00000000 00:00 0 
2b94d4bf5000-2b94d4bf6000 rw-p 00000000 00:00 0 
2b94d4bf6000-2b94d4bfe000 ---p 00000000 00:00 0 
2b94d4bfe000-2b94d4df5000 rw-p 00000000 00:00 0 
2b94d4df5000-2b94d4df6000 ---p 00000000 00:00 0 
2b94d4df6000-2b94d4df7000 rw-p 00000000 00:00 0 
2b94d4df7000-2b94d4dff000 ---p 00000000 00:00 0 
2b94d4dff000-2b94d4ff6000 rw-p 00000000 00:00 0 
2b94d4ff6000-2b94d515e000 r-xp 00000000 fe:03 957203                     /usr/lib/libmono-btls-shared.so
2b94d515e000-2b94d535d000 ---p 00168000 fe:03 957203                     /usr/lib/libmono-btls-shared.so
2b94d535d000-2b94d536f000 r--p 00167000 fe:03 957203                     /usr/lib/libmono-btls-shared.so
2b94d536f000-2b94d5372000 rw-p 00179000 fe:03 957203                     /usr/lib/libmono-btls-shared.so
2b94d5372000-2b94d5373000 ---p 00000000 00:00 0 
2b94d5373000-2b94d5374000 rw-p 00000000 00:00 0 
2b94d5374000-2b94d537c000 ---p 00000000 00:00 0 
2b94d537c000-2b94d5cf8000 rw-p 00000000 00:00 0 
2b94d5cf8000-2b94d5d7f000 rw-p 00000000 00:00 0 
2b94d5d7f000-2b94d5d87000 rw-p 00000000 00:00 0 
2b94d5d87000-2b94d5e3c000 rw-p 00000000 00:00 0 
2b94d5e3c000-2b94d5e4c000 rw-p 00000000 00:00 0 
2b94d5e4c000-2b94d5e4d000 ---p 00000000 00:00 0 
2b94d5e4d000-2b94d5e4e000 rw-p 00000000 00:00 0 
2b94d5e4e000-2b94d5e56000 ---p 00000000 00:00 0 
2b94d5e56000-2b94d608c000 rw-p 00000000 00:00 0 
2b94d6091000-2b94d6092000 rw-p 00000000 00:00 0 
2b94d6093000-2b94d6094000 rw-p 00000000 00:00 0 
2b94d6099000-2b94d60a3000 rw-p 00000000 00:00 0 
2b94d60a4000-2b94d60a5000 rw-p 00000000 00:00 0 
2b94d60a6000-2b94d60a9000 rw-p 00000000 00:00 0 
2b94d60aa000-2b94d60ac000 rw-p 00000000 00:00 0 
2b94d60b5000-2b94d60b6000 rw-p 00000000 00:00 0 
2b94d60bc000-2b94d60be000 rw-p 00000000 00:00 0 
2b94d9372000-2b94d9373000 ---p 00000000 00:00 0 
2b94d9373000-2b94d9374000 rw-p 00000000 00:00 0 
2b94d9374000-2b94d937c000 ---p 00000000 00:00 0 
2b94d937c000-2b94d9573000 rw-p 00000000 00:00 0 
564150bfd000-56415102a000 r-xp 00000000 fe:03 955613                     /usr/bin/mono-sgen
56415122a000-564151231000 r--p 0042d000 fe:03 955613                     /usr/bin/mono-sgen
564151231000-564151235000 rw-p 00434000 fe:03 955613                     /usr/bin/mono-sgen
564151235000-56415124c000 rw-p 00000000 00:00 0 
56415172c000-56415178f000 rw-p 00000000 00:00 0                          [heap]
564151790000-564151798000 ---p 00000000 00:00 0 
7ffd309d6000-7ffd309f7000 rw-p 00000000 00:00 0                          [stack]
7ffd309f7000-7ffd309f9000 r--p 00000000 00:00 0                          [vvar]
7ffd309f9000-7ffd309fb000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

Native stacktrace:

	mono(+0xca660) [0x564150cc7660]
	/lib/x86_64-linux-gnu/libpthread.so.0(+0x110e0) [0x2b94bbd960e0]
	/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf) [0x2b94bc1ebfff]
	/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a) [0x2b94bc1ed42a]
	mono(+0x2eb099) [0x564150ee8099]
	mono(monoeg_g_logv+0x4d) [0x564150efd90d]
	mono(monoeg_g_log+0x8f) [0x564150efd9df]
	mono(monoeg_malloc+0x36) [0x564150efd4f6]
	mono(mono_mempool_alloc+0x9f) [0x564150de702f]
	mono(+0x145be0) [0x564150d42be0]
	mono(+0x85514) [0x564150c82514]
	mono(+0x5f4cd) [0x564150c5c4cd]
	mono(+0x149338) [0x564150d46338]
	mono(+0x14b3e5) [0x564150d483e5]
	mono(+0x3f38d) [0x564150c3c38d]
	mono(+0xcdcd0) [0x564150ccacd0]
	mono(+0xce568) [0x564150ccb568]
	[0x40db0393]

Debug info from gdb:

[New LWP 13900]
[New LWP 13901]
[New LWP 13902]
[New LWP 13903]
[New LWP 13904]
[New LWP 13905]
[New LWP 13906]
[New LWP 13907]
[New LWP 13908]
[New LWP 13909]
[New LWP 13910]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00002b94bbd9217f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  Id   Target Id         Frame 
* 1    Thread 0x2b94bb490680 (LWP 13899) "mono" 0x00002b94bbd9217f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  2    Thread 0x2b94bc7b8700 (LWP 13900) "SGen worker" 0x00002b94bbd9217f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  3    Thread 0x2b94beb09700 (LWP 13901) "Finalizer" 0x00002b94bbd94556 in do_futex_wait.constprop () from /lib/x86_64-linux-gnu/libpthread.so.0
  4    Thread 0x2b94bf84c700 (LWP 13902) "Timer-Scheduler" 0x00002b94bbd92528 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  5    Thread 0x2b94bfb4d700 (LWP 13903) "mono" 0x00002b94bbd92528 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  6    Thread 0x2b94bfd4e700 (LWP 13904) "Thread Pool Wor" 0x00002b94bbd94720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  7    Thread 0x2b94bff4f700 (LWP 13905) "Thread Pool Wor" 0x00002b94bbd94720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  8    Thread 0x2b94d4df4700 (LWP 13906) "Thread Pool I/O" 0x00002b94bc2988bd in poll () from /lib/x86_64-linux-gnu/libc.so.6
  9    Thread 0x2b94d4ff5700 (LWP 13907) "Thread Pool Wor" 0x00002b94bbd94720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  10   Thread 0x2b94d9572700 (LWP 13908) "Thread Pool Wor" 0x00002b94bbd94720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  11   Thread 0x2b94d5572700 (LWP 13909) "Thread Pool Wor" 0x00002b94bbd95b5a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
  12   Thread 0x2b94d604c700 (LWP 13910) "Thread Pool Wor" 0x00002b94bbd94720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0

Thread 12 (Thread 0x2b94d604c700 (LWP 13910)):
#0  0x00002b94bbd94720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00002b94bbd947ef in __new_sem_wait_slow () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00002b94bbd948a2 in sem_timedwait () from /lib/x86_64-linux-gnu/libpthread.so.0
#3  0x0000564150e8b558 in ?? ()
#4  0x0000564150e2e505 in ?? ()
#5  0x00002b94bbd8c4a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x00002b94bc2a1d0f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 11 (Thread 0x2b94d5572700 (LWP 13909)):
#0  0x00002b94bbd95b5a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x0000564150cc773d in ?? ()
#2  <signal handler called>
#3  0x00002b94bc1ebfff in raise () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00002b94bc1ed42a in abort () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x0000564150ee8099 in ?? ()
#6  0x0000564150efd90d in monoeg_g_logv ()
#7  0x0000564150efd9df in monoeg_g_log ()
#8  0x0000564150efd4f6 in monoeg_malloc ()
#9  0x0000564150de702f in mono_mempool_alloc ()
#10 0x0000564150d42be0 in ?? ()
#11 0x0000564150c82514 in ?? ()
#12 0x0000564150c5c4cd in ?? ()
#13 0x0000564150d46338 in ?? ()
#14 0x0000564150d483e5 in ?? ()
#15 0x0000564150c3c38d in ?? ()
#16 0x0000564150ccacd0 in ?? ()
#17 0x0000564150ccb568 in ?? ()
#18 0x0000000040db0393 in ?? ()
#19 0x00002b94bf1f6015 in ?? ()
#20 0x0000564150e2811c in ?? ()
#21 0x00002b94bf613c28 in ?? ()
#22 0x0000564151750048 in ?? ()
#23 0x00002b94bc8d1340 in ?? ()
#24 0x00002b94bc8d1340 in ?? ()
#25 0x00002b94d55715a0 in ?? ()
#26 0x00002b94d55717c0 in ?? ()
#27 0x00002b94bc8d1340 in ?? ()
#28 0x00002b94bc8ed528 in ?? ()
#29 0x00000000003059f0 in ?? ()
#30 0x0000000000000000 in ?? ()

Thread 10 (Thread 0x2b94d9572700 (LWP 13908)):
#0  0x00002b94bbd94720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00002b94bbd947ef in __new_sem_wait_slow () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00002b94bbd948a2 in sem_timedwait () from /lib/x86_64-linux-gnu/libpthread.so.0
#3  0x0000564150e8b558 in ?? ()
#4  0x0000564150e2e505 in ?? ()
#5  0x00002b94bbd8c4a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x00002b94bc2a1d0f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 9 (Thread 0x2b94d4ff5700 (LWP 13907)):
#0  0x00002b94bbd94720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00002b94bbd947ef in __new_sem_wait_slow () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00002b94bbd948a2 in sem_timedwait () from /lib/x86_64-linux-gnu/libpthread.so.0
#3  0x0000564150e8b558 in ?? ()
#4  0x0000564150e2e505 in ?? ()
#5  0x00002b94bbd8c4a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x00002b94bc2a1d0f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 8 (Thread 0x2b94d4df4700 (LWP 13906)):
#0  0x00002b94bc2988bd in poll () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x0000564150e31485 in ?? ()
#2  0x0000564150e3249a in ?? ()
#3  0x0000564150e2e505 in ?? ()
#4  0x00002b94bbd8c4a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5  0x00002b94bc2a1d0f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 7 (Thread 0x2b94bff4f700 (LWP 13905)):
#0  0x00002b94bbd94720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00002b94bbd947ef in __new_sem_wait_slow () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00002b94bbd948a2 in sem_timedwait () from /lib/x86_64-linux-gnu/libpthread.so.0
#3  0x0000564150e8b558 in ?? ()
#4  0x0000564150e2e505 in ?? ()
#5  0x00002b94bbd8c4a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x00002b94bc2a1d0f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 6 (Thread 0x2b94bfd4e700 (LWP 13904)):
#0  0x00002b94bbd94720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00002b94bbd947ef in __new_sem_wait_slow () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00002b94bbd948a2 in sem_timedwait () from /lib/x86_64-linux-gnu/libpthread.so.0
#3  0x0000564150e8b558 in ?? ()
#4  0x0000564150e2e505 in ?? ()
#5  0x00002b94bbd8c4a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x00002b94bc2a1d0f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 5 (Thread 0x2b94bfb4d700 (LWP 13903)):
#0  0x00002b94bbd92528 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x0000564150eea76b in ?? ()
#2  0x0000564150ef4931 in ?? ()
#3  0x0000564150e8c002 in ?? ()
#4  0x0000564150e2e505 in ?? ()
#5  0x00002b94bbd8c4a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x00002b94bc2a1d0f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 4 (Thread 0x2b94bf84c700 (LWP 13902)):
#0  0x00002b94bbd92528 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x0000564150eea76b in ?? ()
#2  0x0000564150e45255 in ?? ()
#3  0x0000564150e46597 in ?? ()
#4  0x0000564150e46cb5 in ?? ()
#5  0x0000564150e2c1d7 in ?? ()
#6  0x0000000041780bbd in ?? ()
#7  0x00002b94bf5d2670 in ?? ()
#8  0x000000000001869e in ?? ()
#9  0x000000000001869e in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x2b94beb09700 (LWP 13901)):
#0  0x00002b94bbd94556 in do_futex_wait.constprop () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00002b94bbd94604 in __new_sem_wait_slow.constprop.0 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x0000564150e765fa in ?? ()
#3  0x0000564150e2e505 in ?? ()
#4  0x00002b94bbd8c4a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5  0x00002b94bc2a1d0f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 2 (Thread 0x2b94bc7b8700 (LWP 13900)):
#0  0x00002b94bbd9217f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x0000564150eddbcb in ?? ()
#2  0x00002b94bbd8c4a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#3  0x00002b94bc2a1d0f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 1 (Thread 0x2b94bb490680 (LWP 13899)):
#0  0x00002b94bbd9217f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x0000564150eea7cd in ?? ()
#2  0x0000564150e45255 in ?? ()
#3  0x0000564150e4653f in ?? ()
#4  0x0000564150e7f630 in ?? ()
#5  0x000000004178e5de in ?? ()
#6  0x00007ffd309f52c0 in ?? ()
#7  0x0000000000000000 in ?? ()

=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

/mnt/nfs/labstore-secondary-tools-project/wikitasks/wrap.sh: line 4: 13899 Aborted                 (core dumped) mono WikiTasks.exe
[2020.10.28 07:31:26] Task finished

128MiB glibc arena

tools.wikitasks@tools-sgebastion-08:~/wp_cyrlat$ gdb --args mono WikiTasks.exe
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from mono...(no debugging symbols found)...done.
(gdb) catch syscall mmap
Catchpoint 1 (syscall 'mmap' [9])
(gdb) commands
Type commands for breakpoint(s) 1, one per line.
End with a line saying just "end".
>silent
>if $rsi < 50 * 1048576
 >c
 >end
>end
(gdb) r
Starting program: /usr/bin/mono WikiTasks.exe
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff67ff700 (LWP 20010)]
[New Thread 0x7ffff44a4700 (LWP 20011)]
[Switching to Thread 0x7ffff44a4700 (LWP 20011)]
(gdb) bt
#0  0x00007ffff6fda64a in __mmap (addr=addr@entry=0x0, len=len@entry=134217728, prot=prot@entry=0, flags=flags@entry=16418, fd=fd@entry=-1, offset=offset@entry=0)
    at ../sysdeps/unix/sysv/linux/wordsize-64/mmap.c:34
#1  0x00007ffff6f6cad9 in new_heap (size=135168, top_pad=<optimized out>) at arena.c:437
#2  0x00007ffff6f708a6 in _int_new_arena (size=<optimized out>) at arena.c:643
#3  arena_get2 (size=size@entry=1656, avoid_arena=avoid_arena@entry=0x0) at arena.c:875
#4  0x00007ffff6f71c32 in arena_get2 (avoid_arena=0x0, size=1656) at malloc.c:3300
#5  __libc_calloc (n=<optimized out>, elem_size=<optimized out>) at malloc.c:3246
#6  0x000055555585456b in monoeg_g_calloc ()
#7  0x000055555584ae1c in ?? ()
#8  0x0000555555785415 in ?? ()
#9  0x00007ffff74b34a4 in start_thread (arg=0x7ffff44a4700) at pthread_create.c:456
#10 0x00007ffff6fded0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

This matches this line: https://code.woboq.org/userspace/glibc/malloc/arena.c.html#487

p1 = (char *) MMAP (0, HEAP_MAX_SIZE << 1, PROT_NONE, MAP_NORESERVE);

where HEAP_MAX_SIZE is (2 * (4 * 1024 * 1024 * sizeof(long))) = 64 MiB.

AFAICT, with per-thread arenas of glibc, you have the minimum VMS requirement of 128 MiB per-thread, whether the arena is 99% in use or 1% in use.

+1 try: big chuck of errors:

$ cat tmp | awk '{ split($1, a, "-"); printf "%s %08x %s %s\n", a[1], strtonum("0x"a[2]) - strtonum("0x"a[1]), $2, $6 }' | sort -k 2 | tail
2b94bdc00000 003cb000 r--p /usr/lib/mono/4.5/mscorlib.dll
564150bfd000 0042d000 r-xp /usr/bin/mono-sgen
2b94bdfcb000 00568000 r-xp /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
2b94d537c000 0097c000 rw-p 
2b94bc800000 01400000 rw-p 
2b94cc120000 03ee0000 ---p 
2b94d00a5000 03f5b000 ---p 
2b94c0021000 03fdf000 ---p 
2b94c4021000 03fdf000 ---p 
2b94c8021000 03fdf000 ---p 

Smells like glibc arenas. The five non-readable non-writable maps account for 318 MiB.

Vort added a comment.Oct 28 2020, 7:47 AM

My "library" uses async features, which means that C# creates several worker threads, usually equal to CPU core count.
Do this means that this feature automatically allocates core_count * 128 MB memory?
That may be large number. How many cores grid computers have?

That may be large number. How many cores grid computers have?

https://openstack-browser.toolforge.org/project/tools

tools-sgeexec-* are 4 cores.

Vort added a comment.Oct 28 2020, 7:50 AM

So there are no sense in using 512m. Need to try higher values.

Honestly, this just reflects

Thanks. I will try 4g. But it is, of course, not a solution.
If 50 MB program requires 4 GB, then 500 MB program will just fail.
So it is better to find a real cause.

-mem to grid is VMS, which may or may not be proportional to RSS.

In this case VMS has nothing to do with actual memory use. The argument "If 50 MB program requires 4 GB, then 500 MB program will just fail." concerns if VMS is proportional to actual memory use but here it is not. And just because a memory region is "mapped" doesn't mean it's "resident" and actually consuming RAM on the machine. Yes VMS is an upper bound on RSS, but VMS can be consumed by all sorts of random reasons. I'd say just don't worry about VMS.

Though, if you really want to dig this I don't mind continuing....

Vort added a comment.Oct 28 2020, 8:08 AM

I have added catch (OutOfMemoryException e) to GZipUnpack with Syscall.kill(currentPID, Signum.SIGSTOP); as you said.
And lowered -mem to 1536m.
If my changes are correct (and OOM will happen inside GZipUnpack as usual), then at some time process will stop after OOM triggered.
If you have other ideas, you can test them too (as I said, this bot can be executed as many times as needed for debugging).

Build command is mcs Program.cs Api.cs MwApi.cs -r:System.Net.Http -r:Mono.Posix.
Code with changes is located at /mnt/nfs/labstore-secondary-tools-project/wikitasks/wp_cyrlat/_build/.

Vort added a comment.Oct 28 2020, 8:11 AM

In this case VMS has nothing to do with actual memory use.

In 512m case, yes.
I agree that there is no need to worry about that 4 x 128 MB mem maps.

But 1536m turns out to be different thing.
However, will see.

I have added catch (OutOfMemoryException e) to GZipUnpack with Syscall.kill(currentPID, Signum.SIGSTOP); as you said.
And lowered -mem to 1536m.
If my changes are correct (and OOM will happen inside GZipUnpack as usual), then at some time process will stop after OOM triggered.

Is it running? I just ran qstat in the context of your tool and got nothing.

Vort added a comment.Oct 28 2020, 10:54 AM

Is it running?

No.
I think it is not good to run it many-many times.

  1. It will waste resources of API server.
  2. Bug may be related, for example, to the servers load. And reproduce only at 2:00.

Now it will be running as usual, once in a day.
But it will have a code, which should pause process if OOM happens.

However, if you think that it is not a problem, then you may run it several times, nothing bad will happen except for larger resource usage.

Also usually I build binary with Visual Studio 2015 and now it is built with Mono.
So reproduction also may not happen at all.

Vort added a comment.Nov 2 2020, 7:06 AM

@zhuyifei1999 OOM condition is triggered.
Please look at the grid task #1861734

107:10:28 0 ✓ zhuyifei1999@tools-sgeexec-0940: ~$ ps uf -u tools.wikitasks
2USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
3tools.w+ 21372 0.0 0.0 11144 2536 ? Ss 02:00 0:00 /bin/bash /mnt/nfs/labstore-secondary-tools-project/wikitasks/wrap.sh wp_cyrlat
4tools.w+ 21380 0.1 0.7 1535360 61520 ? Tl 02:00 0:28 \_ mono WikiTasks.exe
507:10:31 0 ✓ zhuyifei1999@tools-sgeexec-0940: ~$ cat /proc/21380/maps
6cat: /proc/21380/maps: Permission denied
707:10:54 1 ✗ zhuyifei1999@tools-sgeexec-0940: ~$ sudo cat /proc/21380/maps
840e4b000-40e5b000 rwxp 00000000 00:00 0
941b0c000-41c20000 rwxp 00000000 00:00 0
102b4c7e07f000-2b4c7e0a2000 r-xp 00000000 fe:03 1048724 /lib/x86_64-linux-gnu/ld-2.24.so
112b4c7e0a2000-2b4c7e0a3000 r--p 00000000 fe:03 23796 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
122b4c7e0a3000-2b4c7e0aa000 r--s 00000000 fe:03 959197 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
132b4c7e0aa000-2b4c7e0ab000 r--p 00000000 fe:03 1187673 /usr/lib/locale/chr_US/LC_MEASUREMENT
142b4c7e0ab000-2b4c7e0ac000 r--p 00000000 fe:03 1187679 /usr/lib/locale/chr_US/LC_TELEPHONE
152b4c7e0ac000-2b4c7e0ad000 r--p 00000000 fe:03 23795 /usr/lib/locale/en_US.utf8/LC_ADDRESS
162b4c7e0ad000-2b4c7e0ae000 r--p 00000000 fe:03 1187677 /usr/lib/locale/chr_US/LC_NAME
172b4c7e0ae000-2b4c7e0af000 r--p 00000000 fe:03 1187678 /usr/lib/locale/chr_US/LC_PAPER
182b4c7e0af000-2b4c7e0b0000 r--p 00000000 fe:03 17393 /usr/lib/locale/en_AG/LC_MESSAGES/SYS_LC_MESSAGES
192b4c7e0b0000-2b4c7e0b1000 r--p 00000000 fe:03 1187676 /usr/lib/locale/chr_US/LC_MONETARY
202b4c7e0b1000-2b4c7e0b2000 r--p 00000000 fe:03 23798 /usr/lib/locale/en_US.utf8/LC_TIME
212b4c7e0b2000-2b4c7e0b3000 r--p 00000000 fe:03 1072433 /usr/lib/locale/aa_ET/LC_NUMERIC
222b4c7e0b3000-2b4c7e0b4000 rw-s 00000000 00:13 66522200 /dev/shm/mono.21380
232b4c7e0b4000-2b4c7e0bd000 rw-p 00000000 00:00 0
242b4c7e0bd000-2b4c7e0c3000 rw-p 00000000 00:00 0
252b4c7e0c3000-2b4c7e1f3000 r--p 00000000 fe:03 1066084 /usr/lib/locale/aa_DJ.utf8/LC_COLLATE
262b4c7e1f3000-2b4c7e244000 r--p 00000000 fe:03 1066085 /usr/lib/locale/aa_DJ.utf8/LC_CTYPE
272b4c7e244000-2b4c7e248000 rw-p 00000000 00:00 0
282b4c7e248000-2b4c7e24d000 r--p 00000000 00:2a 91168018 /mnt/nfs/labstore-secondary-tools-project/wikitasks/wp_cyrlat/WikiTasks.exe
292b4c7e24d000-2b4c7e2a2000 rw-p 00000000 00:00 0
302b4c7e2a2000-2b4c7e2a3000 r--p 00023000 fe:03 1048724 /lib/x86_64-linux-gnu/ld-2.24.so
312b4c7e2a3000-2b4c7e2a4000 rw-p 00024000 fe:03 1048724 /lib/x86_64-linux-gnu/ld-2.24.so
322b4c7e2a4000-2b4c7e2a5000 rw-p 00000000 00:00 0
332b4c7e2a5000-2b4c7e3a8000 r-xp 00000000 fe:03 1049049 /lib/x86_64-linux-gnu/libm-2.24.so
342b4c7e3a8000-2b4c7e5a7000 ---p 00103000 fe:03 1049049 /lib/x86_64-linux-gnu/libm-2.24.so
352b4c7e5a7000-2b4c7e5a8000 r--p 00102000 fe:03 1049049 /lib/x86_64-linux-gnu/libm-2.24.so
362b4c7e5a8000-2b4c7e5a9000 rw-p 00103000 fe:03 1049049 /lib/x86_64-linux-gnu/libm-2.24.so
372b4c7e5a9000-2b4c7e5b0000 r-xp 00000000 fe:03 1049066 /lib/x86_64-linux-gnu/librt-2.24.so
382b4c7e5b0000-2b4c7e7af000 ---p 00007000 fe:03 1049066 /lib/x86_64-linux-gnu/librt-2.24.so
392b4c7e7af000-2b4c7e7b0000 r--p 00006000 fe:03 1049066 /lib/x86_64-linux-gnu/librt-2.24.so
402b4c7e7b0000-2b4c7e7b1000 rw-p 00007000 fe:03 1049066 /lib/x86_64-linux-gnu/librt-2.24.so
412b4c7e7b1000-2b4c7e7b4000 r-xp 00000000 fe:03 1048971 /lib/x86_64-linux-gnu/libdl-2.24.so
422b4c7e7b4000-2b4c7e9b3000 ---p 00003000 fe:03 1048971 /lib/x86_64-linux-gnu/libdl-2.24.so
432b4c7e9b3000-2b4c7e9b4000 r--p 00002000 fe:03 1048971 /lib/x86_64-linux-gnu/libdl-2.24.so
442b4c7e9b4000-2b4c7e9b5000 rw-p 00003000 fe:03 1048971 /lib/x86_64-linux-gnu/libdl-2.24.so
452b4c7e9b5000-2b4c7e9cd000 r-xp 00000000 fe:03 1049064 /lib/x86_64-linux-gnu/libpthread-2.24.so
462b4c7e9cd000-2b4c7ebcc000 ---p 00018000 fe:03 1049064 /lib/x86_64-linux-gnu/libpthread-2.24.so
472b4c7ebcc000-2b4c7ebcd000 r--p 00017000 fe:03 1049064 /lib/x86_64-linux-gnu/libpthread-2.24.so
482b4c7ebcd000-2b4c7ebce000 rw-p 00018000 fe:03 1049064 /lib/x86_64-linux-gnu/libpthread-2.24.so
492b4c7ebce000-2b4c7ebd2000 rw-p 00000000 00:00 0
502b4c7ebd2000-2b4c7ebe8000 r-xp 00000000 fe:03 1048593 /lib/x86_64-linux-gnu/libgcc_s.so.1
512b4c7ebe8000-2b4c7ede7000 ---p 00016000 fe:03 1048593 /lib/x86_64-linux-gnu/libgcc_s.so.1
522b4c7ede7000-2b4c7ede8000 r--p 00015000 fe:03 1048593 /lib/x86_64-linux-gnu/libgcc_s.so.1
532b4c7ede8000-2b4c7ede9000 rw-p 00016000 fe:03 1048593 /lib/x86_64-linux-gnu/libgcc_s.so.1
542b4c7ede9000-2b4c7ef7e000 r-xp 00000000 fe:03 1048965 /lib/x86_64-linux-gnu/libc-2.24.so
552b4c7ef7e000-2b4c7f17e000 ---p 00195000 fe:03 1048965 /lib/x86_64-linux-gnu/libc-2.24.so
562b4c7f17e000-2b4c7f182000 r--p 00195000 fe:03 1048965 /lib/x86_64-linux-gnu/libc-2.24.so
572b4c7f182000-2b4c7f184000 rw-p 00199000 fe:03 1048965 /lib/x86_64-linux-gnu/libc-2.24.so
582b4c7f184000-2b4c7f188000 rw-p 00000000 00:00 0
592b4c7f188000-2b4c7f189000 rw-p 00000000 00:00 0
602b4c7f189000-2b4c7f1e8000 ---p 00000000 00:00 0
612b4c7f1e8000-2b4c7f1e9000 ---p 00000000 00:00 0
622b4c7f1e9000-2b4c7f400000 rw-p 00000000 00:00 0
632b4c7f400000-2b4c80800000 rw-p 00000000 00:00 0
642b4c80800000-2b4c80bcb000 r--p 00000000 fe:03 963554 /usr/lib/mono/4.5/mscorlib.dll
652b4c80bcb000-2b4c81133000 r-xp 00000000 fe:03 963690 /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
662b4c81133000-2b4c81332000 ---p 00568000 fe:03 963690 /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
672b4c81332000-2b4c81333000 r--p 00567000 fe:03 963690 /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
682b4c81333000-2b4c81334000 rw-p 00568000 fe:03 963690 /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
692b4c81334000-2b4c8135a000 rw-p 00000000 00:00 0
702b4c8135a000-2b4c8140e000 rw-p 00000000 00:00 0
712b4c8140e000-2b4c8142b000 r--p 00000000 fe:03 963326 /usr/lib/mono/gac/System.Net.Http/4.0.0.0__b03f5f7f11d50a3a/System.Net.Http.dll
722b4c8142b000-2b4c8142d000 rw-p 00000000 00:00 0
732b4c81430000-2b4c8143b000 rw-p 00000000 00:00 0
742b4c8143b000-2b4c8153b000 rw-p 00000000 00:00 0
752b4c8153b000-2b4c8153c000 ---p 00000000 00:00 0
762b4c8153c000-2b4c8153d000 rw-p 00000000 00:00 0
772b4c8153d000-2b4c81545000 ---p 00000000 00:00 0
782b4c81545000-2b4c8173c000 rw-p 00000000 00:00 0
792b4c8173c000-2b4c819e7000 r--p 00000000 fe:03 961780 /usr/lib/mono/gac/System/4.0.0.0__b77a5c561934e089/System.dll
802b4c819e7000-2b4c81adf000 r--p 00000000 fe:03 961796 /usr/lib/mono/gac/System.Core/4.0.0.0__b77a5c561934e089/System.Core.dll
812b4c81adf000-2b4c81afd000 r--p 00000000 fe:03 961776 /usr/lib/mono/gac/System.Configuration/4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
822b4c81afd000-2b4c81dff000 r--p 00000000 fe:03 961792 /usr/lib/mono/gac/System.Xml/4.0.0.0__b77a5c561934e089/System.Xml.dll
832b4c81dff000-2b4c82100000 rw-p 00000000 00:00 0
842b4c82100000-2b4c8214c000 r--p 00000000 fe:03 963542 /usr/lib/mono/gac/Mono.Security/4.0.0.0__0738eb9f132ed756/Mono.Security.dll
852b4c8214c000-2b4c8224c000 rw-p 00000000 00:00 0
862b4c8224c000-2b4c8224d000 ---p 00000000 00:00 0
872b4c8224d000-2b4c8224e000 rw-p 00000000 00:00 0
882b4c8224e000-2b4c82256000 ---p 00000000 00:00 0
892b4c82256000-2b4c8254d000 rw-p 00000000 00:00 0
902b4c8254d000-2b4c8254e000 ---p 00000000 00:00 0
912b4c8254e000-2b4c8254f000 rw-p 00000000 00:00 0
922b4c8254f000-2b4c82557000 ---p 00000000 00:00 0
932b4c82557000-2b4c8274e000 rw-p 00000000 00:00 0
942b4c8274e000-2b4c8274f000 ---p 00000000 00:00 0
952b4c8274f000-2b4c82750000 rw-p 00000000 00:00 0
962b4c82750000-2b4c82758000 ---p 00000000 00:00 0
972b4c82758000-2b4c8294f000 rw-p 00000000 00:00 0
982b4c8294f000-2b4c82950000 ---p 00000000 00:00 0
992b4c82950000-2b4c82951000 rw-p 00000000 00:00 0
1002b4c82951000-2b4c82959000 ---p 00000000 00:00 0
1012b4c82959000-2b4c82b6b000 rw-p 00000000 00:00 0
1022b4c82b6b000-2b4c82b75000 r-xp 00000000 fe:03 1049059 /lib/x86_64-linux-gnu/libnss_files-2.24.so
1032b4c82b75000-2b4c82d75000 ---p 0000a000 fe:03 1049059 /lib/x86_64-linux-gnu/libnss_files-2.24.so
1042b4c82d75000-2b4c82d76000 r--p 0000a000 fe:03 1049059 /lib/x86_64-linux-gnu/libnss_files-2.24.so
1052b4c82d76000-2b4c82d77000 rw-p 0000b000 fe:03 1049059 /lib/x86_64-linux-gnu/libnss_files-2.24.so
1062b4c82d77000-2b4c82d7d000 rw-p 00000000 00:00 0
1072b4c82d7d000-2b4c82d82000 r-xp 00000000 fe:03 1049058 /lib/x86_64-linux-gnu/libnss_dns-2.24.so
1082b4c82d82000-2b4c82f81000 ---p 00005000 fe:03 1049058 /lib/x86_64-linux-gnu/libnss_dns-2.24.so
1092b4c82f81000-2b4c82f82000 r--p 00004000 fe:03 1049058 /lib/x86_64-linux-gnu/libnss_dns-2.24.so
1102b4c82f82000-2b4c82f83000 rw-p 00005000 fe:03 1049058 /lib/x86_64-linux-gnu/libnss_dns-2.24.so
1112b4c82f83000-2b4c82f97000 r-xp 00000000 fe:03 1049065 /lib/x86_64-linux-gnu/libresolv-2.24.so
1122b4c82f97000-2b4c83196000 ---p 00014000 fe:03 1049065 /lib/x86_64-linux-gnu/libresolv-2.24.so
1132b4c83196000-2b4c83197000 r--p 00013000 fe:03 1049065 /lib/x86_64-linux-gnu/libresolv-2.24.so
1142b4c83197000-2b4c83198000 rw-p 00014000 fe:03 1049065 /lib/x86_64-linux-gnu/libresolv-2.24.so
1152b4c83198000-2b4c8319a000 rw-p 00000000 00:00 0
1162b4c8319a000-2b4c831a0000 r-xp 00000000 fe:03 1056417 /lib/x86_64-linux-gnu/libnss_db-2.2.3.so
1172b4c831a0000-2b4c8339f000 ---p 00006000 fe:03 1056417 /lib/x86_64-linux-gnu/libnss_db-2.2.3.so
1182b4c8339f000-2b4c833a0000 r--p 00005000 fe:03 1056417 /lib/x86_64-linux-gnu/libnss_db-2.2.3.so
1192b4c833a0000-2b4c833a1000 rw-p 00006000 fe:03 1056417 /lib/x86_64-linux-gnu/libnss_db-2.2.3.so
1202b4c833a1000-2b4c83555000 r-xp 00000000 fe:03 917757 /usr/lib/x86_64-linux-gnu/libdb-5.3.so
1212b4c83555000-2b4c83755000 ---p 001b4000 fe:03 917757 /usr/lib/x86_64-linux-gnu/libdb-5.3.so
1222b4c83755000-2b4c8375c000 r--p 001b4000 fe:03 917757 /usr/lib/x86_64-linux-gnu/libdb-5.3.so
1232b4c8375c000-2b4c8375f000 rw-p 001bb000 fe:03 917757 /usr/lib/x86_64-linux-gnu/libdb-5.3.so
1242b4c8375f000-2b4c83760000 ---p 00000000 00:00 0
1252b4c83760000-2b4c83761000 rw-p 00000000 00:00 0
1262b4c83761000-2b4c83769000 ---p 00000000 00:00 0
1272b4c83769000-2b4c83960000 rw-p 00000000 00:00 0
1282b4c83960000-2b4c83961000 ---p 00000000 00:00 0
1292b4c83961000-2b4c83962000 rw-p 00000000 00:00 0
1302b4c83962000-2b4c8396a000 ---p 00000000 00:00 0
1312b4c8396a000-2b4c83b61000 rw-p 00000000 00:00 0
1322b4c83b61000-2b4c83b62000 ---p 00000000 00:00 0
1332b4c83b62000-2b4c83b63000 rw-p 00000000 00:00 0
1342b4c83b63000-2b4c83b6b000 ---p 00000000 00:00 0
1352b4c83b6b000-2b4c83d62000 rw-p 00000000 00:00 0
1362b4c83d62000-2b4c83d63000 ---p 00000000 00:00 0
1372b4c83d63000-2b4c83d64000 rw-p 00000000 00:00 0
1382b4c83d64000-2b4c83d6c000 ---p 00000000 00:00 0
1392b4c83d6c000-2b4c83f63000 rw-p 00000000 00:00 0
1402b4c83f63000-2b4c83f97000 r--p 00000000 fe:03 961784 /usr/lib/mono/gac/Mono.Posix/4.0.0.0__0738eb9f132ed756/Mono.Posix.dll
1412b4c83f97000-2b4c83fa0000 rw-p 00000000 00:00 0
1422b4c83fa0000-2b4c83fbc000 rw-p 00000000 00:00 0
1432b4c83fc0000-2b4c83fc3000 rw-p 00000000 00:00 0
1442b4c83fc4000-2b4c83fc7000 rw-p 00000000 00:00 0
1452b4c83fc8000-2b4c83fd8000 rw-p 00000000 00:00 0
1462b4c83fdc000-2b4c83fe4000 rw-p 00000000 00:00 0
1472b4c83fe8000-2b4c83ff0000 rw-p 00000000 00:00 0
1482b4c83ff4000-2b4c84000000 rw-p 00000000 00:00 0
1492b4c84000000-2b4c84021000 rw-p 00000000 00:00 0
1502b4c84021000-2b4c88000000 ---p 00000000 00:00 0
1512b4c88000000-2b4c88021000 rw-p 00000000 00:00 0
1522b4c88021000-2b4c8c000000 ---p 00000000 00:00 0
1532b4c8c000000-2b4c8c021000 rw-p 00000000 00:00 0
1542b4c8c021000-2b4c90000000 ---p 00000000 00:00 0
1552b4c90000000-2b4c9010b000 rw-p 00000000 00:00 0
1562b4c9010b000-2b4c94000000 ---p 00000000 00:00 0
1572b4c94000000-2b4c9411a000 rw-p 00000000 00:00 0
1582b4c9411a000-2b4c98000000 ---p 00000000 00:00 0
1592b4c98000000-2b4c98021000 rw-p 00000000 00:00 0
1602b4c98021000-2b4c9c000000 ---p 00000000 00:00 0
1612b4c9c000000-2b4c9c024000 rw-p 00000000 00:00 0
1622b4c9c024000-2b4ca0000000 ---p 00000000 00:00 0
1632b4ca0000000-2b4ca0126000 rw-p 00000000 00:00 0
1642b4ca0126000-2b4ca4000000 ---p 00000000 00:00 0
1652b4ca4000000-2b4ca4168000 r-xp 00000000 fe:03 961827 /usr/lib/libmono-btls-shared.so
1662b4ca4168000-2b4ca4367000 ---p 00168000 fe:03 961827 /usr/lib/libmono-btls-shared.so
1672b4ca4367000-2b4ca4379000 r--p 00167000 fe:03 961827 /usr/lib/libmono-btls-shared.so
1682b4ca4379000-2b4ca437c000 rw-p 00179000 fe:03 961827 /usr/lib/libmono-btls-shared.so
1692b4ca437c000-2b4ca437d000 ---p 00000000 00:00 0
1702b4ca437d000-2b4ca437e000 rw-p 00000000 00:00 0
1712b4ca437e000-2b4ca4386000 ---p 00000000 00:00 0
1722b4ca4386000-2b4ca457d000 rw-p 00000000 00:00 0
1732b4ca457d000-2b4ca457e000 ---p 00000000 00:00 0
1742b4ca457e000-2b4ca457f000 rw-p 00000000 00:00 0
1752b4ca457f000-2b4ca4587000 ---p 00000000 00:00 0
1762b4ca4587000-2b4ca477e000 rw-p 00000000 00:00 0
1772b4ca477e000-2b4ca47c3000 r-xp 00000000 fe:03 961824 /usr/lib/libMonoPosixHelper.so
1782b4ca47c3000-2b4ca49c2000 ---p 00045000 fe:03 961824 /usr/lib/libMonoPosixHelper.so
1792b4ca49c2000-2b4ca49c3000 r--p 00044000 fe:03 961824 /usr/lib/libMonoPosixHelper.so
1802b4ca49c3000-2b4ca49c4000 rw-p 00045000 fe:03 961824 /usr/lib/libMonoPosixHelper.so
1812b4ca49c4000-2b4ca49c5000 rw-p 00000000 00:00 0
1822b4ca49c5000-2b4ca49de000 r-xp 00000000 fe:03 1048743 /lib/x86_64-linux-gnu/libz.so.1.2.8
1832b4ca49de000-2b4ca4bdd000 ---p 00019000 fe:03 1048743 /lib/x86_64-linux-gnu/libz.so.1.2.8
1842b4ca4bdd000-2b4ca4bde000 r--p 00018000 fe:03 1048743 /lib/x86_64-linux-gnu/libz.so.1.2.8
1852b4ca4bde000-2b4ca4bdf000 rw-p 00019000 fe:03 1048743 /lib/x86_64-linux-gnu/libz.so.1.2.8
1862b4ca4bdf000-2b4ca4f04000 rw-p 00000000 00:00 0
1872b4ca4f08000-2b4ca4f31000 rw-p 00000000 00:00 0
1882b4ca4f33000-2b4ca4f41000 rw-p 00000000 00:00 0
1892b4ca4f43000-2b4ca4f4c000 rw-p 00000000 00:00 0
1902b4ca4f50000-2b4ca4f57000 rw-p 00000000 00:00 0
1912b4ca4f58000-2b4ca4f66000 rw-p 00000000 00:00 0
1922b4ca4f67000-2b4ca4f6b000 rw-p 00000000 00:00 0
1932b4ca4f6c000-2b4ca4f88000 rw-p 00000000 00:00 0
1942b4ca4f89000-2b4ca4f8d000 rw-p 00000000 00:00 0
1952b4ca4f8e000-2b4ca4f8f000 rw-p 00000000 00:00 0
1962b4ca4f90000-2b4ca4f95000 rw-p 00000000 00:00 0
1972b4ca4f97000-2b4ca4f9b000 rw-p 00000000 00:00 0
1982b4ca4f9c000-2b4ca4f9e000 rw-p 00000000 00:00 0
1992b4ca4f9f000-2b4ca4fa3000 rw-p 00000000 00:00 0
2002b4ca4fa4000-2b4ca4fa6000 rw-p 00000000 00:00 0
2012b4ca4fa9000-2b4ca4fab000 rw-p 00000000 00:00 0
2022b4ca4fac000-2b4ca4fb0000 rw-p 00000000 00:00 0
2032b4ca4fb5000-2b4ca4fce000 rw-p 00000000 00:00 0
2042b4ca4fd0000-2b4ca4fe8000 rw-p 00000000 00:00 0
2052b4ca4fec000-2b4ca4ff4000 rw-p 00000000 00:00 0
2062b4ca5068000-2b4ca506c000 rw-p 00000000 00:00 0
2072b4ca50bc000-2b4ca50c0000 rw-p 00000000 00:00 0
2082b4ca50e0000-2b4ca50e8000 rw-p 00000000 00:00 0
2092b4ca5168000-2b4ca516c000 rw-p 00000000 00:00 0
2102b4ca51dc000-2b4ca51e0000 rw-p 00000000 00:00 0
2112b4ca5248000-2b4ca524c000 rw-p 00000000 00:00 0
2122b4ca527c000-2b4ca5280000 rw-p 00000000 00:00 0
2132b4ca5300000-2b4ca5380000 rw-p 00000000 00:00 0
2142b4ca53b8000-2b4ca5400000 rw-p 00000000 00:00 0
2152b4ca5448000-2b4ca544c000 rw-p 00000000 00:00 0
2162b4ca5474000-2b4ca5478000 rw-p 00000000 00:00 0
2172b4ca54b8000-2b4ca54bc000 rw-p 00000000 00:00 0
2182b4ca54ec000-2b4ca54f0000 rw-p 00000000 00:00 0
2192b4ca5520000-2b4ca5524000 rw-p 00000000 00:00 0
2202b4ca5544000-2b4ca5548000 rw-p 00000000 00:00 0
2212b4ca5550000-2b4ca5554000 rw-p 00000000 00:00 0
2222b4ca55c4000-2b4ca55c8000 rw-p 00000000 00:00 0
2232b4ca5620000-2b4ca5634000 rw-p 00000000 00:00 0
2242b4ca564c000-2b4ca565c000 rw-p 00000000 00:00 0
2252b4ca5664000-2b4ca5668000 rw-p 00000000 00:00 0
2262b4ca56c8000-2b4ca56d0000 rw-p 00000000 00:00 0
2272b4ca56f0000-2b4ca56f4000 rw-p 00000000 00:00 0
2282b4ca572c000-2b4ca5730000 rw-p 00000000 00:00 0
2292b4ca5740000-2b4ca5744000 rw-p 00000000 00:00 0
2302b4ca5794000-2b4ca5798000 rw-p 00000000 00:00 0
2312b4ca57d0000-2b4ca57d4000 rw-p 00000000 00:00 0
2322b4ca5824000-2b4ca582c000 rw-p 00000000 00:00 0
2332b4ca583c000-2b4ca5844000 rw-p 00000000 00:00 0
2342b4ca587c000-2b4ca5880000 rw-p 00000000 00:00 0
2352b4ca58e8000-2b4ca58ec000 rw-p 00000000 00:00 0
2362b4ca591c000-2b4ca5920000 rw-p 00000000 00:00 0
2372b4ca5938000-2b4ca593c000 rw-p 00000000 00:00 0
2382b4ca5954000-2b4ca5958000 rw-p 00000000 00:00 0
2392b4ca59b8000-2b4ca59bc000 rw-p 00000000 00:00 0
2402b4ca59ec000-2b4ca59f0000 rw-p 00000000 00:00 0
2412b4ca5a28000-2b4ca5a2c000 rw-p 00000000 00:00 0
2422b4ca5a44000-2b4ca5a4c000 rw-p 00000000 00:00 0
2432b4ca5a6c000-2b4ca5a70000 rw-p 00000000 00:00 0
2442b4ca5ae8000-2b4ca5aec000 rw-p 00000000 00:00 0
2452b4ca5b4c000-2b4ca5b50000 rw-p 00000000 00:00 0
2462b4ca5b98000-2b4ca5b9c000 rw-p 00000000 00:00 0
2472b4ca5c0c000-2b4ca5c10000 rw-p 00000000 00:00 0
2482b4ca5c30000-2b4ca5c34000 rw-p 00000000 00:00 0
2492b4ca5cac000-2b4ca5cb4000 rw-p 00000000 00:00 0
2502b4ca5cfc000-2b4ca5d00000 rw-p 00000000 00:00 0
2512b4ca5d18000-2b4ca5d1c000 rw-p 00000000 00:00 0
2522b4ca5d9c000-2b4ca5da0000 rw-p 00000000 00:00 0
2532b4ca5de8000-2b4ca5dec000 rw-p 00000000 00:00 0
2542b4ca5dfc000-2b4ca5e00000 rw-p 00000000 00:00 0
2552b4ca5e40000-2b4ca5e44000 rw-p 00000000 00:00 0
2562b4ca5e54000-2b4ca5e58000 rw-p 00000000 00:00 0
2572b4ca5ea0000-2b4ca5ea4000 rw-p 00000000 00:00 0
2582b4ca5ec4000-2b4ca5ec8000 rw-p 00000000 00:00 0
2592b4ca5f28000-2b4ca5f2c000 rw-p 00000000 00:00 0
2602b4ca5f64000-2b4ca5f68000 rw-p 00000000 00:00 0
2612b4ca5f88000-2b4ca5f90000 rw-p 00000000 00:00 0
2622b4ca5f98000-2b4ca5fa0000 rw-p 00000000 00:00 0
2632b4ca5fa8000-2b4ca5fac000 rw-p 00000000 00:00 0
2642b4ca5fd4000-2b4ca5fd8000 rw-p 00000000 00:00 0
2652b4ca6008000-2b4ca600c000 rw-p 00000000 00:00 0
2662b4ca603c000-2b4ca6040000 rw-p 00000000 00:00 0
2672b4ca60a0000-2b4ca60ac000 rw-p 00000000 00:00 0
2682b4ca60cc000-2b4ca60d4000 rw-p 00000000 00:00 0
2692b4ca614c000-2b4ca6154000 rw-p 00000000 00:00 0
2702b4ca618c000-2b4ca6190000 rw-p 00000000 00:00 0
2712b4ca61c8000-2b4ca61cc000 rw-p 00000000 00:00 0
2722b4ca620c000-2b4ca6210000 rw-p 00000000 00:00 0
2732b4ca6230000-2b4ca6234000 rw-p 00000000 00:00 0
2742b4ca628c000-2b4ca6290000 rw-p 00000000 00:00 0
2752b4ca62d0000-2b4ca62d4000 rw-p 00000000 00:00 0
2762b4ca630c000-2b4ca6310000 rw-p 00000000 00:00 0
2772b4ca6378000-2b4ca637c000 rw-p 00000000 00:00 0
2782b4ca63e4000-2b4ca63e8000 rw-p 00000000 00:00 0
2792b4ca6458000-2b4ca645c000 rw-p 00000000 00:00 0
2802b4ca64a4000-2b4ca64ac000 rw-p 00000000 00:00 0
2812b4ca64d4000-2b4ca64dc000 rw-p 00000000 00:00 0
2822b4ca64fc000-2b4ca6504000 rw-p 00000000 00:00 0
2832b4ca6584000-2b4ca6588000 rw-p 00000000 00:00 0
2842b4ca6608000-2b4ca660c000 rw-p 00000000 00:00 0
2852b4ca6624000-2b4ca6628000 rw-p 00000000 00:00 0
2862b4ca6660000-2b4ca6668000 rw-p 00000000 00:00 0
2872b4ca66b0000-2b4ca66b4000 rw-p 00000000 00:00 0
2882b4ca66fc000-2b4ca6700000 rw-p 00000000 00:00 0
2892b4ca6710000-2b4ca6714000 rw-p 00000000 00:00 0
2902b4ca678c000-2b4ca6790000 rw-p 00000000 00:00 0
2912b4ca67e8000-2b4ca67f0000 rw-p 00000000 00:00 0
2922b4ca6810000-2b4ca6814000 rw-p 00000000 00:00 0
2932b4ca687c000-2b4ca6880000 rw-p 00000000 00:00 0
2942b4ca68c0000-2b4ca68c4000 rw-p 00000000 00:00 0
2952b4ca68fc000-2b4ca6900000 rw-p 00000000 00:00 0
2962b4ca6940000-2b4ca6944000 rw-p 00000000 00:00 0
2972b4ca6954000-2b4ca6958000 rw-p 00000000 00:00 0
2982b4ca69b8000-2b4ca69bc000 rw-p 00000000 00:00 0
2992b4ca69dc000-2b4ca69e0000 rw-p 00000000 00:00 0
3002b4ca6a00000-2b4ca6a08000 rw-p 00000000 00:00 0
3012b4ca6a30000-2b4ca6a3c000 rw-p 00000000 00:00 0
3022b4ca6aac000-2b4ca6ab0000 rw-p 00000000 00:00 0
3032b4ca6ab8000-2b4ca6abc000 rw-p 00000000 00:00 0
3042b4ca6afc000-2b4ca6b00000 rw-p 00000000 00:00 0
3052b4ca6b28000-2b4ca6b2c000 rw-p 00000000 00:00 0
3062b4ca6b4c000-2b4ca6b50000 rw-p 00000000 00:00 0
3072b4ca6b80000-2b4ca6b88000 rw-p 00000000 00:00 0
3082b4ca6ba8000-2b4ca6bac000 rw-p 00000000 00:00 0
3092b4ca6bfc000-2b4ca6c00000 rw-p 00000000 00:00 0
3102b4ca6c10000-2b4ca6c18000 rw-p 00000000 00:00 0
3112b4ca6c38000-2b4ca6c3c000 rw-p 00000000 00:00 0
3122b4ca6c7c000-2b4ca6c80000 rw-p 00000000 00:00 0
3132b4ca6cc0000-2b4ca6cc4000 rw-p 00000000 00:00 0
3142b4ca6cdc000-2b4ca6ce0000 rw-p 00000000 00:00 0
3152b4ca6d20000-2b4ca6d24000 rw-p 00000000 00:00 0
3162b4ca6d8c000-2b4ca6d90000 rw-p 00000000 00:00 0
3172b4ca6d98000-2b4ca6d9c000 rw-p 00000000 00:00 0
3182b4ca6dac000-2b4ca6db0000 rw-p 00000000 00:00 0
3192b4ca6e30000-2b4ca6e3c000 rw-p 00000000 00:00 0
3202b4ca6e5c000-2b4ca6e64000 rw-p 00000000 00:00 0
3212b4ca6e7c000-2b4ca6e80000 rw-p 00000000 00:00 0
3222b4ca6f00000-2b4ca6f01000 ---p 00000000 00:00 0
3232b4ca6f01000-2b4ca6f02000 rw-p 00000000 00:00 0
3242b4ca6f02000-2b4ca6f0a000 ---p 00000000 00:00 0
3252b4ca6f0a000-2b4ca7101000 rw-p 00000000 00:00 0
3262b4ca7180000-2b4ca7181000 ---p 00000000 00:00 0
3272b4ca7181000-2b4ca7182000 rw-p 00000000 00:00 0
3282b4ca7182000-2b4ca718a000 ---p 00000000 00:00 0
3292b4ca718a000-2b4ca7381000 rw-p 00000000 00:00 0
3302b4ca7381000-2b4ca7382000 ---p 00000000 00:00 0
3312b4ca7382000-2b4ca7383000 rw-p 00000000 00:00 0
3322b4ca7383000-2b4ca738b000 ---p 00000000 00:00 0
3332b4ca738b000-2b4ca7582000 rw-p 00000000 00:00 0
3342b4ca758c000-2b4ca7590000 rw-p 00000000 00:00 0
3352b4ca75d0000-2b4ca75d4000 rw-p 00000000 00:00 0
3362b4ca7624000-2b4ca7628000 rw-p 00000000 00:00 0
3372b4ca7688000-2b4ca7694000 rw-p 00000000 00:00 0
3382b4ca76e4000-2b4ca76e8000 rw-p 00000000 00:00 0
3392b4ca7710000-2b4ca7714000 rw-p 00000000 00:00 0
3402b4ca7754000-2b4ca775c000 rw-p 00000000 00:00 0
3412b4ca77dc000-2b4ca77e4000 rw-p 00000000 00:00 0
3422b4ca7834000-2b4ca7838000 rw-p 00000000 00:00 0
3432b4ca7860000-2b4ca7864000 rw-p 00000000 00:00 0
3442b4ca788c000-2b4ca7890000 rw-p 00000000 00:00 0
3452b4ca78e0000-2b4ca78e4000 rw-p 00000000 00:00 0
3462b4ca7924000-2b4ca7928000 rw-p 00000000 00:00 0
3472b4ca7938000-2b4ca793c000 rw-p 00000000 00:00 0
3482b4ca79a4000-2b4ca79a8000 rw-p 00000000 00:00 0
3492b4ca79d8000-2b4ca79dc000 rw-p 00000000 00:00 0
3502b4ca7a30000-2b4ca7a34000 rw-p 00000000 00:00 0
3512b4ca7a64000-2b4ca7a68000 rw-p 00000000 00:00 0
3522b4ca7ac0000-2b4ca7ac4000 rw-p 00000000 00:00 0
3532b4ca7acc000-2b4ca7ad0000 rw-p 00000000 00:00 0
3542b4ca7b00000-2b4ca7b01000 ---p 00000000 00:00 0
3552b4ca7b01000-2b4ca7b02000 rw-p 00000000 00:00 0
3562b4ca7b02000-2b4ca7b0a000 ---p 00000000 00:00 0
3572b4ca7b0a000-2b4ca7d01000 rw-p 00000000 00:00 0
3582b4ca7d84000-2b4ca7d85000 ---p 00000000 00:00 0
3592b4ca7d85000-2b4ca7d86000 rw-p 00000000 00:00 0
3602b4ca7d86000-2b4ca7d8e000 ---p 00000000 00:00 0
3612b4ca7d8e000-2b4ca7f85000 rw-p 00000000 00:00 0
3622b4ca8000000-2b4ca8107000 rw-p 00000000 00:00 0
3632b4ca8107000-2b4cac000000 ---p 00000000 00:00 0
3642b4cac000000-2b4cac034000 rw-p 00000000 00:00 0
3652b4cac034000-2b4cb0000000 ---p 00000000 00:00 0
3662b4cb0000000-2b4cb0077000 rw-p 00000000 00:00 0
3672b4cb0077000-2b4cb4000000 ---p 00000000 00:00 0
3682b4cb4000000-2b4cb4021000 rw-p 00000000 00:00 0
3692b4cb4021000-2b4cb8000000 ---p 00000000 00:00 0
3702b4cb8000000-2b4cb8021000 rw-p 00000000 00:00 0
3712b4cb8021000-2b4cbc000000 ---p 00000000 00:00 0
3722b4cbc000000-2b4cbc021000 rw-p 00000000 00:00 0
3732b4cbc021000-2b4cc0000000 ---p 00000000 00:00 0
3742b4cc0000000-2b4cc0021000 rw-p 00000000 00:00 0
3752b4cc0021000-2b4cc4000000 ---p 00000000 00:00 0
3762b4cc4028000-2b4cc402c000 rw-p 00000000 00:00 0
3772b4cc403c000-2b4cc4040000 rw-p 00000000 00:00 0
3782b4cc4060000-2b4cc4064000 rw-p 00000000 00:00 0
3792b4cc407c000-2b4cc4080000 rw-p 00000000 00:00 0
3802b4cc40d8000-2b4cc40dc000 rw-p 00000000 00:00 0
3812b4cc40fc000-2b4cc4100000 rw-p 00000000 00:00 0
3822b4cc4140000-2b4cc4144000 rw-p 00000000 00:00 0
3832b4cc419c000-2b4cc41a0000 rw-p 00000000 00:00 0
3842b4cc41d0000-2b4cc41d4000 rw-p 00000000 00:00 0
3852b4cc4234000-2b4cc4238000 rw-p 00000000 00:00 0
3862b4cc4278000-2b4cc427c000 rw-p 00000000 00:00 0
3872b4cc42c4000-2b4cc42cc000 rw-p 00000000 00:00 0
3882b4cc42dc000-2b4cc42e4000 rw-p 00000000 00:00 0
3892b4cc42f4000-2b4cc42f8000 rw-p 00000000 00:00 0
3902b4cc4310000-2b4cc4314000 rw-p 00000000 00:00 0
3912b4cc437c000-2b4cc4380000 rw-p 00000000 00:00 0
3922b4cc4390000-2b4cc4394000 rw-p 00000000 00:00 0
3932b4cc43dc000-2b4cc43e0000 rw-p 00000000 00:00 0
3942b4cc4458000-2b4cc445c000 rw-p 00000000 00:00 0
3952b4cc447c000-2b4cc4480000 rw-p 00000000 00:00 0
3962b4cc4500000-2b4cc4501000 ---p 00000000 00:00 0
3972b4cc4501000-2b4cc4502000 rw-p 00000000 00:00 0
3982b4cc4502000-2b4cc450a000 ---p 00000000 00:00 0
3992b4cc450a000-2b4cc4701000 rw-p 00000000 00:00 0
4002b4cc4701000-2b4cc4702000 ---p 00000000 00:00 0
4012b4cc4702000-2b4cc4703000 rw-p 00000000 00:00 0
4022b4cc4703000-2b4cc470b000 ---p 00000000 00:00 0
4032b4cc470b000-2b4cc4902000 rw-p 00000000 00:00 0
4042b4cc4902000-2b4cc4903000 ---p 00000000 00:00 0
4052b4cc4903000-2b4cc4904000 rw-p 00000000 00:00 0
4062b4cc4904000-2b4cc490c000 ---p 00000000 00:00 0
4072b4cc490c000-2b4cc4b03000 rw-p 00000000 00:00 0
4082b4cc4b03000-2b4cc4b04000 ---p 00000000 00:00 0
4092b4cc4b04000-2b4cc4b05000 rw-p 00000000 00:00 0
4102b4cc4b05000-2b4cc4b0d000 ---p 00000000 00:00 0
4112b4cc4b0d000-2b4cc4d04000 rw-p 00000000 00:00 0
4122b4cc4d3c000-2b4cc4d40000 rw-p 00000000 00:00 0
4132b4cc4d58000-2b4cc4d5c000 rw-p 00000000 00:00 0
4142b4cc4da4000-2b4cc4da8000 rw-p 00000000 00:00 0
4152b4cc4dd8000-2b4cc4ddc000 rw-p 00000000 00:00 0
4162b4cc4e3c000-2b4cc4e40000 rw-p 00000000 00:00 0
4172b4cc4e70000-2b4cc4e74000 rw-p 00000000 00:00 0
4182b4cc4e9c000-2b4cc4ea0000 rw-p 00000000 00:00 0
4192b4cc4ee0000-2b4cc4ee4000 rw-p 00000000 00:00 0
4202b4cc4eec000-2b4cc4ef0000 rw-p 00000000 00:00 0
4212b4cc4f10000-2b4cc4f14000 rw-p 00000000 00:00 0
4222b4cc4f34000-2b4cc4f38000 rw-p 00000000 00:00 0
4232b4cc4f58000-2b4cc4f5c000 rw-p 00000000 00:00 0
4242b4cc4f74000-2b4cc4f78000 rw-p 00000000 00:00 0
4252b4cc4fa8000-2b4cc4fac000 rw-p 00000000 00:00 0
4262b4cc4fec000-2b4cc4ff0000 rw-p 00000000 00:00 0
4272b4cc5000000-2b4cc5004000 rw-p 00000000 00:00 0
4282b4cc501c000-2b4cc5020000 rw-p 00000000 00:00 0
4292b4cc5028000-2b4cc502c000 rw-p 00000000 00:00 0
4302b4cc504c000-2b4cc5050000 rw-p 00000000 00:00 0
4312b4cc5060000-2b4cc5064000 rw-p 00000000 00:00 0
4322b4cc506c000-2b4cc5070000 rw-p 00000000 00:00 0
4332b4cc5080000-2b4cc5084000 rw-p 00000000 00:00 0
4342b4cc508c000-2b4cc5090000 rw-p 00000000 00:00 0
4352b4cc5100000-2b4cc5104000 rw-p 00000000 00:00 0
4362b4cc5138000-2b4cc5140000 rw-p 00000000 00:00 0
4372b4cc5168000-2b4cc516c000 rw-p 00000000 00:00 0
4382b4cc51bc000-2b4cc51c0000 rw-p 00000000 00:00 0
4392b4cc5230000-2b4cc5238000 rw-p 00000000 00:00 0
4402b4cc5258000-2b4cc5260000 rw-p 00000000 00:00 0
4412b4cc5290000-2b4cc5298000 rw-p 00000000 00:00 0
4422b4cc5310000-2b4cc5314000 rw-p 00000000 00:00 0
4432b4cc535c000-2b4cc5360000 rw-p 00000000 00:00 0
4442b4cc53a0000-2b4cc53a4000 rw-p 00000000 00:00 0
4452b4cc5414000-2b4cc5418000 rw-p 00000000 00:00 0
4462b4cc5420000-2b4cc5424000 rw-p 00000000 00:00 0
4472b4cc5434000-2b4cc5438000 rw-p 00000000 00:00 0
4482b4cc54b8000-2b4cc54bc000 rw-p 00000000 00:00 0
4492b4cc54e4000-2b4cc54e8000 rw-p 00000000 00:00 0
4502b4cc5528000-2b4cc552c000 rw-p 00000000 00:00 0
4512b4cc5544000-2b4cc5548000 rw-p 00000000 00:00 0
4522b4cc5560000-2b4cc5564000 rw-p 00000000 00:00 0
4532b4cc556c000-2b4cc5570000 rw-p 00000000 00:00 0
4542b4cc5578000-2b4cc557c000 rw-p 00000000 00:00 0
4552b4cc55ac000-2b4cc55b0000 rw-p 00000000 00:00 0
4562b4cc5618000-2b4cc561c000 rw-p 00000000 00:00 0
4572b4cc562c000-2b4cc5634000 rw-p 00000000 00:00 0
4582b4cc569c000-2b4cc56a0000 rw-p 00000000 00:00 0
4592b4cc56e0000-2b4cc56e4000 rw-p 00000000 00:00 0
4602b4cc5724000-2b4cc5728000 rw-p 00000000 00:00 0
4612b4cc57a8000-2b4cc57ac000 rw-p 00000000 00:00 0
4622b4cc57b4000-2b4cc57b8000 rw-p 00000000 00:00 0
4632b4cc57f8000-2b4cc57fc000 rw-p 00000000 00:00 0
4642b4cc5834000-2b4cc5838000 rw-p 00000000 00:00 0
4652b4cc5878000-2b4cc587c000 rw-p 00000000 00:00 0
4662b4cc58e4000-2b4cc58ec000 rw-p 00000000 00:00 0
4672b4cc593c000-2b4cc5940000 rw-p 00000000 00:00 0
4682b4cc5960000-2b4cc5964000 rw-p 00000000 00:00 0
4692b4cc599c000-2b4cc59a0000 rw-p 00000000 00:00 0
4702b4cc59e8000-2b4cc59ec000 rw-p 00000000 00:00 0
4712b4cc5a6c000-2b4cc5a70000 rw-p 00000000 00:00 0
4722b4cc5a80000-2b4cc5a88000 rw-p 00000000 00:00 0
4732b4cc5a90000-2b4cc5a94000 rw-p 00000000 00:00 0
4742b4cc5b04000-2b4cc5b08000 rw-p 00000000 00:00 0
4752b4cc5b80000-2b4cc5b88000 rw-p 00000000 00:00 0
4762b4cc5bc8000-2b4cc5bd0000 rw-p 00000000 00:00 0
4772b4cc5c48000-2b4cc5c4c000 rw-p 00000000 00:00 0
4782b4cc5ccc000-2b4cc5cd4000 rw-p 00000000 00:00 0
4792b4cc5d04000-2b4cc5d0c000 rw-p 00000000 00:00 0
4802b4cc5d84000-2b4cc5d88000 rw-p 00000000 00:00 0
4812b4cc5da8000-2b4cc5dac000 rw-p 00000000 00:00 0
4822b4cc5e04000-2b4cc5e08000 rw-p 00000000 00:00 0
4832b4cc5e18000-2b4cc5e1c000 rw-p 00000000 00:00 0
4842b4cc5e24000-2b4cc5e2c000 rw-p 00000000 00:00 0
4852b4cc5eac000-2b4cc5eb0000 rw-p 00000000 00:00 0
4862b4cc5ec8000-2b4cc5ed0000 rw-p 00000000 00:00 0
4872b4cc5f08000-2b4cc5f0c000 rw-p 00000000 00:00 0
4882b4cc5f2c000-2b4cc5f30000 rw-p 00000000 00:00 0
4892b4cc5f78000-2b4cc5f7c000 rw-p 00000000 00:00 0
4902b4cc5f94000-2b4cc5f98000 rw-p 00000000 00:00 0
4912b4cc5ff0000-2b4cc5ff4000 rw-p 00000000 00:00 0
4922b4cc6014000-2b4cc6018000 rw-p 00000000 00:00 0
4932b4cc6088000-2b4cc608c000 rw-p 00000000 00:00 0
4942b4cc60fc000-2b4cc6100000 rw-p 00000000 00:00 0
4952b4cc6158000-2b4cc6160000 rw-p 00000000 00:00 0
4962b4cc6190000-2b4cc6194000 rw-p 00000000 00:00 0
4972b4cc619c000-2b4cc61a0000 rw-p 00000000 00:00 0
4982b4cc61e8000-2b4cc61ec000 rw-p 00000000 00:00 0
4992b4cc622c000-2b4cc6230000 rw-p 00000000 00:00 0
5002b4cc6240000-2b4cc6244000 rw-p 00000000 00:00 0
5012b4cc6254000-2b4cc6258000 rw-p 00000000 00:00 0
5022b4cc62c0000-2b4cc62c8000 rw-p 00000000 00:00 0
5032b4cc62d0000-2b4cc62d4000 rw-p 00000000 00:00 0
5042b4cc631c000-2b4cc6320000 rw-p 00000000 00:00 0
5052b4cc6370000-2b4cc6374000 rw-p 00000000 00:00 0
5062b4cc63dc000-2b4cc63e0000 rw-p 00000000 00:00 0
5072b4cc6460000-2b4cc6468000 rw-p 00000000 00:00 0
5082b4cc6498000-2b4cc649c000 rw-p 00000000 00:00 0
5092b4cc64c4000-2b4cc64c8000 rw-p 00000000 00:00 0
5102b4cc6508000-2b4cc650c000 rw-p 00000000 00:00 0
5112b4cc651c000-2b4cc6520000 rw-p 00000000 00:00 0
5122b4cc6580000-2b4cc6584000 rw-p 00000000 00:00 0
5132b4cc65bc000-2b4cc65c0000 rw-p 00000000 00:00 0
5142b4cc65d0000-2b4cc65d4000 rw-p 00000000 00:00 0
5152b4cc664c000-2b4cc6650000 rw-p 00000000 00:00 0
5162b4cc6680000-2b4cc6684000 rw-p 00000000 00:00 0
5172b4cc66fc000-2b4cc6700000 rw-p 00000000 00:00 0
5182b4cc6738000-2b4cc673c000 rw-p 00000000 00:00 0
5192b4cc678c000-2b4cc6790000 rw-p 00000000 00:00 0
5202b4cc67b8000-2b4cc67bc000 rw-p 00000000 00:00 0
5212b4cc67c4000-2b4cc67cc000 rw-p 00000000 00:00 0
5222b4cc67ec000-2b4cc67f0000 rw-p 00000000 00:00 0
5232b4cc6848000-2b4cc684c000 rw-p 00000000 00:00 0
5242b4cc6894000-2b4cc6898000 rw-p 00000000 00:00 0
5252b4cc68c8000-2b4cc68cc000 rw-p 00000000 00:00 0
5262b4cc68e4000-2b4cc68e8000 rw-p 00000000 00:00 0
5272b4cc68f8000-2b4cc68fc000 rw-p 00000000 00:00 0
5282b4cc6914000-2b4cc6918000 rw-p 00000000 00:00 0
5292b4cc6958000-2b4cc6960000 rw-p 00000000 00:00 0
5302b4cc6980000-2b4cc6984000 rw-p 00000000 00:00 0
5312b4cc69fc000-2b4cc6a00000 rw-p 00000000 00:00 0
5322b4cc6a60000-2b4cc6a64000 rw-p 00000000 00:00 0
5332b4cc6ae4000-2b4cc6ae8000 rw-p 00000000 00:00 0
5342b4cc6b20000-2b4cc6b28000 rw-p 00000000 00:00 0
5352b4cc6b38000-2b4cc6b40000 rw-p 00000000 00:00 0
5362b4cc6b80000-2b4cc6b84000 rw-p 00000000 00:00 0
5372b4cc6be4000-2b4cc6be8000 rw-p 00000000 00:00 0
5382b4cc6c20000-2b4cc6c24000 rw-p 00000000 00:00 0
5392b4cc6c2c000-2b4cc6c30000 rw-p 00000000 00:00 0
5402b4cc6c38000-2b4cc6c3c000 rw-p 00000000 00:00 0
5412b4cc6cbc000-2b4cc6cc0000 rw-p 00000000 00:00 0
5422b4cc6ce8000-2b4cc6cfc000 rw-p 00000000 00:00 0
5432b4cc6d04000-2b4cc6d08000 rw-p 00000000 00:00 0
5442b4cc6d48000-2b4cc6d4c000 rw-p 00000000 00:00 0
5452b4cc6d64000-2b4cc6d68000 rw-p 00000000 00:00 0
5462b4cc6da8000-2b4cc6dac000 rw-p 00000000 00:00 0
5472b4cc6dbc000-2b4cc6dc4000 rw-p 00000000 00:00 0
5482b4cc6e04000-2b4cc6e08000 rw-p 00000000 00:00 0
5492b4cc6e10000-2b4cc6e14000 rw-p 00000000 00:00 0
5502b4cc6e2c000-2b4cc6e30000 rw-p 00000000 00:00 0
5512b4cc6e40000-2b4cc6e44000 rw-p 00000000 00:00 0
5522b4cc6e5c000-2b4cc6e60000 rw-p 00000000 00:00 0
5532b4cc6ea8000-2b4cc6eac000 rw-p 00000000 00:00 0
5542b4cc6f04000-2b4cc6f08000 rw-p 00000000 00:00 0
5552b4cc6f78000-2b4cc6f7c000 rw-p 00000000 00:00 0
5562b4cc6f84000-2b4cc6f88000 rw-p 00000000 00:00 0
5572b4cc6f90000-2b4cc6f94000 rw-p 00000000 00:00 0
5582b4cc6fac000-2b4cc6fb0000 rw-p 00000000 00:00 0
5592b4cc6fc0000-2b4cc6fc4000 rw-p 00000000 00:00 0
5602b4cc6fd4000-2b4cc6fdc000 rw-p 00000000 00:00 0
5612b4cc6fe4000-2b4cc6fe8000 rw-p 00000000 00:00 0
5622b4cc7018000-2b4cc701c000 rw-p 00000000 00:00 0
5632b4cc702c000-2b4cc7030000 rw-p 00000000 00:00 0
5642b4cc7058000-2b4cc705c000 rw-p 00000000 00:00 0
5652b4cc706c000-2b4cc7070000 rw-p 00000000 00:00 0
5662b4cc70b0000-2b4cc70b4000 rw-p 00000000 00:00 0
5672b4cc70ec000-2b4cc70f0000 rw-p 00000000 00:00 0
5682b4cc70f8000-2b4cc7100000 rw-p 00000000 00:00 0
5692b4cc7130000-2b4cc7134000 rw-p 00000000 00:00 0
5702b4cc715c000-2b4cc7160000 rw-p 00000000 00:00 0
5712b4cc7198000-2b4cc719c000 rw-p 00000000 00:00 0
5722b4cc71bc000-2b4cc71c0000 rw-p 00000000 00:00 0
5732b4cc71e0000-2b4cc71e4000 rw-p 00000000 00:00 0
5742b4cc71fc000-2b4cc7200000 rw-p 00000000 00:00 0
5752b4cc7230000-2b4cc7234000 rw-p 00000000 00:00 0
5762b4cc7244000-2b4cc7248000 rw-p 00000000 00:00 0
5772b4cc72a8000-2b4cc72b4000 rw-p 00000000 00:00 0
5782b4cc72c4000-2b4cc72c8000 rw-p 00000000 00:00 0
5792b4cc7338000-2b4cc733c000 rw-p 00000000 00:00 0
5802b4cc734c000-2b4cc7350000 rw-p 00000000 00:00 0
5812b4cc7358000-2b4cc735c000 rw-p 00000000 00:00 0
5822b4cc737c000-2b4cc7380000 rw-p 00000000 00:00 0
5832b4cc73b8000-2b4cc73bc000 rw-p 00000000 00:00 0
5842b4cc73dc000-2b4cc73e0000 rw-p 00000000 00:00 0
5852b4cc7430000-2b4cc7434000 rw-p 00000000 00:00 0
5862b4cc744c000-2b4cc7450000 rw-p 00000000 00:00 0
5872b4cc7458000-2b4cc745c000 rw-p 00000000 00:00 0
5882b4cc74a4000-2b4cc74a8000 rw-p 00000000 00:00 0
5892b4cc74b0000-2b4cc74b4000 rw-p 00000000 00:00 0
5902b4cc74ec000-2b4cc74f0000 rw-p 00000000 00:00 0
5912b4cc7510000-2b4cc7514000 rw-p 00000000 00:00 0
5922b4cc7594000-2b4cc7598000 rw-p 00000000 00:00 0
5932b4cc7618000-2b4cc761c000 rw-p 00000000 00:00 0
5942b4cc765c000-2b4cc7660000 rw-p 00000000 00:00 0
5952b4cc7670000-2b4cc7674000 rw-p 00000000 00:00 0
5962b4cc767c000-2b4cc7680000 rw-p 00000000 00:00 0
5972b4cc76b0000-2b4cc76b4000 rw-p 00000000 00:00 0
5982b4cc76fc000-2b4cc7700000 rw-p 00000000 00:00 0
5992b4cc7710000-2b4cc7714000 rw-p 00000000 00:00 0
6002b4cc771c000-2b4cc7720000 rw-p 00000000 00:00 0
6012b4cc7740000-2b4cc7744000 rw-p 00000000 00:00 0
6022b4cc776c000-2b4cc7770000 rw-p 00000000 00:00 0
6032b4cc7790000-2b4cc7794000 rw-p 00000000 00:00 0
6042b4cc77e4000-2b4cc77ec000 rw-p 00000000 00:00 0
6052b4cc7814000-2b4cc7818000 rw-p 00000000 00:00 0
6062b4cc7850000-2b4cc7854000 rw-p 00000000 00:00 0
6072b4cc789c000-2b4cc78a0000 rw-p 00000000 00:00 0
6082b4cc7920000-2b4cc7924000 rw-p 00000000 00:00 0
6092b4cc7954000-2b4cc7958000 rw-p 00000000 00:00 0
6102b4cc7988000-2b4cc7994000 rw-p 00000000 00:00 0
6112b4cc7a04000-2b4cc7a08000 rw-p 00000000 00:00 0
6122b4cc7a58000-2b4cc7a5c000 rw-p 00000000 00:00 0
6132b4cc7a84000-2b4cc7a88000 rw-p 00000000 00:00 0
6142b4cc7af8000-2b4cc7b00000 rw-p 00000000 00:00 0
6152b4cc7b28000-2b4cc7b2c000 rw-p 00000000 00:00 0
6162b4cc7b44000-2b4cc7b48000 rw-p 00000000 00:00 0
6172b4cc7b70000-2b4cc7b74000 rw-p 00000000 00:00 0
6182b4cc7b94000-2b4cc7b98000 rw-p 00000000 00:00 0
6192b4cc7bf8000-2b4cc7bfc000 rw-p 00000000 00:00 0
6202b4cc7c7c000-2b4cc7c80000 rw-p 00000000 00:00 0
6212b4cc7cd8000-2b4cc7ce0000 rw-p 00000000 00:00 0
6222b4cc7ce8000-2b4cc7cec000 rw-p 00000000 00:00 0
6232b4cc7cfc000-2b4cc7d00000 rw-p 00000000 00:00 0
6242b4cc7d08000-2b4cc7d0c000 rw-p 00000000 00:00 0
6252b4cc7d5c000-2b4cc7d60000 rw-p 00000000 00:00 0
6262b4cc7d78000-2b4cc7d7c000 rw-p 00000000 00:00 0
6272b4cc7dfc000-2b4cc7e00000 rw-p 00000000 00:00 0
6282b4cc7e10000-2b4cc7e14000 rw-p 00000000 00:00 0
6292b4cc7e24000-2b4cc7e28000 rw-p 00000000 00:00 0
6302b4cc7e80000-2b4cc7e84000 rw-p 00000000 00:00 0
6312b4cc7e94000-2b4cc7e98000 rw-p 00000000 00:00 0
6322b4cc7ef0000-2b4cc7ef4000 rw-p 00000000 00:00 0
6332b4cc7efc000-2b4cc7f00000 rw-p 00000000 00:00 0
6342b4cc7f50000-2b4cc7f54000 rw-p 00000000 00:00 0
6352b4cc7f7c000-2b4cc7f80000 rw-p 00000000 00:00 0
6362b4cc8000000-2b4cc8021000 rw-p 00000000 00:00 0
6372b4cc8021000-2b4ccc000000 ---p 00000000 00:00 0
6382b4ccc000000-2b4ccc021000 rw-p 00000000 00:00 0
6392b4ccc021000-2b4cd0000000 ---p 00000000 00:00 0
6402b4cd0000000-2b4cd0021000 rw-p 00000000 00:00 0
6412b4cd0021000-2b4cd4000000 ---p 00000000 00:00 0
6422b4cd4000000-2b4cd4021000 rw-p 00000000 00:00 0
6432b4cd4021000-2b4cd8000000 ---p 00000000 00:00 0
6442b4cd8000000-2b4cd8021000 rw-p 00000000 00:00 0
6452b4cd8021000-2b4cdc000000 ---p 00000000 00:00 0
6462b4cdc020000-2b4cdc024000 rw-p 00000000 00:00 0
6472b4cdc07c000-2b4cdc080000 rw-p 00000000 00:00 0
6482b4cdc0c8000-2b4cdc0cc000 rw-p 00000000 00:00 0
6492b4cdc0dc000-2b4cdc0e0000 rw-p 00000000 00:00 0
6502b4cdc118000-2b4cdc11c000 rw-p 00000000 00:00 0
6512b4cdc174000-2b4cdc178000 rw-p 00000000 00:00 0
6522b4cdc1a8000-2b4cdc1ac000 rw-p 00000000 00:00 0
6532b4cdc1b4000-2b4cdc1b8000 rw-p 00000000 00:00 0
6542b4cdc1e8000-2b4cdc1ec000 rw-p 00000000 00:00 0
6552b4cdc1fc000-2b4cdc200000 rw-p 00000000 00:00 0
6562b4cdc238000-2b4cdc23c000 rw-p 00000000 00:00 0
6572b4cdc25c000-2b4cdc260000 rw-p 00000000 00:00 0
6582b4cdc2c0000-2b4cdc2c4000 rw-p 00000000 00:00 0
6592b4cdc2fc000-2b4cdc300000 rw-p 00000000 00:00 0
6602b4cdc340000-2b4cdc344000 rw-p 00000000 00:00 0
6612b4cdc37c000-2b4cdc380000 rw-p 00000000 00:00 0
6622b4cdc3f8000-2b4cdc3fc000 rw-p 00000000 00:00 0
6632b4cdc41c000-2b4cdc420000 rw-p 00000000 00:00 0
6642b4cdc468000-2b4cdc46c000 rw-p 00000000 00:00 0
6652b4cdc474000-2b4cdc478000 rw-p 00000000 00:00 0
6662b4cdc4a8000-2b4cdc4ac000 rw-p 00000000 00:00 0
6672b4cdc52c000-2b4cdc530000 rw-p 00000000 00:00 0
6682b4cdc580000-2b4cdc584000 rw-p 00000000 00:00 0
6692b4cdc5ac000-2b4cdc5b4000 rw-p 00000000 00:00 0
6702b4cdc614000-2b4cdc618000 rw-p 00000000 00:00 0
6712b4cdc650000-2b4cdc654000 rw-p 00000000 00:00 0
6722b4cdc694000-2b4cdc698000 rw-p 00000000 00:00 0
6732b4cdc718000-2b4cdc71c000 rw-p 00000000 00:00 0
6742b4cdc784000-2b4cdc788000 rw-p 00000000 00:00 0
6752b4cdc7a8000-2b4cdc7ac000 rw-p 00000000 00:00 0
6762b4cdc7fc000-2b4cdc800000 rw-p 00000000 00:00 0
6772b4cdc848000-2b4cdc850000 rw-p 00000000 00:00 0
6782b4cdc860000-2b4cdc868000 rw-p 00000000 00:00 0
6792b4cdc8e8000-2b4cdc8ec000 rw-p 00000000 00:00 0
6802b4cdc8fc000-2b4cdc900000 rw-p 00000000 00:00 0
6812b4cdc920000-2b4cdc928000 rw-p 00000000 00:00 0
6822b4cdc940000-2b4cdc944000 rw-p 00000000 00:00 0
6832b4cdc97c000-2b4cdc980000 rw-p 00000000 00:00 0
6842b4cdc998000-2b4cdc99c000 rw-p 00000000 00:00 0
6852b4cdc9a4000-2b4cdc9a8000 rw-p 00000000 00:00 0
6862b4cdca10000-2b4cdca14000 rw-p 00000000 00:00 0
6872b4cdca54000-2b4cdca58000 rw-p 00000000 00:00 0
6882b4cdcad0000-2b4cdcad4000 rw-p 00000000 00:00 0
6892b4cdcaf4000-2b4cdcaf8000 rw-p 00000000 00:00 0
6902b4cdcb28000-2b4cdcb2c000 rw-p 00000000 00:00 0
6912b4cdcb9c000-2b4cdcba0000 rw-p 00000000 00:00 0
6922b4cdcc00000-2b4cdcc04000 rw-p 00000000 00:00 0
6932b4cdcc74000-2b4cdcc78000 rw-p 00000000 00:00 0
6942b4cdcc80000-2b4cdcc84000 rw-p 00000000 00:00 0
6952b4cdccac000-2b4cdccb0000 rw-p 00000000 00:00 0
6962b4cdccd0000-2b4cdccd4000 rw-p 00000000 00:00 0
6972b4cdccfc000-2b4cdcd00000 rw-p 00000000 00:00 0
6982b4cdcd48000-2b4cdcd4c000 rw-p 00000000 00:00 0
6992b4cdcd6c000-2b4cdcd70000 rw-p 00000000 00:00 0
7002b4cdcd88000-2b4cdcd8c000 rw-p 00000000 00:00 0
7012b4cdcdfc000-2b4cdce00000 rw-p 00000000 00:00 0
7022b4cdce80000-2b4cdce84000 rw-p 00000000 00:00 0
7032b4cdcee4000-2b4cdcee8000 rw-p 00000000 00:00 0
7042b4cdcf50000-2b4cdcf54000 rw-p 00000000 00:00 0
7052b4cdcf7c000-2b4cdcf80000 rw-p 00000000 00:00 0
7062b4cdcf98000-2b4cdcf9c000 rw-p 00000000 00:00 0
7072b4cdcfec000-2b4cdcff0000 rw-p 00000000 00:00 0
7082b4cdd010000-2b4cdd014000 rw-p 00000000 00:00 0
7092b4cdd06c000-2b4cdd070000 rw-p 00000000 00:00 0
7102b4cdd078000-2b4cdd07c000 rw-p 00000000 00:00 0
7112b4cdd0fc000-2b4cdd100000 rw-p 00000000 00:00 0
7122b4cdd158000-2b4cdd15c000 rw-p 00000000 00:00 0
7132b4cdd164000-2b4cdd16c000 rw-p 00000000 00:00 0
7142b4cdd194000-2b4cdd19c000 rw-p 00000000 00:00 0
7152b4cdd204000-2b4cdd208000 rw-p 00000000 00:00 0
7162b4cdd210000-2b4cdd214000 rw-p 00000000 00:00 0
7172b4cdd254000-2b4cdd258000 rw-p 00000000 00:00 0
7182b4cdd278000-2b4cdd27c000 rw-p 00000000 00:00 0
7192b4cdd2dc000-2b4cdd2e0000 rw-p 00000000 00:00 0
7202b4cdd2e8000-2b4cdd2ec000 rw-p 00000000 00:00 0
7212b4cdd2fc000-2b4cdd300000 rw-p 00000000 00:00 0
7222b4cdd320000-2b4cdd328000 rw-p 00000000 00:00 0
7232b4cdd380000-2b4cdd384000 rw-p 00000000 00:00 0
7242b4cdd3a4000-2b4cdd3a8000 rw-p 00000000 00:00 0
7252b4cdd3d0000-2b4cdd3d4000 rw-p 00000000 00:00 0
7262b4cdd424000-2b4cdd428000 rw-p 00000000 00:00 0
7272b4cdd490000-2b4cdd494000 rw-p 00000000 00:00 0
7282b4cdd4a4000-2b4cdd4a8000 rw-p 00000000 00:00 0
7292b4cdd510000-2b4cdd514000 rw-p 00000000 00:00 0
7302b4cdd594000-2b4cdd598000 rw-p 00000000 00:00 0
7312b4cdd5c0000-2b4cdd5c4000 rw-p 00000000 00:00 0
7322b4cdd624000-2b4cdd628000 rw-p 00000000 00:00 0
7332b4cdd640000-2b4cdd644000 rw-p 00000000 00:00 0
7342b4cdd6a4000-2b4cdd6a8000 rw-p 00000000 00:00 0
7352b4cdd6b8000-2b4cdd6bc000 rw-p 00000000 00:00 0
7362b4cdd6c4000-2b4cdd6c8000 rw-p 00000000 00:00 0
7372b4cdd748000-2b4cdd750000 rw-p 00000000 00:00 0
7382b4cdd760000-2b4cdd764000 rw-p 00000000 00:00 0
7392b4cdd784000-2b4cdd78c000 rw-p 00000000 00:00 0
7402b4cdd7bc000-2b4cdd7c0000 rw-p 00000000 00:00 0
7412b4cdd7e0000-2b4cdd7e4000 rw-p 00000000 00:00 0
7422b4cdd7fc000-2b4cdd800000 rw-p 00000000 00:00 0
7432b4cdd810000-2b4cdd814000 rw-p 00000000 00:00 0
7442b4cdd84c000-2b4cdd850000 rw-p 00000000 00:00 0
7452b4cdd868000-2b4cdd86c000 rw-p 00000000 00:00 0
7462b4cdd88c000-2b4cdd890000 rw-p 00000000 00:00 0
7472b4cdd8f0000-2b4cdd8f8000 rw-p 00000000 00:00 0
7482b4cdd938000-2b4cdd93c000 rw-p 00000000 00:00 0
7492b4cdd97c000-2b4cdd980000 rw-p 00000000 00:00 0
7502b4cdd9a0000-2b4cdd9a4000 rw-p 00000000 00:00 0
7512b4cdd9e4000-2b4cdd9ec000 rw-p 00000000 00:00 0
7522b4cddabc000-2b4cddac0000 rw-p 00000000 00:00 0
7532b4cddac8000-2b4cddacc000 rw-p 00000000 00:00 0
7542b4cddaf4000-2b4cddaf8000 rw-p 00000000 00:00 0
7552b4cddb08000-2b4cddb0c000 rw-p 00000000 00:00 0
7562b4cddb24000-2b4cddb28000 rw-p 00000000 00:00 0
7572b4cddb48000-2b4cddb4c000 rw-p 00000000 00:00 0
7582b4cddb74000-2b4cddb78000 rw-p 00000000 00:00 0
7592b4cddbb8000-2b4cddbbc000 rw-p 00000000 00:00 0
7602b4cddc04000-2b4cddc08000 rw-p 00000000 00:00 0
7612b4cddc10000-2b4cddc14000 rw-p 00000000 00:00 0
7622b4cddccc000-2b4cddcd0000 rw-p 00000000 00:00 0
7632b4cddce0000-2b4cddce4000 rw-p 00000000 00:00 0
7642b4cddcf4000-2b4cddcf8000 rw-p 00000000 00:00 0
7652b4cddd08000-2b4cddd0c000 rw-p 00000000 00:00 0
7662b4cddd24000-2b4cddd28000 rw-p 00000000 00:00 0
7672b4cddd68000-2b4cddd70000 rw-p 00000000 00:00 0
7682b4cddde8000-2b4cdddec000 rw-p 00000000 00:00 0
7692b4cdde0c000-2b4cdde10000 rw-p 00000000 00:00 0
7702b4cdde20000-2b4cdde24000 rw-p 00000000 00:00 0
7712b4cdde3c000-2b4cdde40000 rw-p 00000000 00:00 0
7722b4cdde48000-2b4cdde4c000 rw-p 00000000 00:00 0
7732b4cdde6c000-2b4cdde70000 rw-p 00000000 00:00 0
7742b4cddea0000-2b4cddea4000 rw-p 00000000 00:00 0
7752b4cddeac000-2b4cddeb0000 rw-p 00000000 00:00 0
7762b4cdded0000-2b4cdded4000 rw-p 00000000 00:00 0
7772b4cddee4000-2b4cddee8000 rw-p 00000000 00:00 0
7782b4cddf60000-2b4cddf64000 rw-p 00000000 00:00 0
7792b4cddf7c000-2b4cddf80000 rw-p 00000000 00:00 0
7802b4cde050000-2b4cde054000 rw-p 00000000 00:00 0
7812b4cde05c000-2b4cde060000 rw-p 00000000 00:00 0
7822b4cde0c8000-2b4cde0cc000 rw-p 00000000 00:00 0
7832b4cde13c000-2b4cde144000 rw-p 00000000 00:00 0
7842b4cde17c000-2b4cde180000 rw-p 00000000 00:00 0
7852b4cde1b0000-2b4cde1b4000 rw-p 00000000 00:00 0
7862b4cde1fc000-2b4cde200000 rw-p 00000000 00:00 0
7872b4cde250000-2b4cde254000 rw-p 00000000 00:00 0
7882b4cde27c000-2b4cde280000 rw-p 00000000 00:00 0
7892b4cde290000-2b4cde294000 rw-p 00000000 00:00 0
7902b4cde2ac000-2b4cde2b0000 rw-p 00000000 00:00 0
7912b4cde2b8000-2b4cde2bc000 rw-p 00000000 00:00 0
7922b4cde2fc000-2b4cde300000 rw-p 00000000 00:00 0
7932b4cde340000-2b4cde348000 rw-p 00000000 00:00 0
7942b4cde390000-2b4cde394000 rw-p 00000000 00:00 0
7952b4cde3c4000-2b4cde3c8000 rw-p 00000000 00:00 0
7962b4cde400000-2b4cde401000 ---p 00000000 00:00 0
7972b4cde401000-2b4cde402000 rw-p 00000000 00:00 0
7982b4cde402000-2b4cde40a000 ---p 00000000 00:00 0
7992b4cde40a000-2b4cde601000 rw-p 00000000 00:00 0
8002b4cde6f4000-2b4cde6fc000 rw-p 00000000 00:00 0
8012b4cde724000-2b4cde728000 rw-p 00000000 00:00 0
8022b4cde748000-2b4cde750000 rw-p 00000000 00:00 0
8032b4cde758000-2b4cde75c000 rw-p 00000000 00:00 0
8042b4cde7dc000-2b4cde7e0000 rw-p 00000000 00:00 0
8052b4cde7e8000-2b4cde7ec000 rw-p 00000000 00:00 0
8062b4cde820000-2b4cde824000 rw-p 00000000 00:00 0
8072b4cde87c000-2b4cde880000 rw-p 00000000 00:00 0
8082b4cde900000-2b4cde901000 ---p 00000000 00:00 0
8092b4cde901000-2b4cde902000 rw-p 00000000 00:00 0
8102b4cde902000-2b4cde90a000 ---p 00000000 00:00 0
8112b4cde90a000-2b4cdeb01000 rw-p 00000000 00:00 0
8122b4cdeb64000-2b4cdeb68000 rw-p 00000000 00:00 0
8132b4cdeb80000-2b4cdeb88000 rw-p 00000000 00:00 0
8142b4cdebb0000-2b4cdebb8000 rw-p 00000000 00:00 0
8152b4cdec00000-2b4cdec04000 rw-p 00000000 00:00 0
8162b4cdec74000-2b4cdec84000 rw-p 00000000 00:00 0
8172b4cdecb4000-2b4cdecd8000 rw-p 00000000 00:00 0
8182b4cdece0000-2b4cdece8000 rw-p 00000000 00:00 0
8192b4cdecf8000-2b4cdecfc000 rw-p 00000000 00:00 0
8202b4cded74000-2b4cded84000 rw-p 00000000 00:00 0
8212b4cdee28000-2b4cdee2c000 rw-p 00000000 00:00 0
8222b4cdee7c000-2b4cdee80000 rw-p 00000000 00:00 0
8232b4cdf118000-2b4cdf11c000 rw-p 00000000 00:00 0
8242b4cdf154000-2b4cdf158000 rw-p 00000000 00:00 0
8252b4cdf400000-2b4cdf408000 rw-p 00000000 00:00 0
8262b4cdf418000-2b4cdf41c000 rw-p 00000000 00:00 0
8272b4cdf42c000-2b4cdf430000 rw-p 00000000 00:00 0
8282b4cdf448000-2b4cdf44c000 rw-p 00000000 00:00 0
8292b4cdf454000-2b4cdf458000 rw-p 00000000 00:00 0
8302b4cdf900000-2b4cdfb00000 rw-p 00000000 00:00 0
8312b4cdfd00000-2b4ce0021000 rw-p 00000000 00:00 0
8322b4ce0021000-2b4ce4000000 ---p 00000000 00:00 0
83355d6f16c2000-55d6f1aef000 r-xp 00000000 fe:03 961825 /usr/bin/mono-sgen
83455d6f1cef000-55d6f1cf6000 r--p 0042d000 fe:03 961825 /usr/bin/mono-sgen
83555d6f1cf6000-55d6f1cfa000 rw-p 00434000 fe:03 961825 /usr/bin/mono-sgen
83655d6f1cfa000-55d6f1d11000 rw-p 00000000 00:00 0
83755d6f2460000-55d6f24c3000 rw-p 00000000 00:00 0 [heap]
83855d6f24c4000-55d6f24cc000 ---p 00000000 00:00 0
8397fff44307000-7fff44328000 rw-p 00000000 00:00 0 [stack]
8407fff443da000-7fff443dc000 r--p 00000000 00:00 0 [vvar]
8417fff443dc000-7fff443de000 r-xp 00000000 00:00 0 [vdso]
842ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
84307:10:57 0 ✓ zhuyifei1999@tools-sgeexec-0940: ~$ sudo kill -STOP 21380

$ cat tmp | awk '{ split($1, a, "-"); printf "%s %08x %s %s\n", a[1], strtonum("0x"a[2]) - strtonum("0x"a[1]), $2, $6 }' | sort -k 2 | less -N:

811 2b4c80800000 003cb000 r--p /usr/lib/mono/4.5/mscorlib.dll
812 55d6f16c2000 0042d000 r-xp /usr/bin/mono-sgen
813 2b4c80bcb000 00568000 r-xp /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
814 2b4c7f400000 01400000 rw-p 
815 2b4ca0126000 03eda000 ---p 
816 2b4c9411a000 03ee6000 ---p 
817 2b4c9010b000 03ef5000 ---p 
818 2b4ca8107000 03ef9000 ---p 
819 2b4cb0077000 03f89000 ---p 
820 2b4cac034000 03fcc000 ---p 
821 2b4c9c024000 03fdc000 ---p 
822 2b4c84021000 03fdf000 ---p 
823 2b4c88021000 03fdf000 ---p 
824 2b4c8c021000 03fdf000 ---p 
825 2b4c98021000 03fdf000 ---p 
826 2b4cb4021000 03fdf000 ---p 
827 2b4cb8021000 03fdf000 ---p 
828 2b4cbc021000 03fdf000 ---p 
829 2b4cc0021000 03fdf000 ---p 
830 2b4cc8021000 03fdf000 ---p 
831 2b4ccc021000 03fdf000 ---p 
832 2b4cd0021000 03fdf000 ---p 
833 2b4cd4021000 03fdf000 ---p 
834 2b4cd8021000 03fdf000 ---p 
835 2b4ce0021000 03fdf000 ---p

Lots and lots of almost empty glibc heaps. 21 heaps accounting for 1402015744 bytes = 1.3GiB of basically unused address space. Also notice how RSS is just 61520 KiB?

Vort added a comment.Nov 2 2020, 7:30 AM

How many threads are running? There should be not many of them. 5 or so (1 main + 4 workers). Maybe + some threads used by runtime library.

This was when I generated the core:

07:11:39 0 ✓ zhuyifei1999@tools-sgeexec-0940: ~$ sudo gdb -p 21380 -batch -ex 'generate-core-file ~tools.wikitasks/T266377-2.core'
[New LWP 21382]
[New LWP 21385]
[New LWP 21546]
[New LWP 21550]
[New LWP 21551]
[New LWP 21554]
[New LWP 21555]
[New LWP 21556]
[New LWP 21561]
[New LWP 21562]
[New LWP 21563]
[New LWP 21702]
[New LWP 21875]
[New LWP 21968]
[New LWP 22015]
[New LWP 22085]
[New LWP 22111]
[New LWP 22112]
[New LWP 22113]
[New LWP 22122]
[New LWP 24038]
[New LWP 24039]
[New LWP 24085]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00002b4c7e9c217f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
warning: target file /proc/21380/cmdline contained unexpected null characters
Saved corefile /data/project/wikitasks/T266377-2.core

Something like 23 threads.

I accidentally killed the process while gdb-ing, but now gdb-ing in the core dump:

(gdb) info threads
  Id   Target Id         Frame 
* 1    Thread 0x2b4c7e0c0680 (LWP 21380) 0x00002b4c7e9c217f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  2    Thread 0x2b4c7f3e8700 (LWP 21382) 0x00002b4c7e9c217f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  3    Thread 0x2b4c8173b700 (LWP 21385) 0x00002b4c7e9c4556 in do_futex_wait.constprop () from /lib/x86_64-linux-gnu/libpthread.so.0
  4    Thread 0x2b4c8244c700 (LWP 21546) 0x00002b4c7e9c2528 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  5    Thread 0x2b4c8294e700 (LWP 21550) 0x000055d6f177f5a0 in ?? ()
  6    Thread 0x2b4c82b4f700 (LWP 21551) 0x00002b4c7e9c4720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  7    Thread 0x2b4c8395f700 (LWP 21554) 0x00002b4c7eec88bd in poll () from /lib/x86_64-linux-gnu/libc.so.6
  8    Thread 0x2b4c83b60700 (LWP 21555) 0x00002b4c7e9c4720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  9    Thread 0x2b4c83d61700 (LWP 21556) 0x00002b4c7e9c4720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  10   Thread 0x2b4c83f62700 (LWP 21561) 0x00002b4c7e9c4720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  11   Thread 0x2b4ca457c700 (LWP 21562) 0x00002b4c7e9c4720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  12   Thread 0x2b4ca477d700 (LWP 21563) 0x00002b4c7e9c4720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  13   Thread 0x2b4ca7380700 (LWP 21702) 0x00002b4c7e9c4720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  14   Thread 0x2b4ca7581700 (LWP 21875) 0x00002b4c7e9c4720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  15   Thread 0x2b4ca7100700 (LWP 21968) 0x00002b4c7e9c4720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  16   Thread 0x2b4ca7d00700 (LWP 22015) 0x00002b4c7e9c4720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  17   Thread 0x2b4ca7f84700 (LWP 22085) 0x00002b4c7e9c4720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  18   Thread 0x2b4cc4700700 (LWP 22111) 0x00002b4c7e9c4720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  19   Thread 0x2b4cc4901700 (LWP 22112) 0x00002b4c7e9c4720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  20   Thread 0x2b4cc4b02700 (LWP 22113) 0x00002b4c7e9c4720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  21   Thread 0x2b4cc4d03700 (LWP 22122) 0x00002b4c7e9c4720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  22   Thread 0x2b4cde600700 (LWP 24038) 0x00002b4c7e9c4720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  23   Thread 0x2b4cdeb00700 (LWP 24039) 0x00002b4c7e9c4720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  24   Thread 0x2b4c8274d700 (LWP 24085) 0x00002b4c7e9c2528 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0

Hmm. I wonder if I can get the stack trace of the mono frames

Vort added a comment.Nov 2 2020, 7:37 AM

The number of operations that can be queued to the thread pool is limited only by available memory. However, the thread pool limits the number of threads that can be active in the process simultaneously. If all thread pool threads are busy, additional work items are queued until threads to execute them become available. Beginning with the .NET Framework 4, the default size of the thread pool for a process depends on several factors, such as the size of the virtual address space. A process can call the ThreadPool.GetMaxThreads method to determine the number of threads.

You can control the maximum number of threads by using the ThreadPool.GetMaxThreads and ThreadPool.SetMaxThreads methods.

https://docs.microsoft.com/en-us/dotnet/standard/threading/the-managed-thread-pool

Looks like this is what I should try (ThreadPool.SetMaxThreads).

What does ThreadPool.GetMaxThreads return?

https://github.com/mono/mono/blob/ab893a369a3b24ee637db711e1d7ffe96ebe7f4a/mono/metadata/threadpool-worker-default.c#L280 :

	if (!(threads_per_cpu_env = g_getenv ("MONO_THREADS_PER_CPU")))
		threads_per_cpu = 1;
	else
		threads_per_cpu = CLAMP (atoi (threads_per_cpu_env), 1, 50);

	threads_count = mono_cpu_count () * threads_per_cpu;

	worker.limit_worker_min = threads_count;

#if defined (HOST_ANDROID) || defined (HOST_IOS)
	worker.limit_worker_max = CLAMP (threads_count * 100, MIN (threads_count, 200), MAX (threads_count, 200));
#else
	worker.limit_worker_max = threads_count * 100;
#endif

I'm guessing it'll return 400?

Vort added a comment.EditedNov 2 2020, 7:47 AM

What does ThreadPool.GetMaxThreads return?

On my PC (8 GB of RAM, 4 core CPU, original .NET, Win 7 x64):
workerThreads: 2047, completionPortThreads: 1000

On Toolforge grid:
workerThreads: 400, completionPortThreads: 200

Yeah, if so many threads start they get different glibc heaps and it would not be efficient with the address space....

Vort added a comment.EditedNov 2 2020, 7:51 AM

So it not becomes a problem on other systems because they do not enforce limits on virtual memory?

Correct. Grid stubbornly enforce virtual memory limits because enforcing RSS isn't as trivial. The former is just an rlimit whereas the latter needs cgroups. I really hate how it works but it is how it is. :/

(btw, k8s's memory limits is on RSS, using cgroups, IIRC)

Vort added a comment.Nov 2 2020, 8:12 AM

I have removed OOM debugging code and added call to this function:

static void LowerThreadPoolLimits()
{
    int workerThreads;
    int completionPortThreads;
    ThreadPool.GetMinThreads(out workerThreads, out completionPortThreads);
    ThreadPool.SetMaxThreads(workerThreads * 2, completionPortThreads * 2);
}

Will see if program will work correctly now.

ThreadPool.GetMinThreads(out workerThreads, out completionPortThreads);
ThreadPool.SetMaxThreads(workerThreads * 2, completionPortThreads * 2);

Am I understanding correctly that this doubles the limit? How would that fix the problem? You want to decrease not increase the limit right?

Vort added a comment.Nov 2 2020, 8:16 AM

Am I understanding correctly that this doubles the limit?

It doubles minimum limit.
Which is 4 / 4. And now will be 8 / 8.

Vort added a comment.Nov 3 2020, 7:50 AM

Bug do not want to go away.
With pool size = 2 * minimum, another exception happened:

Unhandled Exception:
Nested exception detected.
Original Exception: at System.Threading.Tasks.Task.ThrowIfExceptional (bool) [0x00011] in <71d8ad678db34313b7f718a414dfcb25>:0
at System.Threading.Tasks.Task`1<string>.GetResultCore (bool) [0x0002b] in <71d8ad678db34313b7f718a414dfcb25>:0
at System.Threading.Tasks.Task`1<string>.get_Result () [0x0000f] in <71d8ad678db34313b7f718a414dfcb25>:0
at WikiTasks.MwApi.PostRequest (object[]) [0x00008] in <6ae7d1ae85b14f5b8c51c0c443295fee>:0
at WikiTasks.Program.GetCyrLat (int,System.Collections.Generic.List`1<string>) [0x0000b] in <6ae7d1ae85b14f5b8c51c0c443295fee>:0
at WikiTasks.Program..ctor () [0x0008f] in <6ae7d1ae85b14f5b8c51c0c443295fee>:0
at WikiTasks.Program.Main (string[]) [0x00001] in <6ae7d1ae85b14f5b8c51c0c443295fee>:0

Nested exception:at (wrapper managed-to-native) string.FastAllocateString (int) [0x00024] in <71d8ad678db34313b7f718a414dfcb25>:0
at System.Text.StringBuilder.ToString () [0x00014] in <71d8ad678db34313b7f718a414dfcb25>:0
at System.Text.StringBuilderCache.GetStringAndRelease (System.Text.StringBuilder) [0x00000] in <71d8ad678db34313b7f718a414dfcb25>:0
at string.FormatHelper (System.IFormatProvider,string,System.ParamsArray) [0x0002b] in <71d8ad678db34313b7f718a414dfcb25>:0
at string.Format (System.IFormatProvider,string,object[]) [0x00020] in <71d8ad678db34313b7f718a414dfcb25>:0
at System.AggregateException.ToString () [0x00058] in <71d8ad678db34313b7f718a414dfcb25>:0


[ERROR] FATAL UNHANDLED EXCEPTION: Nested exception detected.
Original Exception: at System.Threading.Tasks.Task.ThrowIfExceptional (bool) [0x00011] in <71d8ad678db34313b7f718a414dfcb25>:0
at System.Threading.Tasks.Task`1<string>.GetResultCore (bool) [0x0002b] in <71d8ad678db34313b7f718a414dfcb25>:0
at System.Threading.Tasks.Task`1<string>.get_Result () [0x0000f] in <71d8ad678db34313b7f718a414dfcb25>:0
at WikiTasks.MwApi.PostRequest (object[]) [0x00008] in <6ae7d1ae85b14f5b8c51c0c443295fee>:0
at WikiTasks.Program.GetCyrLat (int,System.Collections.Generic.List`1<string>) [0x0000b] in <6ae7d1ae85b14f5b8c51c0c443295fee>:0
at WikiTasks.Program..ctor () [0x0008f] in <6ae7d1ae85b14f5b8c51c0c443295fee>:0
at WikiTasks.Program.Main (string[]) [0x00001] in <6ae7d1ae85b14f5b8c51c0c443295fee>:0

Nested exception:at (wrapper managed-to-native) string.FastAllocateString (int) [0x00024] in <71d8ad678db34313b7f718a414dfcb25>:0
at string.ConcatArray (string[],int) [0x00000] in <71d8ad678db34313b7f718a414dfcb25>:0
at string.Concat (string[]) [0x0004e] in <71d8ad678db34313b7f718a414dfcb25>:0
at System.Exception.ToString (bool,bool) [0x0007a] in <71d8ad678db34313b7f718a414dfcb25>:0
at System.Exception.ToString () [0x00000] in <71d8ad678db34313b7f718a414dfcb25>:0
at System.AggregateException.ToString () [0x00000] in <71d8ad678db34313b7f718a414dfcb25>:0
Vort added a comment.EditedNov 3 2020, 7:53 AM

Now I change max pool size to be equal to min pool size.
Can't go lower than that with Mono (Windows .NET allows to set 1 / 1).
And add exception handler to all program code:

ThreadPool.GetMinThreads(out workerThreads, out completionPortThreads);
ThreadPool.SetMaxThreads(workerThreads, completionPortThreads);
static void Main(string[] args)
{
    LowerThreadPoolLimits();
    try
    {
        new Program();
    }
    catch
    {
        Console.Write($"[ExceptionCaught,ThreadCount:{Process.GetCurrentProcess().Threads.Count},Pausing]");
        Thread.Sleep(Timeout.Infinite);
    }
}

With such settings, usual value for .Threads.Count is 11.
Let's see if it will go to 23 as before when (or if) glitch happens.

FWIW, (ASLR offsets redacted since the process is kept running)

07:52:24 0 ✓ zhuyifei1999@tools-sgeexec-0952: ~$ ps uf -u tools.wikitasks
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
tools.w+ 23874  0.0  0.0  11144  2716 ?        Ss   07:48   0:00 /bin/bash /mnt/nfs/labstore-secondary-tools-project/wikitasks/wrap.sh wp_cyrlat
tools.w+ 23880 25.2  0.8 1136144 69404 ?       Sl   07:48   1:11  \_ mono WikiTasks.exe
07:53:43 0 ✓ zhuyifei1999@tools-sgeexec-0952: ~$ sudo gdb -batch -ex 'info thread' -p 23880
[New LWP 23881]
[New LWP 23882]
[New LWP 23883]
[New LWP 23884]
[New LWP 23885]
[New LWP 23886]
[New LWP 23887]
[New LWP 23888]
[New LWP 23889]
[New LWP 23890]
[New LWP 23891]
[New LWP 23894]
[New LWP 23898]
[New LWP 23899]
[New LWP 23972]
[New LWP 23973]
[New LWP 24176]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00002[...]44c17f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  Id   Target Id         Frame 
* 1    Thread 0x2[...]b4a680 (LWP 23880) "mono" 0x00002[...]44c17f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  2    Thread 0x2[...]e72700 (LWP 23881) "SGen worker" 0x00002[...]44c17f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  3    Thread 0x2[...]25a700 (LWP 23882) "Finalizer" 0x00002[...]44e556 in do_futex_wait.constprop () from /lib/x86_64-linux-gnu/libpthread.so.0
  4    Thread 0x2[...]f00700 (LWP 23883) "Timer-Scheduler" 0x00002[...]44c528 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  5    Thread 0x2[...]201700 (LWP 23884) "mono" 0x00002[...]44c528 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  6    Thread 0x2[...]402700 (LWP 23885) "Thread Pool Wor" 0x00002[...]44e720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  7    Thread 0x2[...]603700 (LWP 23886) "Thread Pool Wor" 0x00002[...]44e720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  8    Thread 0x2[...]3f8700 (LWP 23887) "Thread Pool I/O" 0x00002[...]9528bd in poll () from /lib/x86_64-linux-gnu/libc.so.6
  9    Thread 0x2[...]5f9700 (LWP 23888) "Thread Pool Wor" 0x00002[...]44e720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  10   Thread 0x2[...]b76700 (LWP 23889) "Thread Pool Wor" 0x00002[...]44e720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  11   Thread 0x2[...]d77700 (LWP 23890) "Thread Pool Wor" 0x00002[...]44e720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  12   Thread 0x2[...]f78700 (LWP 23891) "Thread Pool Wor" 0x00002[...]44e720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  13   Thread 0x2[...]8da700 (LWP 23894) "Thread Pool Wor" 0x00002[...]44e720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  14   Thread 0x2[...]500700 (LWP 23898) "Thread Pool Wor" 0x00002[...]44e720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  15   Thread 0x2[...]800700 (LWP 23899) "Thread Pool Wor" 0x00002[...]44e720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  16   Thread 0x2[...]700700 (LWP 23972) "Thread Pool Wor" 0x00002[...]44e720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  17   Thread 0x2[...]e00700 (LWP 23973) "Thread Pool Wor" 0x00002[...]44e720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  18   Thread 0x2[...]200700 (LWP 24176) "Thread Pool Wor" 0x00002[...]44e720 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
Vort added a comment.Nov 3 2020, 8:04 AM

I suspect that ThreadPool.SetMaxThreads do not what I think.
Either I do not understand correctly its functions
Or it have bugs.

https://docs.microsoft.com/en-us/dotnet/api/system.threading.threadpool.setmaxthreads :

Sets the number of requests to the thread pool that can be active concurrently. All requests above that number remain queued until thread pool threads become available.

For sure, 12 threads in your output are larger than 4 + 4 limits, which was requested.

Do you have a minimal code that starts a thread pool and pause infinitely? I can do a debug build of mono on my laptop and see what is going on.

Vort added a comment.Nov 3 2020, 8:26 AM

First of all I want to point at one interesting thing:
Look at thread IDs: 23881, 23882, ..., 23891: 11 of them was created sequentially.
Next 3 was created slightly later: 23894, 23898, 23899, next 2 after more delay: 23972, 23973.
And the last id, 24176, differs from previous by 203.

So additional threads are created only at specific conditions.
Maybe because of bug, who knows.

Vort added a comment.Nov 3 2020, 8:29 AM

Do you have a minimal code that starts a thread pool and pause infinitely? I can do a debug build of mono on my laptop and see what is going on.

Looks like it is enough just to make web requests to fill pool with threads.
Maybe this code will help:

mcs ToolforgeDebug.cs

using System;
using System.Threading;
using System.Diagnostics;
using System.Net;

namespace ToolforgeDebug
{
    class Program
    {
        static void Main(string[] args)
        {
            int workerThreads;
            int completionPortThreads;
            ThreadPool.GetMinThreads(out workerThreads, out completionPortThreads);
            ThreadPool.SetMaxThreads(workerThreads, completionPortThreads);

            var wc = new WebClient();
            for (int i = 0; i < 4; i++)
            {
                wc.DownloadString("https://worldtimeapi.org/api/timezone/Europe/London.txt");
                Console.Write('.');
                Thread.Sleep(500);
            }

            Console.Write($"[ThreadCount:{Process.GetCurrentProcess().Threads.Count}]");
            Thread.Sleep(Timeout.Infinite);
        }
    }
}

Example result:

mono ToolforgeDebug.exe
....[ThreadCount:11]^C

My local laptop is a 4-core 8-thread machine, but I fail to get that many threads. I tried ti trace where the threads starts:

zhuyifei1999@zhuyifei1999-ThinkPad-T480 ~/T266377 $ cat ToolforgeDebug.cs 
using System;
using System.Threading;
using System.Diagnostics;
using System.Net;

namespace ToolforgeDebug
{
    class Program
    {
        static void Main(string[] args)
        {
            int workerThreads;
            int completionPortThreads;
            ThreadPool.GetMinThreads(out workerThreads, out completionPortThreads);
            ThreadPool.SetMaxThreads(workerThreads, completionPortThreads);

            Console.WriteLine("{0}, {1}", workerThreads, completionPortThreads);

            var wc = new WebClient();
            for (int i = 0; i < workerThreads; i++)
            {
                wc.DownloadString("https://worldtimeapi.org/api/timezone/Europe/London.txt");
                Console.Write('.');
                Thread.Sleep(500);
            }

            Console.Write($"[ThreadCount:{Process.GetCurrentProcess().Threads.Count}]");
            Thread.Sleep(Timeout.Infinite);
        }
    }
}
zhuyifei1999@zhuyifei1999-ThinkPad-T480 ~/T266377 $ cat debug.gdb 
define mono_stack
 set $mono_thread = mono_thread_internal_current ()
 if ($mono_thread == 0x00)
   printf "No mono thread associated with this thread\n"
 else
   set $ucp = malloc (sizeof (ucontext_t))
   call (void) getcontext ($ucp)
   call (void) mono_print_thread_dump ($ucp)
   call (void) free ($ucp)
 end
end

break clone
commands
mono_stack
# bt
c
end

gef config context.enable false

# set unwindonsignal on

r

1zhuyifei1999@zhuyifei1999-ThinkPad-T480 ~/T266377 $ gdb -batch -ex 'source debug.gdb' --args mono ToolforgeDebug.exe
2GEF for linux ready, type `gef' to start, `gef config' to configure
375 commands loaded for GDB 10.1 using Python engine 3.7
4[*] 5 commands could not be loaded, run `gef missing` to know why.
5[+] Configuration from '/home/zhuyifei1999/.gef.rc' restored
6[+] 10 extra commands added from '/home/zhuyifei1999/gdb/gef-extras/scripts'
7warning: File "/usr/bin/mono-sgen-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/src/linux/".
8To enable execution of this file add
9 add-auto-load-safe-path /usr/bin/mono-sgen-gdb.py
10line to your configuration file "/home/zhuyifei1999/.gdbinit".
11To completely disable this security protection add
12 set auto-load safe-path /
13line to your configuration file "/home/zhuyifei1999/.gdbinit".
14For more information about this security protection see the
15"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
16 info "(gdb)Auto-loading safe path"
17Function "clone" not defined.
18Breakpoint 1 (clone) pending.
19[Thread debugging using libthread_db enabled]
20Using host libthread_db library "/lib64/libthread_db.so.1".
21
22Breakpoint 1, clone () at /usr/src/debug/sys-libs/glibc-2.32-r2/glibc-2.32/sysdeps/unix/sysv/linux/x86_64/clone.S:52
2352 movq $-EINVAL,%rax
24No mono thread associated with this thread
25
26Thread 1 "mono" hit Breakpoint 1, clone () at /usr/src/debug/sys-libs/glibc-2.32-r2/glibc-2.32/sysdeps/unix/sysv/linux/x86_64/clone.S:52
2752 movq $-EINVAL,%rax
28
29"<unnamed thread>" tid=0x7ffff7c34740 this=0x7ffff6f7c130 , thread handle : 0x555555a1e0c0, state : not waiting
30
318, 8
32
33Thread 1 "mono" hit Breakpoint 1, clone () at /usr/src/debug/sys-libs/glibc-2.32-r2/glibc-2.32/sysdeps/unix/sysv/linux/x86_64/clone.S:52
3452 movq $-EINVAL,%rax
35
36"<unnamed thread>" tid=0x7ffff7c34740 this=0x7ffff6f7c130 , thread handle : 0x555555a1e0c0, state : not waiting
37 at <unknown> <0xffffffff>
38 at (wrapper managed-to-native) System.Threading.ThreadPool.RequestWorkerThread () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
39 at System.Threading.ThreadPoolWorkQueue.EnsureThreadRequested () [0x0001f] in <12384316d9c54a0b99f94c42f2c92f80>:0
40 at System.Threading.ThreadPoolWorkQueue.Enqueue (System.Threading.IThreadPoolWorkItem,bool) [0x00071] in <12384316d9c54a0b99f94c42f2c92f80>:0
41 at System.Threading.ThreadPool.UnsafeQueueCustomWorkItem (System.Threading.IThreadPoolWorkItem,bool) [0x00007] in <12384316d9c54a0b99f94c42f2c92f80>:0
42 at System.Threading.Tasks.ThreadPoolTaskScheduler.QueueTask (System.Threading.Tasks.Task) [0x00072] in <12384316d9c54a0b99f94c42f2c92f80>:0
43 at System.Threading.Tasks.Task.ScheduleAndStart (bool) [0x00063] in <12384316d9c54a0b99f94c42f2c92f80>:0
44 at System.Threading.Tasks.Task`1<TResult_REF>.StartNew (System.Threading.Tasks.Task,System.Func`1<TResult_REF>,System.Threading.CancellationToken,System.Threading.Tasks.TaskCreationOptions,System.Threading.Tasks.InternalTaskOptions,System.Threading.Tasks.TaskScheduler) [0x00030] in <12384316d9c54a0b99f94c42f2c92f80>:0
45 at System.Threading.Tasks.TaskFactory.StartNew<TResult_REF> (System.Func`1<TResult_REF>,System.Threading.CancellationToken,System.Threading.Tasks.TaskCreationOptions,System.Threading.Tasks.TaskScheduler) [0x00006] in <12384316d9c54a0b99f94c42f2c92f80>:0
46 at System.Threading.Tasks.Task.Run<TResult_REF> (System.Func`1<System.Threading.Tasks.Task`1<TResult_REF>>,System.Threading.CancellationToken) [0x0002b] in <12384316d9c54a0b99f94c42f2c92f80>:0
47 at System.Threading.Tasks.Task.Run<TResult_REF> (System.Func`1<System.Threading.Tasks.Task`1<TResult_REF>>) [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
48 at System.Net.WebRequest.GetResponseAsync () [0x00059] in <35f76809d1304f7b9923e43abe01f2f7>:0
49 at System.Net.HttpWebRequest.GetResponse () [0x00000] in <35f76809d1304f7b9923e43abe01f2f7>:0
50 at System.Net.WebClient.GetWebResponse (System.Net.WebRequest) [0x00000] in <35f76809d1304f7b9923e43abe01f2f7>:0
51 at System.Net.WebClient.DownloadBits (System.Net.WebRequest,System.IO.Stream) [0x00000] in <35f76809d1304f7b9923e43abe01f2f7>:0
52 at System.Net.WebClient.DownloadDataInternal (System.Uri,System.Net.WebRequest&) [0x00023] in <35f76809d1304f7b9923e43abe01f2f7>:0
53 at System.Net.WebClient.DownloadString (System.Uri) [0x00011] in <35f76809d1304f7b9923e43abe01f2f7>:0
54 at System.Net.WebClient.DownloadString (string) [0x00008] in <35f76809d1304f7b9923e43abe01f2f7>:0
55 at (wrapper remoting-invoke-with-check) System.Net.WebClient.DownloadString (string) [0x00032] in <35f76809d1304f7b9923e43abe01f2f7>:0
56 at ToolforgeDebug.Program.Main (string[]) [0x00034] in <84b01e721dd4492facb611d0e0fa6106>:0
57 at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object (object,intptr,intptr,intptr) [0x0002a] in <84b01e721dd4492facb611d0e0fa6106>:0
58
59
60Thread 1 "mono" hit Breakpoint 1, clone () at /usr/src/debug/sys-libs/glibc-2.32-r2/glibc-2.32/sysdeps/unix/sysv/linux/x86_64/clone.S:52
6152 movq $-EINVAL,%rax
62
63"<unnamed thread>" tid=0x7ffff7c34740 this=0x7ffff6f7c130 , thread handle : 0x555555a1e0c0, state : not waiting
64 at <unknown> <0xffffffff>
65 at (wrapper managed-to-native) System.Threading.ThreadPool.RequestWorkerThread () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
66 at System.Threading.ThreadPoolWorkQueue.EnsureThreadRequested () [0x0001f] in <12384316d9c54a0b99f94c42f2c92f80>:0
67 at System.Threading.ThreadPoolWorkQueue.Enqueue (System.Threading.IThreadPoolWorkItem,bool) [0x00071] in <12384316d9c54a0b99f94c42f2c92f80>:0
68 at System.Threading.ThreadPool.UnsafeQueueCustomWorkItem (System.Threading.IThreadPoolWorkItem,bool) [0x00007] in <12384316d9c54a0b99f94c42f2c92f80>:0
69 at System.Threading.Tasks.ThreadPoolTaskScheduler.QueueTask (System.Threading.Tasks.Task) [0x00072] in <12384316d9c54a0b99f94c42f2c92f80>:0
70 at System.Threading.Tasks.Task.ScheduleAndStart (bool) [0x00063] in <12384316d9c54a0b99f94c42f2c92f80>:0
71 at System.Threading.Tasks.Task`1<TResult_REF>.StartNew (System.Threading.Tasks.Task,System.Func`1<TResult_REF>,System.Threading.CancellationToken,System.Threading.Tasks.TaskCreationOptions,System.Threading.Tasks.InternalTaskOptions,System.Threading.Tasks.TaskScheduler) [0x00030] in <12384316d9c54a0b99f94c42f2c92f80>:0
72 at System.Threading.Tasks.TaskFactory.StartNew<TResult_REF> (System.Func`1<TResult_REF>,System.Threading.CancellationToken,System.Threading.Tasks.TaskCreationOptions,System.Threading.Tasks.TaskScheduler) [0x00006] in <12384316d9c54a0b99f94c42f2c92f80>:0
73 at System.Threading.Tasks.Task.Run<TResult_REF> (System.Func`1<System.Threading.Tasks.Task`1<TResult_REF>>,System.Threading.CancellationToken) [0x0002b] in <12384316d9c54a0b99f94c42f2c92f80>:0
74 at System.Threading.Tasks.Task.Run<TResult_REF> (System.Func`1<System.Threading.Tasks.Task`1<TResult_REF>>) [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
75 at System.Net.WebRequest.GetResponseAsync () [0x00059] in <35f76809d1304f7b9923e43abe01f2f7>:0
76 at System.Net.HttpWebRequest.GetResponse () [0x00000] in <35f76809d1304f7b9923e43abe01f2f7>:0
77 at System.Net.WebClient.GetWebResponse (System.Net.WebRequest) [0x00000] in <35f76809d1304f7b9923e43abe01f2f7>:0
78 at System.Net.WebClient.DownloadBits (System.Net.WebRequest,System.IO.Stream) [0x00000] in <35f76809d1304f7b9923e43abe01f2f7>:0
79 at System.Net.WebClient.DownloadDataInternal (System.Uri,System.Net.WebRequest&) [0x00023] in <35f76809d1304f7b9923e43abe01f2f7>:0
80 at System.Net.WebClient.DownloadString (System.Uri) [0x00011] in <35f76809d1304f7b9923e43abe01f2f7>:0
81 at System.Net.WebClient.DownloadString (string) [0x00008] in <35f76809d1304f7b9923e43abe01f2f7>:0
82 at (wrapper remoting-invoke-with-check) System.Net.WebClient.DownloadString (string) [0x00032] in <35f76809d1304f7b9923e43abe01f2f7>:0
83 at ToolforgeDebug.Program.Main (string[]) [0x00034] in <84b01e721dd4492facb611d0e0fa6106>:0
84 at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object (object,intptr,intptr,intptr) [0x0002a] in <84b01e721dd4492facb611d0e0fa6106>:0
85
86[Switching to Thread 0x7fffeffff640 (LWP 254284)]
87
88Thread 5 "Thread Pool Wor" hit Breakpoint 1, clone () at /usr/src/debug/sys-libs/glibc-2.32-r2/glibc-2.32/sysdeps/unix/sysv/linux/x86_64/clone.S:52
8952 movq $-EINVAL,%rax
90
91"Thread Pool Worker" tid=0x7fffeffff640 this=0x7ffff6f7c508 , thread handle : 0x7fffe4001200, state : not waiting
92 at <unknown> <0xffffffff>
93 at (wrapper managed-to-native) System.Threading.ThreadPool.RequestWorkerThread () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
94 at System.Threading.ThreadPoolWorkQueue.EnsureThreadRequested () [0x0001f] in <12384316d9c54a0b99f94c42f2c92f80>:0
95 at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x0003a] in <12384316d9c54a0b99f94c42f2c92f80>:0
96 at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
97 at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <12384316d9c54a0b99f94c42f2c92f80>:0
98
99[Switching to Thread 0x7fffefdfe640 (LWP 254285)]
100
101Thread 6 "Thread Pool Wor" hit Breakpoint 1, clone () at /usr/src/debug/sys-libs/glibc-2.32-r2/glibc-2.32/sysdeps/unix/sysv/linux/x86_64/clone.S:52
10252 movq $-EINVAL,%rax
103
104"Thread Pool Worker" tid=0x7fffefdfe640 this=0x7ffff6f7c650 , thread handle : 0x7fffdc001200, state : not waiting
105 at <unknown> <0xffffffff>
106 at (wrapper managed-to-native) System.Threading.ThreadPool.RequestWorkerThread () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
107 at System.Threading.ThreadPoolWorkQueue.EnsureThreadRequested () [0x0001f] in <12384316d9c54a0b99f94c42f2c92f80>:0
108 at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x0003a] in <12384316d9c54a0b99f94c42f2c92f80>:0
109 at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
110 at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <12384316d9c54a0b99f94c42f2c92f80>:0
111
112[Switching to Thread 0x7fffeffff640 (LWP 254284)]
113
114Thread 5 "Thread Pool Wor" hit Breakpoint 1, clone () at /usr/src/debug/sys-libs/glibc-2.32-r2/glibc-2.32/sysdeps/unix/sysv/linux/x86_64/clone.S:52
11552 movq $-EINVAL,%rax
116
117"Thread Pool Worker" tid=0x7fffeffff640 this=0x7ffff6f7c508 , thread handle : 0x7fffe4001200, state : not waiting
118 at <unknown> <0xffffffff>
119 at (wrapper managed-to-native) System.Threading.ThreadPool.RequestWorkerThread () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
120 at System.Threading.ThreadPoolWorkQueue.EnsureThreadRequested () [0x0001f] in <12384316d9c54a0b99f94c42f2c92f80>:0
121 at System.Threading.ThreadPoolWorkQueue.Enqueue (System.Threading.IThreadPoolWorkItem,bool) [0x00071] in <12384316d9c54a0b99f94c42f2c92f80>:0
122 at System.Threading.ThreadPool.UnsafeQueueCustomWorkItem (System.Threading.IThreadPoolWorkItem,bool) [0x00007] in <12384316d9c54a0b99f94c42f2c92f80>:0
123 at System.Threading.Tasks.ThreadPoolTaskScheduler.QueueTask (System.Threading.Tasks.Task) [0x00072] in <12384316d9c54a0b99f94c42f2c92f80>:0
124 at System.Threading.Tasks.Task.ScheduleAndStart (bool) [0x00063] in <12384316d9c54a0b99f94c42f2c92f80>:0
125 at System.Threading.Tasks.Task`1<bool>.StartNew (System.Threading.Tasks.Task,System.Func`2<object, bool>,object,System.Threading.CancellationToken,System.Threading.Tasks.TaskCreationOptions,System.Threading.Tasks.InternalTaskOptions,System.Threading.Tasks.TaskScheduler) [0x00032] in <12384316d9c54a0b99f94c42f2c92f80>:0
126 at System.Threading.Tasks.TaskFactory.StartNew<bool> (System.Func`2<object, bool>,object,System.Threading.CancellationToken,System.Threading.Tasks.TaskCreationOptions,System.Threading.Tasks.TaskScheduler) [0x00007] in <12384316d9c54a0b99f94c42f2c92f80>:0
127 at System.Net.ServicePointScheduler/AsyncManualResetEvent.Set () [0x00039] in <35f76809d1304f7b9923e43abe01f2f7>:0
128 at System.Net.ServicePointScheduler.Run () [0x00021] in <35f76809d1304f7b9923e43abe01f2f7>:0
129 at System.Net.ServicePointScheduler.SendRequest (System.Net.WebOperation,string) [0x0001e] in <35f76809d1304f7b9923e43abe01f2f7>:0
130 at System.Net.ServicePoint.SendRequest (System.Net.WebOperation,string) [0x0002f] in <35f76809d1304f7b9923e43abe01f2f7>:0
131 at System.Net.HttpWebRequest.SendRequest (bool,System.Net.BufferOffsetSize,System.Threading.CancellationToken) [0x00076] in <35f76809d1304f7b9923e43abe01f2f7>:0
132 at System.Net.HttpWebRequest/<MyGetResponseAsync>d__246.MoveNext () [0x000cb] in <35f76809d1304f7b9923e43abe01f2f7>:0
133 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<System.Net.HttpWebResponse>.Start<System.Net.HttpWebRequest/<MyGetResponseAsync>d__246> (System.Net.HttpWebRequest/<MyGetResponseAsync>d__246&) [0x0002c] in <12384316d9c54a0b99f94c42f2c92f80>:0
134 at System.Net.HttpWebRequest.MyGetResponseAsync (System.Threading.CancellationToken) [0x00024] in <35f76809d1304f7b9923e43abe01f2f7>:0
135 at System.Net.HttpWebRequest.RunWithTimeout<T_REF> (System.Func`2<System.Threading.CancellationToken, System.Threading.Tasks.Task`1<T_REF>>) [0x0000d] in <35f76809d1304f7b9923e43abe01f2f7>:0
136 at System.Net.HttpWebRequest.BeginGetResponse (System.AsyncCallback,object) [0x0003d] in <35f76809d1304f7b9923e43abe01f2f7>:0
137 at System.Threading.Tasks.TaskFactory`1<TResult_REF>.FromAsyncImpl (System.Func`3<System.AsyncCallback, object, System.IAsyncResult>,System.Func`2<System.IAsyncResult, TResult_REF>,System.Action`1<System.IAsyncResult>,object,System.Threading.Tasks.TaskCreationOptions) [0x0008a] in <12384316d9c54a0b99f94c42f2c92f80>:0
138 at System.Threading.Tasks.TaskFactory`1<TResult_REF>.FromAsync (System.Func`3<System.AsyncCallback, object, System.IAsyncResult>,System.Func`2<System.IAsyncResult, TResult_REF>,object) [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
139 at System.Net.WebRequest.<GetResponseAsync>b__79_0 () [0x00005] in <35f76809d1304f7b9923e43abe01f2f7>:0
140 at System.Threading.Tasks.Task`1<TResult_REF>.InnerInvoke () [0x0000f] in <12384316d9c54a0b99f94c42f2c92f80>:0
141 at System.Threading.Tasks.Task.Execute () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
142 at System.Threading.Tasks.Task.ExecutionContextCallback (object) [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
143 at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <12384316d9c54a0b99f94c42f2c92f80>:0
144 at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
145 at System.Threading.Tasks.Task.ExecuteWithThreadLocal (System.Threading.Tasks.Task&) [0x00034] in <12384316d9c54a0b99f94c42f2c92f80>:0
146 at System.Threading.Tasks.Task.ExecuteEntry (bool) [0x0004a] in <12384316d9c54a0b99f94c42f2c92f80>:0
147 at System.Threading.Tasks.Task.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
148 at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <12384316d9c54a0b99f94c42f2c92f80>:0
149 at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
150 at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <12384316d9c54a0b99f94c42f2c92f80>:0
151
152[Switching to Thread 0x7fffefbfd640 (LWP 254286)]
153
154Thread 7 "Thread Pool Wor" hit Breakpoint 1, clone () at /usr/src/debug/sys-libs/glibc-2.32-r2/glibc-2.32/sysdeps/unix/sysv/linux/x86_64/clone.S:52
15552 movq $-EINVAL,%rax
156
157"Thread Pool Worker" tid=0x7fffefbfd640 this=0x7ffff6f7c798 , thread handle : 0x7fffe0001200, state : not waiting
158 at <unknown> <0xffffffff>
159 at (wrapper managed-to-native) System.Threading.ThreadPool.RequestWorkerThread () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
160 at System.Threading.ThreadPoolWorkQueue.EnsureThreadRequested () [0x0001f] in <12384316d9c54a0b99f94c42f2c92f80>:0
161 at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x0003a] in <12384316d9c54a0b99f94c42f2c92f80>:0
162 at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
163 at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <12384316d9c54a0b99f94c42f2c92f80>:0
164
165[Switching to Thread 0x7fffefdfe640 (LWP 254285)]
166
167Thread 6 "Thread Pool Wor" hit Breakpoint 1, clone () at /usr/src/debug/sys-libs/glibc-2.32-r2/glibc-2.32/sysdeps/unix/sysv/linux/x86_64/clone.S:52
16852 movq $-EINVAL,%rax
169
170"Thread Pool Worker" tid=0x7fffefdfe640 this=0x7ffff6f7c650 , thread handle : 0x7fffdc001200, state : not waiting
171 at <unknown> <0xffffffff>
172 at (wrapper managed-to-native) System.Threading.Thread.Thread_internal (System.Threading.Thread,System.MulticastDelegate) [0x0000c] in <12384316d9c54a0b99f94c42f2c92f80>:0
173 at System.Threading.Thread.StartInternal (object,System.Threading.StackCrawlMark&) [0x0001a] in <12384316d9c54a0b99f94c42f2c92f80>:0
174 at System.Threading.Thread.Start (System.Threading.StackCrawlMark&) [0x00028] in <12384316d9c54a0b99f94c42f2c92f80>:0
175 at System.Threading.Thread.Start () [0x00002] in <12384316d9c54a0b99f94c42f2c92f80>:0
176 at System.Threading.Timer/Scheduler.InitScheduler () [0x00024] in <12384316d9c54a0b99f94c42f2c92f80>:0
177 at System.Threading.Timer/Scheduler..ctor () [0x0002e] in <12384316d9c54a0b99f94c42f2c92f80>:0
178 at System.Threading.Timer/Scheduler..cctor () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
179 at (wrapper runtime-invoke) object.runtime_invoke_void (object,intptr,intptr,intptr) [0x0002a] in <12384316d9c54a0b99f94c42f2c92f80>:0
180 at <unknown> <0xffffffff>
181 at System.Threading.Timer.get_scheduler () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
182 at System.Threading.Timer.Change (long,long,bool) [0x000a5] in <12384316d9c54a0b99f94c42f2c92f80>:0
183 at System.Threading.Timer.Init (System.Threading.TimerCallback,object,long,long) [0x0002a] in <12384316d9c54a0b99f94c42f2c92f80>:0
184 at System.Threading.Timer..ctor (System.Threading.TimerCallback,object,int,int) [0x00006] in <12384316d9c54a0b99f94c42f2c92f80>:0
185 at (wrapper remoting-invoke-with-check) System.Threading.Timer..ctor (System.Threading.TimerCallback,object,int,int) [0x0001d] in <12384316d9c54a0b99f94c42f2c92f80>:0
186 at System.Threading.Tasks.Task.Delay (int,System.Threading.CancellationToken) [0x0006e] in <12384316d9c54a0b99f94c42f2c92f80>:0
187 at System.Net.ServicePointScheduler/<WaitAsync>d__46.MoveNext () [0x0002b] in <35f76809d1304f7b9923e43abe01f2f7>:0
188 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<bool>.Start<System.Net.ServicePointScheduler/<WaitAsync>d__46> (System.Net.ServicePointScheduler/<WaitAsync>d__46&) [0x0002c] in <12384316d9c54a0b99f94c42f2c92f80>:0
189 at System.Net.ServicePointScheduler.WaitAsync (System.Threading.Tasks.Task,int) [0x00024] in <35f76809d1304f7b9923e43abe01f2f7>:0
190 at System.Net.ServicePointScheduler/AsyncManualResetEvent.WaitAsync (int) [0x0000d] in <35f76809d1304f7b9923e43abe01f2f7>:0
191 at System.Net.ServicePointScheduler/<RunScheduler>d__32.MoveNext () [0x000ac] in <35f76809d1304f7b9923e43abe01f2f7>:0
192 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<System.Net.ServicePointScheduler/<RunScheduler>d__32> (System.Net.ServicePointScheduler/<RunScheduler>d__32&) [0x0002c] in <12384316d9c54a0b99f94c42f2c92f80>:0
193 at System.Net.ServicePointScheduler.RunScheduler () [0x0001c] in <35f76809d1304f7b9923e43abe01f2f7>:0
194 at System.Net.ServicePointScheduler.<Run>b__31_0 () [0x00000] in <35f76809d1304f7b9923e43abe01f2f7>:0
195 at System.Threading.Tasks.Task`1<TResult_REF>.InnerInvoke () [0x0000f] in <12384316d9c54a0b99f94c42f2c92f80>:0
196 at System.Threading.Tasks.Task.Execute () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
197 at System.Threading.Tasks.Task.ExecutionContextCallback (object) [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
198 at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <12384316d9c54a0b99f94c42f2c92f80>:0
199 at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
200 at System.Threading.Tasks.Task.ExecuteWithThreadLocal (System.Threading.Tasks.Task&) [0x00034] in <12384316d9c54a0b99f94c42f2c92f80>:0
201 at System.Threading.Tasks.Task.ExecuteEntry (bool) [0x0004a] in <12384316d9c54a0b99f94c42f2c92f80>:0
202 at System.Threading.Tasks.Task.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
203 at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <12384316d9c54a0b99f94c42f2c92f80>:0
204 at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
205 at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <12384316d9c54a0b99f94c42f2c92f80>:0
206
207
208Thread 6 "Thread Pool Wor" hit Breakpoint 1, clone () at /usr/src/debug/sys-libs/glibc-2.32-r2/glibc-2.32/sysdeps/unix/sysv/linux/x86_64/clone.S:52
20952 movq $-EINVAL,%rax
210
211"Thread Pool Worker" tid=0x7fffefdfe640 this=0x7ffff6f7c650 , thread handle : 0x7fffdc001200, state : not waiting
212 at <unknown> <0xffffffff>
213 at (wrapper managed-to-native) System.IOSelector.Add (intptr,System.IOSelectorJob) [0x00000] in <35f76809d1304f7b9923e43abe01f2f7>:0
214 at System.Net.Sockets.Socket.BeginSConnect (System.Net.Sockets.SocketAsyncResult) [0x001da] in <35f76809d1304f7b9923e43abe01f2f7>:0
215 at System.Net.Sockets.Socket.BeginConnect (System.Net.EndPoint,System.AsyncCallback,object) [0x0002b] in <35f76809d1304f7b9923e43abe01f2f7>:0
216 at System.Net.WebConnection/<>c.<Connect>b__16_0 (System.Net.IPEndPoint,System.AsyncCallback,object) [0x00000] in <35f76809d1304f7b9923e43abe01f2f7>:0
217 at System.Threading.Tasks.TaskFactory`1<System.Threading.Tasks.VoidTaskResult>.FromAsyncImpl<System.Net.IPEndPoint> (System.Func`4<System.Net.IPEndPoint, System.AsyncCallback, object, System.IAsyncResult>,System.Func`2<System.IAsyncResult, System.Threading.Tasks.VoidTaskResult>,System.Action`1<System.IAsyncResult>,System.Net.IPEndPoint,object,System.Threading.Tasks.TaskCreationOptions) [0x0008b] in <12384316d9c54a0b99f94c42f2c92f80>:0
218 at System.Threading.Tasks.TaskFactory.FromAsync<TArg1_REF> (System.Func`4<TArg1_REF, System.AsyncCallback, object, System.IAsyncResult>,System.Action`1<System.IAsyncResult>,TArg1_REF,object,System.Threading.Tasks.TaskCreationOptions) [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
219 at System.Threading.Tasks.TaskFactory.FromAsync<TArg1_REF> (System.Func`4<TArg1_REF, System.AsyncCallback, object, System.IAsyncResult>,System.Action`1<System.IAsyncResult>,TArg1_REF,object) [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
220 at System.Net.WebConnection/<Connect>d__16.MoveNext () [0x0013b] in <35f76809d1304f7b9923e43abe01f2f7>:0
221 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<System.Net.WebConnection/<Connect>d__16> (System.Net.WebConnection/<Connect>d__16&) [0x0002c] in <12384316d9c54a0b99f94c42f2c92f80>:0
222 at System.Net.WebConnection.Connect (System.Net.WebOperation,System.Threading.CancellationToken) [0x0002c] in <35f76809d1304f7b9923e43abe01f2f7>:0
223 at System.Net.WebConnection/<InitConnection>d__19.MoveNext () [0x00055] in <35f76809d1304f7b9923e43abe01f2f7>:0
224 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<System.Net.WebRequestStream>.Start<System.Net.WebConnection/<InitConnection>d__19> (System.Net.WebConnection/<InitConnection>d__19&) [0x0002c] in <12384316d9c54a0b99f94c42f2c92f80>:0
225 at System.Net.WebConnection.InitConnection (System.Net.WebOperation,System.Threading.CancellationToken) [0x0002c] in <35f76809d1304f7b9923e43abe01f2f7>:0
226 at System.Net.WebOperation/<Run>d__58.MoveNext () [0x0003d] in <35f76809d1304f7b9923e43abe01f2f7>:0
227 at System.Runtime.CompilerServices.AsyncVoidMethodBuilder.Start<System.Net.WebOperation/<Run>d__58> (System.Net.WebOperation/<Run>d__58&) [0x0002c] in <12384316d9c54a0b99f94c42f2c92f80>:0
228 at System.Net.WebOperation.Run () [0x0001c] in <35f76809d1304f7b9923e43abe01f2f7>:0
229 at System.Net.WebConnection.StartOperation (System.Net.WebOperation,bool) [0x00075] in <35f76809d1304f7b9923e43abe01f2f7>:0
230 at System.Net.ServicePointScheduler/ConnectionGroup.CreateOrReuseConnection (System.Net.WebOperation,bool) [0x00051] in <35f76809d1304f7b9923e43abe01f2f7>:0
231 at System.Net.ServicePointScheduler.SchedulerIteration (System.Net.ServicePointScheduler/ConnectionGroup) [0x00012] in <35f76809d1304f7b9923e43abe01f2f7>:0
232 at System.Net.ServicePointScheduler.RunSchedulerIteration () [0x0000b] in <35f76809d1304f7b9923e43abe01f2f7>:0
233 at System.Net.ServicePointScheduler/<RunScheduler>d__32.MoveNext () [0x002c4] in <35f76809d1304f7b9923e43abe01f2f7>:0
234 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<System.Net.ServicePointScheduler/<RunScheduler>d__32> (System.Net.ServicePointScheduler/<RunScheduler>d__32&) [0x0002c] in <12384316d9c54a0b99f94c42f2c92f80>:0
235 at System.Net.ServicePointScheduler.RunScheduler () [0x0001c] in <35f76809d1304f7b9923e43abe01f2f7>:0
236 at System.Net.ServicePointScheduler.<Run>b__31_0 () [0x00000] in <35f76809d1304f7b9923e43abe01f2f7>:0
237 at System.Threading.Tasks.Task`1<TResult_REF>.InnerInvoke () [0x0000f] in <12384316d9c54a0b99f94c42f2c92f80>:0
238 at System.Threading.Tasks.Task.Execute () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
239 at System.Threading.Tasks.Task.ExecutionContextCallback (object) [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
240 at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <12384316d9c54a0b99f94c42f2c92f80>:0
241 at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
242 at System.Threading.Tasks.Task.ExecuteWithThreadLocal (System.Threading.Tasks.Task&) [0x00034] in <12384316d9c54a0b99f94c42f2c92f80>:0
243 at System.Threading.Tasks.Task.ExecuteEntry (bool) [0x0004a] in <12384316d9c54a0b99f94c42f2c92f80>:0
244 at System.Threading.Tasks.Task.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
245 at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <12384316d9c54a0b99f94c42f2c92f80>:0
246 at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
247 at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <12384316d9c54a0b99f94c42f2c92f80>:0
248
249........[ThreadCount:11]^C
250Thread 1 "mono" received signal SIGINT, Interrupt.
251[Switching to Thread 0x7ffff7c34740 (LWP 254277)]
252futex_wait_cancelable (private=0x0, expected=0x0, futex_word=0x555555a11008 <sleep_cond+40>) at /var/tmp/portage/sys-libs/glibc-2.32-r2/work/glibc-2.32/nptl/../sysdeps/nptl/futex-internal.h:183
253183 /var/tmp/portage/sys-libs/glibc-2.32-r2/work/glibc-2.32/nptl/../sysdeps/nptl/futex-internal.h: No such file or directory.

They are related to https://github.com/mono/mono/blob/0be756d6096e07bbd90350e6d5e5e536af031b53/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs#L641 :

internal void EnsureThreadRequested()
{
    //
    // If we have not yet requested #procs threads from the VM, then request a new thread.
    // Note that there is a separate count in the VM which will also be incremented in this case, 
    // which is handled by RequestWorkerThread.
    //
    int count = numOutstandingThreadRequests;
    while (count < ThreadPoolGlobals.processorCount)
    {
        int prev = Interlocked.CompareExchange(ref numOutstandingThreadRequests, count+1, count);
        if (prev == count)
        {
            ThreadPool.RequestWorkerThread();
            break;
        }
        count = prev;
    }
}

Which looks to me just means "create worker thread if it's less than processor count"...

Vort added a comment.Nov 3 2020, 11:13 AM

Let's see how thread count grows with wp_cyrlat code.
I have changed usual dots to thread count in program output:

//Console.Write('.');
Console.Write($"[{Process.GetCurrentProcess().Threads.Count}]");

Here is the result:

Scanning page titles[13][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][15][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21][21] Done

So rise in thread count happens not instantly.
Which means 4 requests, which I made in test program may be not enough to reproduce problem.
I will try to improve test program.

Vort added a comment.Tue, Nov 3, 12:03 PM

More close recreation of what happens in my bot:
mcs ToolforgeDebug.cs -r:System.Net.Http

using System;
using System.Threading;
using System.Diagnostics;
using System.Threading.Tasks;
using System.Net.Http;

namespace ToolforgeDebug
{
    class Program
    {
        HttpClient hc;

        async Task<string> MakeRequestAsync()
        {
            var request = new HttpRequestMessage(HttpMethod.Get,
                "https://worldtimeapi.org/api/timezone/Europe/London.txt");
            var response = await hc.SendAsync(request);
            return await response.Content.ReadAsStringAsync();
        }

        string MakeRequest()
        {
            return MakeRequestAsync().Result;
        }

        Program()
        {
            hc = new HttpClient();

            int workerThreads;
            int completionPortThreads;
            ThreadPool.GetMinThreads(out workerThreads, out completionPortThreads);
            ThreadPool.SetMaxThreads(workerThreads, completionPortThreads);

            for (int i = 0; i < 256; i++)
            {
                MakeRequest();
                Console.Write($"[{Process.GetCurrentProcess().Threads.Count}]");
                Thread.Sleep(500);
            }


            Console.Write($"Done");
            Thread.Sleep(Timeout.Infinite);

        }

        static void Main(string[] args)
        {
            new Program();
        }
    }
}

Same pattern can be seen.

tools.wikitasks@tools-sgebastion-07:~/tf_debug$ mono ToolforgeDebug.exe
[11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][14][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15]Done^C

By the way, I have found that in MS .NET something similar happens:

[17][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][18][20][20][20][20][20][20][19][19][20][20][19][20][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][17][17][17][17][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][21][21][21][21][21][21][21][21][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][19][19][20][20][20]Done^C

But results looks more stable - thread count is going down sometimes.

Vort added a comment.Tue, Nov 3, 4:58 PM

It looks like Mono ThreadPool have 3 thread counters:
gint16 num_active = counter._.starting + counter._.working + counter._.parked;
counter._.working should be limited by counter._.working >= counter._.max_working condition.
Also I was not able to find where maximum value for completion_port_threads is used. Maybe Mono just ignores it.
So thread pool should have not more than 4 working threads + have some amount of starting and parked threads.
If working threads count exceeds 4, most likely it is a bug.

Mono have such lines in its code:
mono_trace (G_LOG_LEVEL_DEBUG, MONO_TRACE_THREADPOOL,
Maybe enabling MONO_TRACE_THREADPOOL option and running my bot can help with revealing what is actually happens there.

Sorry, took a break. I can't reproduce this locally, for some reason.

1zhuyifei1999@zhuyifei1999-ThinkPad-T480 ~/T266377 $ gdb -batch -ex 'source debug.gdb' --args mono ToolforgeDebug.exe
2GEF for linux ready, type `gef' to start, `gef config' to configure
375 commands loaded for GDB 10.1 using Python engine 3.7
4[*] 5 commands could not be loaded, run `gef missing` to know why.
5[+] Configuration from '/home/zhuyifei1999/.gef.rc' restored
6[+] 10 extra commands added from '/home/zhuyifei1999/gdb/gef-extras/scripts'
7warning: File "/usr/bin/mono-sgen-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/src/linux/".
8To enable execution of this file add
9 add-auto-load-safe-path /usr/bin/mono-sgen-gdb.py
10line to your configuration file "/home/zhuyifei1999/.gdbinit".
11To completely disable this security protection add
12 set auto-load safe-path /
13line to your configuration file "/home/zhuyifei1999/.gdbinit".
14For more information about this security protection see the
15"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
16 info "(gdb)Auto-loading safe path"
17Function "clone" not defined.
18Breakpoint 1 (clone) pending.
19[Thread debugging using libthread_db enabled]
20Using host libthread_db library "/lib64/libthread_db.so.1".
21
22Breakpoint 1, clone () at /usr/src/debug/sys-libs/glibc-2.32-r2/glibc-2.32/sysdeps/unix/sysv/linux/x86_64/clone.S:52
2352 movq $-EINVAL,%rax
24No mono thread associated with this thread
25
26Thread 1 "mono" hit Breakpoint 1, clone () at /usr/src/debug/sys-libs/glibc-2.32-r2/glibc-2.32/sysdeps/unix/sysv/linux/x86_64/clone.S:52
2752 movq $-EINVAL,%rax
28
29"<unnamed thread>" tid=0x7ffff7c34740 this=0x7ffff6f7c130 , thread handle : 0x555555a1e0c0, state : not waiting
30
31
32Thread 1 "mono" hit Breakpoint 1, clone () at /usr/src/debug/sys-libs/glibc-2.32-r2/glibc-2.32/sysdeps/unix/sysv/linux/x86_64/clone.S:52
3352 movq $-EINVAL,%rax
34
35"<unnamed thread>" tid=0x7ffff7c34740 this=0x7ffff6f7c130 , thread handle : 0x555555a1e0c0, state : not waiting
36 at <unknown> <0xffffffff>
37 at (wrapper managed-to-native) System.Threading.Thread.Thread_internal (System.Threading.Thread,System.MulticastDelegate) [0x0000c] in <12384316d9c54a0b99f94c42f2c92f80>:0
38 at System.Threading.Thread.StartInternal (object,System.Threading.StackCrawlMark&) [0x0001a] in <12384316d9c54a0b99f94c42f2c92f80>:0
39 at System.Threading.Thread.Start (System.Threading.StackCrawlMark&) [0x00028] in <12384316d9c54a0b99f94c42f2c92f80>:0
40 at System.Threading.Thread.Start () [0x00002] in <12384316d9c54a0b99f94c42f2c92f80>:0
41 at System.Threading.Timer/Scheduler.InitScheduler () [0x00024] in <12384316d9c54a0b99f94c42f2c92f80>:0
42 at System.Threading.Timer/Scheduler..ctor () [0x0002e] in <12384316d9c54a0b99f94c42f2c92f80>:0
43 at System.Threading.Timer/Scheduler..cctor () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
44 at (wrapper runtime-invoke) object.runtime_invoke_void (object,intptr,intptr,intptr) [0x0002a] in <12384316d9c54a0b99f94c42f2c92f80>:0
45 at <unknown> <0xffffffff>
46 at System.Threading.Timer.get_scheduler () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
47 at System.Threading.Timer.Change (long,long,bool) [0x000a5] in <12384316d9c54a0b99f94c42f2c92f80>:0
48 at System.Threading.Timer.Change (int,int) [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
49 at (wrapper remoting-invoke-with-check) System.Threading.Timer.Change (int,int) [0x00033] in <12384316d9c54a0b99f94c42f2c92f80>:0
50 at System.Threading.CancellationTokenSource.CancelAfter (int) [0x0004c] in <12384316d9c54a0b99f94c42f2c92f80>:0
51 at System.Threading.CancellationTokenSource.CancelAfter (System.TimeSpan) [0x00022] in <12384316d9c54a0b99f94c42f2c92f80>:0
52 at System.Net.Http.HttpClient.SendAsync (System.Net.Http.HttpRequestMessage,System.Net.Http.HttpCompletionOption,System.Threading.CancellationToken) [0x0005b] in <a4197fdaee01421b945cbd6ee2f46b92>:0
53 at System.Net.Http.HttpClient.SendAsync (System.Net.Http.HttpRequestMessage) [0x00008] in <a4197fdaee01421b945cbd6ee2f46b92>:0
54 at ToolforgeDebug.Program/<MakeRequestAsync>c__async0.MoveNext () [0x0003a] in <7a3da679d10d4fdeac3bc097aedddb23>:0
55 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<string>.Start<ToolforgeDebug.Program/<MakeRequestAsync>c__async0> (ToolforgeDebug.Program/<MakeRequestAsync>c__async0&) [0x0002c] in <12384316d9c54a0b99f94c42f2c92f80>:0
56 at ToolforgeDebug.Program.MakeRequestAsync () [0x00014] in <7a3da679d10d4fdeac3bc097aedddb23>:0
57 at ToolforgeDebug.Program.MakeRequest () [0x00000] in <7a3da679d10d4fdeac3bc097aedddb23>:0
58 at ToolforgeDebug.Program..ctor () [0x00029] in <7a3da679d10d4fdeac3bc097aedddb23>:0
59 at ToolforgeDebug.Program.Main (string[]) [0x00000] in <7a3da679d10d4fdeac3bc097aedddb23>:0
60 at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object (object,intptr,intptr,intptr) [0x0002a] in <7a3da679d10d4fdeac3bc097aedddb23>:0
61
62
63Thread 1 "mono" hit Breakpoint 1, clone () at /usr/src/debug/sys-libs/glibc-2.32-r2/glibc-2.32/sysdeps/unix/sysv/linux/x86_64/clone.S:52
6452 movq $-EINVAL,%rax
65
66"<unnamed thread>" tid=0x7ffff7c34740 this=0x7ffff6f7c130 , thread handle : 0x555555a1e0c0, state : not waiting
67 at <unknown> <0xffffffff>
68 at (wrapper managed-to-native) System.Threading.ThreadPool.RequestWorkerThread () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
69 at System.Threading.ThreadPoolWorkQueue.EnsureThreadRequested () [0x0001f] in <12384316d9c54a0b99f94c42f2c92f80>:0
70 at System.Threading.ThreadPoolWorkQueue.Enqueue (System.Threading.IThreadPoolWorkItem,bool) [0x00071] in <12384316d9c54a0b99f94c42f2c92f80>:0
71 at System.Threading.ThreadPool.UnsafeQueueCustomWorkItem (System.Threading.IThreadPoolWorkItem,bool) [0x00007] in <12384316d9c54a0b99f94c42f2c92f80>:0
72 at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object_byte (object,intptr,intptr,intptr) [0x0002a] in <12384316d9c54a0b99f94c42f2c92f80>:0
73 at <unknown> <0xffffffff>
74 at (wrapper managed-to-native) object.__icall_wrapper_mono_delegate_begin_invoke (object,intptr) [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
75 at (wrapper delegate-begin-invoke) <Module>.begin_invoke_IAsyncResult__this___string_AsyncCallback_object (string,System.AsyncCallback,object) [0x0001d] in <35f76809d1304f7b9923e43abe01f2f7>:0
76 at System.Net.Dns.BeginGetHostAddresses (string,System.AsyncCallback,object) [0x00048] in <35f76809d1304f7b9923e43abe01f2f7>:0
77 at System.Net.Sockets.MultipleConnectAsync.StartConnectAsync (System.Net.Sockets.SocketAsyncEventArgs,System.Net.DnsEndPoint) [0x000a5] in <35f76809d1304f7b9923e43abe01f2f7>:0
78 at System.Net.Sockets.Socket.ConnectAsync (System.Net.Sockets.SocketType,System.Net.Sockets.ProtocolType,System.Net.Sockets.SocketAsyncEventArgs) [0x00087] in <35f76809d1304f7b9923e43abe01f2f7>:0
79 at System.Net.Http.ConnectHelper/<ConnectAsync>d__2.MoveNext () [0x00058] in <a4197fdaee01421b945cbd6ee2f46b92>:0
80 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<System.ValueTuple`2<System.Net.Sockets.Socket, System.IO.Stream>>.Start<System.Net.Http.ConnectHelper/<ConnectAsync>d__2> (System.Net.Http.ConnectHelper/<ConnectAsync>d__2&) [0x0002c] in <12384316d9c54a0b99f94c42f2c92f80>:0
81 at System.Net.Http.ConnectHelper.ConnectAsync (string,int,System.Threading.CancellationToken) [0x0002c] in <a4197fdaee01421b945cbd6ee2f46b92>:0
82 at System.Net.Http.HttpConnectionPool/<CreateConnectionAsync>d__44.MoveNext () [0x000be] in <a4197fdaee01421b945cbd6ee2f46b92>:0
83 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<System.ValueTuple`2<System.Net.Http.HttpConnection, System.Net.Http.HttpResponseMessage>>.Start<System.Net.Http.HttpConnectionPool/<CreateConnectionAsync>d__44> (System.Net.Http.HttpConnectionPool/<CreateConnectionAsync>d__44&) [0x0002c] in <12384316d9c54a0b99f94c42f2c92f80>:0
84 at System.Net.Http.HttpConnectionPool.CreateConnectionAsync (System.Net.Http.HttpRequestMessage,System.Threading.CancellationToken) [0x0002c] in <a4197fdaee01421b945cbd6ee2f46b92>:0
85 at System.Net.Http.HttpConnectionPool.GetConnectionAsync (System.Net.Http.HttpRequestMessage,System.Threading.CancellationToken) [0x000af] in <a4197fdaee01421b945cbd6ee2f46b92>:0
86 at System.Net.Http.HttpConnectionPool/<SendWithRetryAsync>d__39.MoveNext () [0x00018] in <a4197fdaee01421b945cbd6ee2f46b92>:0
87 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<System.Net.Http.HttpResponseMessage>.Start<System.Net.Http.HttpConnectionPool/<SendWithRetryAsync>d__39> (System.Net.Http.HttpConnectionPool/<SendWithRetryAsync>d__39&) [0x0002c] in <12384316d9c54a0b99f94c42f2c92f80>:0
88 at System.Net.Http.HttpConnectionPool.SendWithRetryAsync (System.Net.Http.HttpRequestMessage,bool,System.Threading.CancellationToken) [0x00034] in <a4197fdaee01421b945cbd6ee2f46b92>:0
89 at System.Net.Http.HttpConnectionPool.SendWithProxyAuthAsync (System.Net.Http.HttpRequestMessage,bool,System.Threading.CancellationToken) [0x0003a] in <a4197fdaee01421b945cbd6ee2f46b92>:0
90 at System.Net.Http.HttpConnectionPool.SendAsync (System.Net.Http.HttpRequestMessage,bool,System.Threading.CancellationToken) [0x0002f] in <a4197fdaee01421b945cbd6ee2f46b92>:0
91 at System.Net.Http.HttpConnectionPoolManager.SendAsyncCore (System.Net.Http.HttpRequestMessage,System.Uri,bool,bool,System.Threading.CancellationToken) [0x000cb] in <a4197fdaee01421b945cbd6ee2f46b92>:0
92 at System.Net.Http.HttpConnectionPoolManager.SendAsync (System.Net.Http.HttpRequestMessage,bool,System.Threading.CancellationToken) [0x00008] in <a4197fdaee01421b945cbd6ee2f46b92>:0
93 at System.Net.Http.HttpConnectionHandler.SendAsync (System.Net.Http.HttpRequestMessage,System.Threading.CancellationToken) [0x00000] in <a4197fdaee01421b945cbd6ee2f46b92>:0
94 at System.Net.Http.RedirectHandler/<SendAsync>d__4.MoveNext () [0x00048] in <a4197fdaee01421b945cbd6ee2f46b92>:0
95 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<System.Net.Http.HttpResponseMessage>.Start<System.Net.Http.RedirectHandler/<SendAsync>d__4> (System.Net.Http.RedirectHandler/<SendAsync>d__4&) [0x0002c] in <12384316d9c54a0b99f94c42f2c92f80>:0
96 at System.Net.Http.RedirectHandler.SendAsync (System.Net.Http.HttpRequestMessage,System.Threading.CancellationToken) [0x0002c] in <a4197fdaee01421b945cbd6ee2f46b92>:0
97 at System.Net.Http.SocketsHttpHandler.SendAsync (System.Net.Http.HttpRequestMessage,System.Threading.CancellationToken) [0x00029] in <a4197fdaee01421b945cbd6ee2f46b92>:0
98 at System.Net.Http.SocketsHttpHandler.System.Net.Http.IMonoHttpClientHandler.SendAsync (System.Net.Http.HttpRequestMessage,System.Threading.CancellationToken) [0x00000] in <a4197fdaee01421b945cbd6ee2f46b92>:0
99 at System.Net.Http.HttpClientHandler.SendAsync (System.Net.Http.HttpRequestMessage,System.Threading.CancellationToken) [0x00000] in <a4197fdaee01421b945cbd6ee2f46b92>:0
100 at System.Net.Http.HttpMessageInvoker.SendAsync (System.Net.Http.HttpRequestMessage,System.Threading.CancellationToken) [0x0002e] in <a4197fdaee01421b945cbd6ee2f46b92>:0
101 at System.Net.Http.HttpClient.SendAsync (System.Net.Http.HttpRequestMessage,System.Net.Http.HttpCompletionOption,System.Threading.CancellationToken) [0x0007b] in <a4197fdaee01421b945cbd6ee2f46b92>:0
102 at System.Net.Http.HttpClient.SendAsync (System.Net.Http.HttpRequestMessage) [0x00008] in <a4197fdaee01421b945cbd6ee2f46b92>:0
103 at ToolforgeDebug.Program/<MakeRequestAsync>c__async0.MoveNext () [0x0003a] in <7a3da679d10d4fdeac3bc097aedddb23>:0
104 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<string>.Start<ToolforgeDebug.Program/<MakeRequestAsync>c__async0> (ToolforgeDebug.Program/<MakeRequestAsync>c__async0&) [0x0002c] in <12384316d9c54a0b99f94c42f2c92f80>:0
105 at ToolforgeDebug.Program.MakeRequestAsync () [0x00014] in <7a3da679d10d4fdeac3bc097aedddb23>:0
106 at ToolforgeDebug.Program.MakeRequest () [0x00000] in <7a3da679d10d4fdeac3bc097aedddb23>:0
107 at ToolforgeDebug.Program..ctor () [0x00029] in <7a3da679d10d4fdeac3bc097aedddb23>:0
108 at ToolforgeDebug.Program.Main (string[]) [0x00000] in <7a3da679d10d4fdeac3bc097aedddb23>:0
109 at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object (object,intptr,intptr,intptr) [0x0002a] in <7a3da679d10d4fdeac3bc097aedddb23>:0
110
111
112Thread 1 "mono" hit Breakpoint 1, clone () at /usr/src/debug/sys-libs/glibc-2.32-r2/glibc-2.32/sysdeps/unix/sysv/linux/x86_64/clone.S:52
11352 movq $-EINVAL,%rax
114
115"<unnamed thread>" tid=0x7ffff7c34740 this=0x7ffff6f7c130 , thread handle : 0x555555a1e0c0, state : not waiting
116 at <unknown> <0xffffffff>
117 at (wrapper managed-to-native) System.Threading.ThreadPool.RequestWorkerThread () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
118 at System.Threading.ThreadPoolWorkQueue.EnsureThreadRequested () [0x0001f] in <12384316d9c54a0b99f94c42f2c92f80>:0
119 at System.Threading.ThreadPoolWorkQueue.Enqueue (System.Threading.IThreadPoolWorkItem,bool) [0x00071] in <12384316d9c54a0b99f94c42f2c92f80>:0
120 at System.Threading.ThreadPool.UnsafeQueueCustomWorkItem (System.Threading.IThreadPoolWorkItem,bool) [0x00007] in <12384316d9c54a0b99f94c42f2c92f80>:0
121 at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object_byte (object,intptr,intptr,intptr) [0x0002a] in <12384316d9c54a0b99f94c42f2c92f80>:0
122 at <unknown> <0xffffffff>
123 at (wrapper managed-to-native) object.__icall_wrapper_mono_delegate_begin_invoke (object,intptr) [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
124 at (wrapper delegate-begin-invoke) <Module>.begin_invoke_IAsyncResult__this___string_AsyncCallback_object (string,System.AsyncCallback,object) [0x0001d] in <35f76809d1304f7b9923e43abe01f2f7>:0
125 at System.Net.Dns.BeginGetHostAddresses (string,System.AsyncCallback,object) [0x00048] in <35f76809d1304f7b9923e43abe01f2f7>:0
126 at System.Net.Sockets.MultipleConnectAsync.StartConnectAsync (System.Net.Sockets.SocketAsyncEventArgs,System.Net.DnsEndPoint) [0x000a5] in <35f76809d1304f7b9923e43abe01f2f7>:0
127 at System.Net.Sockets.Socket.ConnectAsync (System.Net.Sockets.SocketType,System.Net.Sockets.ProtocolType,System.Net.Sockets.SocketAsyncEventArgs) [0x00087] in <35f76809d1304f7b9923e43abe01f2f7>:0
128 at System.Net.Http.ConnectHelper/<ConnectAsync>d__2.MoveNext () [0x00058] in <a4197fdaee01421b945cbd6ee2f46b92>:0
129 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<System.ValueTuple`2<System.Net.Sockets.Socket, System.IO.Stream>>.Start<System.Net.Http.ConnectHelper/<ConnectAsync>d__2> (System.Net.Http.ConnectHelper/<ConnectAsync>d__2&) [0x0002c] in <12384316d9c54a0b99f94c42f2c92f80>:0
130 at System.Net.Http.ConnectHelper.ConnectAsync (string,int,System.Threading.CancellationToken) [0x0002c] in <a4197fdaee01421b945cbd6ee2f46b92>:0
131 at System.Net.Http.HttpConnectionPool/<CreateConnectionAsync>d__44.MoveNext () [0x000be] in <a4197fdaee01421b945cbd6ee2f46b92>:0
132 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<System.ValueTuple`2<System.Net.Http.HttpConnection, System.Net.Http.HttpResponseMessage>>.Start<System.Net.Http.HttpConnectionPool/<CreateConnectionAsync>d__44> (System.Net.Http.HttpConnectionPool/<CreateConnectionAsync>d__44&) [0x0002c] in <12384316d9c54a0b99f94c42f2c92f80>:0
133 at System.Net.Http.HttpConnectionPool.CreateConnectionAsync (System.Net.Http.HttpRequestMessage,System.Threading.CancellationToken) [0x0002c] in <a4197fdaee01421b945cbd6ee2f46b92>:0
134 at System.Net.Http.HttpConnectionPool.GetConnectionAsync (System.Net.Http.HttpRequestMessage,System.Threading.CancellationToken) [0x000af] in <a4197fdaee01421b945cbd6ee2f46b92>:0
135 at System.Net.Http.HttpConnectionPool/<SendWithRetryAsync>d__39.MoveNext () [0x00018] in <a4197fdaee01421b945cbd6ee2f46b92>:0
136 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<System.Net.Http.HttpResponseMessage>.Start<System.Net.Http.HttpConnectionPool/<SendWithRetryAsync>d__39> (System.Net.Http.HttpConnectionPool/<SendWithRetryAsync>d__39&) [0x0002c] in <12384316d9c54a0b99f94c42f2c92f80>:0
137 at System.Net.Http.HttpConnectionPool.SendWithRetryAsync (System.Net.Http.HttpRequestMessage,bool,System.Threading.CancellationToken) [0x00034] in <a4197fdaee01421b945cbd6ee2f46b92>:0
138 at System.Net.Http.HttpConnectionPool.SendWithProxyAuthAsync (System.Net.Http.HttpRequestMessage,bool,System.Threading.CancellationToken) [0x0003a] in <a4197fdaee01421b945cbd6ee2f46b92>:0
139 at System.Net.Http.HttpConnectionPool.SendAsync (System.Net.Http.HttpRequestMessage,bool,System.Threading.CancellationToken) [0x0002f] in <a4197fdaee01421b945cbd6ee2f46b92>:0
140 at System.Net.Http.HttpConnectionPoolManager.SendAsyncCore (System.Net.Http.HttpRequestMessage,System.Uri,bool,bool,System.Threading.CancellationToken) [0x000cb] in <a4197fdaee01421b945cbd6ee2f46b92>:0
141 at System.Net.Http.HttpConnectionPoolManager.SendAsync (System.Net.Http.HttpRequestMessage,bool,System.Threading.CancellationToken) [0x00008] in <a4197fdaee01421b945cbd6ee2f46b92>:0
142 at System.Net.Http.HttpConnectionHandler.SendAsync (System.Net.Http.HttpRequestMessage,System.Threading.CancellationToken) [0x00000] in <a4197fdaee01421b945cbd6ee2f46b92>:0
143 at System.Net.Http.RedirectHandler/<SendAsync>d__4.MoveNext () [0x00048] in <a4197fdaee01421b945cbd6ee2f46b92>:0
144 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<System.Net.Http.HttpResponseMessage>.Start<System.Net.Http.RedirectHandler/<SendAsync>d__4> (System.Net.Http.RedirectHandler/<SendAsync>d__4&) [0x0002c] in <12384316d9c54a0b99f94c42f2c92f80>:0
145 at System.Net.Http.RedirectHandler.SendAsync (System.Net.Http.HttpRequestMessage,System.Threading.CancellationToken) [0x0002c] in <a4197fdaee01421b945cbd6ee2f46b92>:0
146 at System.Net.Http.SocketsHttpHandler.SendAsync (System.Net.Http.HttpRequestMessage,System.Threading.CancellationToken) [0x00029] in <a4197fdaee01421b945cbd6ee2f46b92>:0
147 at System.Net.Http.SocketsHttpHandler.System.Net.Http.IMonoHttpClientHandler.SendAsync (System.Net.Http.HttpRequestMessage,System.Threading.CancellationToken) [0x00000] in <a4197fdaee01421b945cbd6ee2f46b92>:0
148 at System.Net.Http.HttpClientHandler.SendAsync (System.Net.Http.HttpRequestMessage,System.Threading.CancellationToken) [0x00000] in <a4197fdaee01421b945cbd6ee2f46b92>:0
149 at System.Net.Http.HttpMessageInvoker.SendAsync (System.Net.Http.HttpRequestMessage,System.Threading.CancellationToken) [0x0002e] in <a4197fdaee01421b945cbd6ee2f46b92>:0
150 at System.Net.Http.HttpClient.SendAsync (System.Net.Http.HttpRequestMessage,System.Net.Http.HttpCompletionOption,System.Threading.CancellationToken) [0x0007b] in <a4197fdaee01421b945cbd6ee2f46b92>:0
151 at System.Net.Http.HttpClient.SendAsync (System.Net.Http.HttpRequestMessage) [0x00008] in <a4197fdaee01421b945cbd6ee2f46b92>:0
152 at ToolforgeDebug.Program/<MakeRequestAsync>c__async0.MoveNext () [0x0003a] in <7a3da679d10d4fdeac3bc097aedddb23>:0
153 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<string>.Start<ToolforgeDebug.Program/<MakeRequestAsync>c__async0> (ToolforgeDebug.Program/<MakeRequestAsync>c__async0&) [0x0002c] in <12384316d9c54a0b99f94c42f2c92f80>:0
154 at ToolforgeDebug.Program.MakeRequestAsync () [0x00014] in <7a3da679d10d4fdeac3bc097aedddb23>:0
155 at ToolforgeDebug.Program.MakeRequest () [0x00000] in <7a3da679d10d4fdeac3bc097aedddb23>:0
156 at ToolforgeDebug.Program..ctor () [0x00029] in <7a3da679d10d4fdeac3bc097aedddb23>:0
157 at ToolforgeDebug.Program.Main (string[]) [0x00000] in <7a3da679d10d4fdeac3bc097aedddb23>:0
158 at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object (object,intptr,intptr,intptr) [0x0002a] in <7a3da679d10d4fdeac3bc097aedddb23>:0
159
160[Switching to Thread 0x7fffefdfe640 (LWP 1997547)]
161
162Thread 6 "Thread Pool Wor" hit Breakpoint 1, clone () at /usr/src/debug/sys-libs/glibc-2.32-r2/glibc-2.32/sysdeps/unix/sysv/linux/x86_64/clone.S:52
16352 movq $-EINVAL,%rax
164
165"Thread Pool Worker" tid=0x7fffefdfe640 this=0x7ffff6f7c650 , thread handle : 0x7fffdc001200, state : not waiting
166 at <unknown> <0xffffffff>
167 at (wrapper managed-to-native) System.Threading.ThreadPool.RequestWorkerThread () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
168 at System.Threading.ThreadPoolWorkQueue.EnsureThreadRequested () [0x0001f] in <12384316d9c54a0b99f94c42f2c92f80>:0
169 at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x0003a] in <12384316d9c54a0b99f94c42f2c92f80>:0
170 at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
171 at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <12384316d9c54a0b99f94c42f2c92f80>:0
172
173
174Thread 6 "Thread Pool Wor" hit Breakpoint 1, clone () at /usr/src/debug/sys-libs/glibc-2.32-r2/glibc-2.32/sysdeps/unix/sysv/linux/x86_64/clone.S:52
17552 movq $-EINVAL,%rax
176
177"Thread Pool Worker" tid=0x7fffefdfe640 this=0x7ffff6f7c650 , thread handle : 0x7fffdc001200, state : not waiting
178 at <unknown> <0xffffffff>
179 at (wrapper managed-to-native) System.IOSelector.Add (intptr,System.IOSelectorJob) [0x00000] in <35f76809d1304f7b9923e43abe01f2f7>:0
180 at System.Net.Sockets.Socket.BeginSConnect (System.Net.Sockets.SocketAsyncResult) [0x001da] in <35f76809d1304f7b9923e43abe01f2f7>:0
181 at System.Net.Sockets.Socket.ConnectAsync (System.Net.Sockets.SocketAsyncEventArgs) [0x0005b] in <35f76809d1304f7b9923e43abe01f2f7>:0
182 at System.Net.Sockets.MultipleConnectAsync.AttemptConnection (System.Net.Sockets.Socket,System.Net.Sockets.SocketAsyncEventArgs) [0x00013] in <35f76809d1304f7b9923e43abe01f2f7>:0
183 at System.Net.Sockets.MultipleConnectAsync.AttemptConnection () [0x00035] in <35f76809d1304f7b9923e43abe01f2f7>:0
184 at System.Net.Sockets.MultipleConnectAsync.DoDnsCallback (System.IAsyncResult,bool) [0x000aa] in <35f76809d1304f7b9923e43abe01f2f7>:0
185 at System.Net.Sockets.MultipleConnectAsync.DnsCallback (System.IAsyncResult) [0x00008] in <35f76809d1304f7b9923e43abe01f2f7>:0
186 at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object (object,intptr,intptr,intptr) [0x0002a] in <12384316d9c54a0b99f94c42f2c92f80>:0
187 at <unknown> <0xffffffff>
188 at (wrapper managed-to-native) System.Runtime.Remoting.Messaging.AsyncResult.Invoke (System.Runtime.Remoting.Messaging.AsyncResult) [0x0000c] in <12384316d9c54a0b99f94c42f2c92f80>:0
189 at System.Runtime.Remoting.Messaging.AsyncResult.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
190 at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <12384316d9c54a0b99f94c42f2c92f80>:0
191 at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
192 at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <12384316d9c54a0b99f94c42f2c92f80>:0
193
194
195Thread 6 "Thread Pool Wor" hit Breakpoint 1, clone () at /usr/src/debug/sys-libs/glibc-2.32-r2/glibc-2.32/sysdeps/unix/sysv/linux/x86_64/clone.S:52
19652 movq $-EINVAL,%rax
197
198"Thread Pool Worker" tid=0x7fffefdfe640 this=0x7ffff6f7c650 , thread handle : 0x7fffdc001200, state : not waiting
199 at <unknown> <0xffffffff>
200 at (wrapper managed-to-native) System.Threading.ThreadPool.RequestWorkerThread () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
201 at System.Threading.ThreadPoolWorkQueue.EnsureThreadRequested () [0x0001f] in <12384316d9c54a0b99f94c42f2c92f80>:0
202 at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x0003a] in <12384316d9c54a0b99f94c42f2c92f80>:0
203 at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
204 at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <12384316d9c54a0b99f94c42f2c92f80>:0
205
206[Switching to Thread 0x7fffefbfd640 (LWP 1997548)]
207
208Thread 7 "Thread Pool Wor" hit Breakpoint 1, clone () at /usr/src/debug/sys-libs/glibc-2.32-r2/glibc-2.32/sysdeps/unix/sysv/linux/x86_64/clone.S:52
20952 movq $-EINVAL,%rax
210
211"Thread Pool Worker" tid=0x7fffefbfd640 this=0x7ffff6f7c798 , thread handle : 0x7fffe0001200, state : not waiting
212 at <unknown> <0xffffffff>
213 at (wrapper managed-to-native) System.Threading.ThreadPool.RequestWorkerThread () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
214 at System.Threading.ThreadPoolWorkQueue.EnsureThreadRequested () [0x0001f] in <12384316d9c54a0b99f94c42f2c92f80>:0
215 at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x0003a] in <12384316d9c54a0b99f94c42f2c92f80>:0
216 at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
217 at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <12384316d9c54a0b99f94c42f2c92f80>:0
218
219[Switching to Thread 0x7fffefdfe640 (LWP 1997547)]
220
221Thread 6 "Thread Pool Wor" hit Breakpoint 1, clone () at /usr/src/debug/sys-libs/glibc-2.32-r2/glibc-2.32/sysdeps/unix/sysv/linux/x86_64/clone.S:52
22252 movq $-EINVAL,%rax
223
224"Thread Pool Worker" tid=0x7fffefdfe640 this=0x7ffff6f7c650 , thread handle : 0x7fffdc001200, state : not waiting
225 at <unknown> <0xffffffff>
226 at (wrapper managed-to-native) System.Threading.ThreadPool.RequestWorkerThread () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
227 at System.Threading.ThreadPoolWorkQueue.EnsureThreadRequested () [0x0001f] in <12384316d9c54a0b99f94c42f2c92f80>:0
228 at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x0003a] in <12384316d9c54a0b99f94c42f2c92f80>:0
229 at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
230 at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <12384316d9c54a0b99f94c42f2c92f80>:0
231
232[11][Switching to Thread 0x7fffef273640 (LWP 1997570)]
233
234Thread 11 "Thread Pool Wor" hit Breakpoint 1, clone () at /usr/src/debug/sys-libs/glibc-2.32-r2/glibc-2.32/sysdeps/unix/sysv/linux/x86_64/clone.S:52
23552 movq $-EINVAL,%rax
236
237"Thread Pool Worker" tid=0x7fffef273640 this=0x7ffff6f7ccb8 , thread handle : 0x7fffd0001200, state : not waiting
238 at <unknown> <0xffffffff>
239 at (wrapper managed-to-native) System.Threading.ThreadPool.RequestWorkerThread () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
240 at System.Threading.ThreadPoolWorkQueue.EnsureThreadRequested () [0x0001f] in <12384316d9c54a0b99f94c42f2c92f80>:0
241 at System.Threading.ThreadPoolWorkQueue.Enqueue (System.Threading.IThreadPoolWorkItem,bool) [0x00071] in <12384316d9c54a0b99f94c42f2c92f80>:0
242 at System.Threading.ThreadPool.QueueUserWorkItemHelper (System.Threading.WaitCallback,object,System.Threading.StackCrawlMark&,bool,bool) [0x00016] in <12384316d9c54a0b99f94c42f2c92f80>:0
243 at System.Threading.ThreadPool.UnsafeQueueUserWorkItem (System.Threading.WaitCallback,object) [0x00002] in <12384316d9c54a0b99f94c42f2c92f80>:0
244 at System.Net.Sockets.SocketAsyncResult.Complete () [0x0005f] in <35f76809d1304f7b9923e43abe01f2f7>:0
245 at System.Net.Sockets.SocketAsyncResult.Complete (int) [0x00007] in <35f76809d1304f7b9923e43abe01f2f7>:0
246 at System.Net.Sockets.Socket/<>c.<.cctor>b__366_11 (System.IOAsyncResult) [0x0007c] in <35f76809d1304f7b9923e43abe01f2f7>:0
247 at System.IOSelectorJob.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00000] in <35f76809d1304f7b9923e43abe01f2f7>:0
248 at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <12384316d9c54a0b99f94c42f2c92f80>:0
249 at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
250 at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <12384316d9c54a0b99f94c42f2c92f80>:0
251
252[Switching to Thread 0x7fffefdfe640 (LWP 1997547)]
253
254Thread 6 "Thread Pool Wor" hit Breakpoint 1, clone () at /usr/src/debug/sys-libs/glibc-2.32-r2/glibc-2.32/sysdeps/unix/sysv/linux/x86_64/clone.S:52
25552 movq $-EINVAL,%rax
256
257"Thread Pool Worker" tid=0x7fffefdfe640 this=0x7ffff6f7c650 , thread handle : 0x7fffdc001200, state : not waiting
258 at <unknown> <0xffffffff>
259 at (wrapper managed-to-native) System.Threading.ThreadPool.RequestWorkerThread () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
260 at System.Threading.ThreadPoolWorkQueue.EnsureThreadRequested () [0x0001f] in <12384316d9c54a0b99f94c42f2c92f80>:0
261 at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x0003a] in <12384316d9c54a0b99f94c42f2c92f80>:0
262 at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
263 at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <12384316d9c54a0b99f94c42f2c92f80>:0
264
265[13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][Switching to Thread 0x7fffeee71640 (LWP 1997644)]
266
267Thread 13 "Thread Pool Wor" hit Breakpoint 1, clone () at /usr/src/debug/sys-libs/glibc-2.32-r2/glibc-2.32/sysdeps/unix/sysv/linux/x86_64/clone.S:52
26852 movq $-EINVAL,%rax
269
270"Thread Pool Worker" tid=0x7fffeee71640 this=0x7ffff6f7cf48 , thread handle : 0x7fffc8001200, state : not waiting
271 at <unknown> <0xffffffff>
272 at (wrapper managed-to-native) System.Threading.ThreadPool.RequestWorkerThread () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
273 at System.Threading.ThreadPoolWorkQueue.EnsureThreadRequested () [0x0001f] in <12384316d9c54a0b99f94c42f2c92f80>:0
274 at System.Threading.ThreadPoolWorkQueue.Enqueue (System.Threading.IThreadPoolWorkItem,bool) [0x00071] in <12384316d9c54a0b99f94c42f2c92f80>:0
275 at System.Threading.ThreadPool.QueueUserWorkItemHelper (System.Threading.WaitCallback,object,System.Threading.StackCrawlMark&,bool,bool) [0x00016] in <12384316d9c54a0b99f94c42f2c92f80>:0
276 at System.Threading.ThreadPool.UnsafeQueueUserWorkItem (System.Threading.WaitCallback,object) [0x00002] in <12384316d9c54a0b99f94c42f2c92f80>:0
277 at System.Net.Sockets.SocketAsyncResult.Complete () [0x0005f] in <35f76809d1304f7b9923e43abe01f2f7>:0
278 at System.Net.Sockets.SocketAsyncResult.Complete (int) [0x00007] in <35f76809d1304f7b9923e43abe01f2f7>:0
279 at System.Net.Sockets.Socket/<>c.<.cctor>b__366_11 (System.IOAsyncResult) [0x0007c] in <35f76809d1304f7b9923e43abe01f2f7>:0
280 at System.IOSelectorJob.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00000] in <35f76809d1304f7b9923e43abe01f2f7>:0
281 at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <12384316d9c54a0b99f94c42f2c92f80>:0
282 at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
283 at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <12384316d9c54a0b99f94c42f2c92f80>:0
284
285[Switching to Thread 0x7fffef273640 (LWP 1997570)]
286
287Thread 11 "Thread Pool Wor" hit Breakpoint 1, clone () at /usr/src/debug/sys-libs/glibc-2.32-r2/glibc-2.32/sysdeps/unix/sysv/linux/x86_64/clone.S:52
28852 movq $-EINVAL,%rax
289
290"Thread Pool Worker" tid=0x7fffef273640 this=0x7ffff6f7ccb8 , thread handle : 0x7fffd0001200, state : not waiting
291 at <unknown> <0xffffffff>
292 at (wrapper managed-to-native) System.Threading.ThreadPool.RequestWorkerThread () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
293 at System.Threading.ThreadPoolWorkQueue.EnsureThreadRequested () [0x0001f] in <12384316d9c54a0b99f94c42f2c92f80>:0
294 at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x0003a] in <12384316d9c54a0b99f94c42f2c92f80>:0
295 at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
296 at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <12384316d9c54a0b99f94c42f2c92f80>:0
297
298[14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][Switching to Thread 0x7fffef9fc640 (LWP 1997553)]
299
300Thread 8 "Thread Pool I/O" hit Breakpoint 1, clone () at /usr/src/debug/sys-libs/glibc-2.32-r2/glibc-2.32/sysdeps/unix/sysv/linux/x86_64/clone.S:52
30152 movq $-EINVAL,%rax
302
303"Thread Pool I/O Selector" tid=0x7fffef9fc640 this=0x7ffff6f7c8e0 , thread handle : 0x7fffd4001200, state : waiting
304 at <unknown> <0xffffffff>
305 at (wrapper managed-to-native) System.Threading.ThreadPool.RequestWorkerThread () [0x00000] in <12384316d9c54a0b99f94c42f2c92f80>:0
306 at System.Threading.ThreadPoolWorkQueue.EnsureThreadRequested () [0x0001f] in <12384316d9c54a0b99f94c42f2c92f80>:0
307 at System.Threading.ThreadPoolWorkQueue.Enqueue (System.Threading.IThreadPoolWorkItem,bool) [0x00071] in <12384316d9c54a0b99f94c42f2c92f80>:0
308 at System.Threading.ThreadPool.UnsafeQueueCustomWorkItem (System.Threading.IThreadPoolWorkItem,bool) [0x00007] in <12384316d9c54a0b99f94c42f2c92f80>:0
309 at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object_byte (object,intptr,intptr,intptr) [0x0002a] in <12384316d9c54a0b99f94c42f2c92f80>:0
310
311[14][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13]
312Thread 7 "Thread Pool Wor" received signal SIG35, Real-time event 35.
313
314Thread 11 "Thread Pool Wor" received signal SIG35, Real-time event 35.
315
316Thread 6 "Thread Pool Wor" received signal SIG35, Real-time event 35.
317
318Thread 15 "Thread Pool Wor" received signal SIG35, Real-time event 35.
319
320Thread 4 "Timer-Scheduler" received signal SIG35, Real-time event 35.
321
322Thread 16 "Thread Pool I/O" received signal SIG35, Real-time event 35.
323
324Thread 13 "Thread Pool Wor" received signal SIG35, Real-time event 35.
325
326Thread 9 "Thread Pool Wor" received signal SIG35, Real-time event 35.
327
328Thread 14 "Thread Pool Wor" received signal SIG35, Real-time event 35.
329
330Thread 16 "Thread Pool I/O" received signal SIG36, Real-time event 36.
331
332Thread 7 "Thread Pool Wor" received signal SIG36, Real-time event 36.
333
334Thread 13 "Thread Pool Wor" received signal SIG36, Real-time event 36.
335
336Thread 14 "Thread Pool Wor" received signal SIG36, Real-time event 36.
337
338Thread 15 "Thread Pool Wor" received signal SIG36, Real-time event 36.
339
340Thread 9 "Thread Pool Wor" received signal SIG36, Real-time event 36.
341
342Thread 4 "Timer-Scheduler" received signal SIG36, Real-time event 36.
343
344Thread 11 "Thread Pool Wor" received signal SIG36, Real-time event 36.
345
346Thread 6 "Thread Pool Wor" received signal SIG36, Real-time event 36.
347[13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13]
348Thread 14 "Thread Pool Wor" received signal SIG35, Real-time event 35.
349
350Thread 7 "Thread Pool Wor" received signal SIG35, Real-time event 35.
351
352Thread 11 "Thread Pool Wor" received signal SIG35, Real-time event 35.
353
354Thread 4 "Timer-Scheduler" received signal SIG35, Real-time event 35.
355
356Thread 15 "Thread Pool Wor" received signal SIG35, Real-time event 35.
357
358Thread 16 "Thread Pool I/O" received signal SIG35, Real-time event 35.
359
360Thread 13 "Thread Pool Wor" received signal SIG35, Real-time event 35.
361
362Thread 6 "Thread Pool Wor" received signal SIG35, Real-time event 35.
363
364Thread 9 "Thread Pool Wor" received signal SIG35, Real-time event 35.
365
366Thread 7 "Thread Pool Wor" received signal SIG36, Real-time event 36.
367
368Thread 6 "Thread Pool Wor" received signal SIG36, Real-time event 36.
369
370Thread 4 "Timer-Scheduler" received signal SIG36, Real-time event 36.
371
372Thread 11 "Thread Pool Wor" received signal SIG36, Real-time event 36.
373
374Thread 9 "Thread Pool Wor" received signal SIG36, Real-time event 36.
375
376Thread 14 "Thread Pool Wor" received signal SIG36, Real-time event 36.
377
378Thread 16 "Thread Pool I/O" received signal SIG36, Real-time event 36.
379
380Thread 15 "Thread Pool Wor" received signal SIG36, Real-time event 36.
381
382Thread 13 "Thread Pool Wor" received signal SIG36, Real-time event 36.
383
384Thread 14 "Thread Pool Wor" received signal SIG35, Real-time event 35.
385
386Thread 13 "Thread Pool Wor" received signal SIG35, Real-time event 35.
387
388Thread 15 "Thread Pool Wor" received signal SIG35, Real-time event 35.
389
390Thread 16 "Thread Pool I/O" received signal SIG35, Real-time event 35.
391
392Thread 7 "Thread Pool Wor" received signal SIG35, Real-time event 35.
393
394Thread 11 "Thread Pool Wor" received signal SIG35, Real-time event 35.
395
396Thread 9 "Thread Pool Wor" received signal SIG35, Real-time event 35.
397
398Thread 4 "Timer-Scheduler" received signal SIG35, Real-time event 35.
399
400Thread 6 "Thread Pool Wor" received signal SIG35, Real-time event 35.
401
402Thread 6 "Thread Pool Wor" received signal SIG36, Real-time event 36.
403
404Thread 4 "Timer-Scheduler" received signal SIG36, Real-time event 36.
405
406Thread 15 "Thread Pool Wor" received signal SIG36, Real-time event 36.
407
408Thread 13 "Thread Pool Wor" received signal SIG36, Real-time event 36.
409
410Thread 7 "Thread Pool Wor" received signal SIG36, Real-time event 36.
411
412Thread 9 "Thread Pool Wor" received signal SIG36, Real-time event 36.
413
414Thread 16 "Thread Pool I/O" received signal SIG36, Real-time event 36.
415
416Thread 14 "Thread Pool Wor" received signal SIG36, Real-time event 36.
417
418Thread 11 "Thread Pool Wor" received signal SIG36, Real-time event 36.
419[13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13]Done^C
420Thread 1 "mono" received signal SIGINT, Interrupt.
421[Switching to Thread 0x7ffff7c34740 (LWP 1997485)]
422futex_wait_cancelable (private=0x0, expected=0x0, futex_word=0x555555a11008 <sleep_cond+40>) at /var/tmp/portage/sys-libs/glibc-2.32-r2/work/glibc-2.32/nptl/../sysdeps/nptl/futex-internal.h:183
423183 /var/tmp/portage/sys-libs/glibc-2.32-r2/work/glibc-2.32/nptl/../sysdeps/nptl/futex-internal.h: No such file or directory.

This might need some look-into by someone with knowledge of mono threadpool internals. From a C POV it's just glibc heaps going too many because too many threads are spawning.

Other things that might me worth considering are:

  • Alternative malloc implementations -- jemalloc maybe? I never tested that so don't take my word for it
  • Hook on mmap, mprotect, munmap syscalls so we manage our own virtual memory, unnecessary pages don't get mapped at all -- this is possible, but a lot of effort.
Vort added a comment.Fri, Nov 6, 2:43 PM

I can't reproduce this locally, for some reason.

Do you use the same version as on Toolforge? v5.12.0.226
Other version may not have this bug.

Other things that might me worth considering are:

First of all, location of the problem should be calculated.
If that is not a bug, but limitation, then it is possible to think about hacks.

So now we know that sometimes Mono spawns more "Thread Pool Worker" threads than needed.
It may happen because of bug in Mono (most likely) or because of bugs in other components (libraries, OS, maybe even CPU).

Problem happens randomly (wp_cyrlat may end with 23 threads as well as 17, on Grid).
Which means it may be related to computer's load.
Problem is related to number of CPU cores. So it may worth testing inside VM with 4 cores, the same amount as used on Grid.

Vort added a comment.Fri, Nov 6, 3:02 PM

Mono 6.8.0.105 inside my VM:

vort@ubuntu:~/toolforge$ mono ToolforgeDebug.exe
[11][11][11][11][11][11][11][11][11][11][11][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][12][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][10][10][10][10][11]^C

This is for sure not what happens at Toolforge.

Vort added a comment.Fri, Nov 6, 3:03 PM

So looks like I need to build latest Mono at Toolforge.
Again. Some time ago I reverted to "default" mono.

Hmm interesting. I'm running 6.10.0.104 and didn't consider version to be an issue. Will test later.

Mono v5.12.0.226 didn't compile:

make[2]: Leaving directory '/home/zhuyifei1999/T266377/mono-5.12.0.226/data'
Making all in runtime
make[2]: Entering directory '/home/zhuyifei1999/T266377/mono-5.12.0.226/runtime'
if test -w /home/zhuyifei1999/T266377/mono-5.12.0.226/mcs; then :; else chmod -R +w /home/zhuyifei1999/T266377/mono-5.12.0.226/mcs; fi
cd /home/zhuyifei1999/T266377/mono-5.12.0.226/mcs && make --no-print-directory -s NO_DIR_CHECK=1 PROFILES='binary_reference_assemblies net_4_x xbuild_12 xbuild_14           ' CC='gcc' all-profiles
Microsoft (R) Visual C# Compiler version 2.6.0.62309 (d3f6b8e7)
Copyright (C) Microsoft Corporation. All rights reserved.

AOT     [build-linux] mscorlib.dll.so
Cannot open assembly ',profile=./../../class/lib/build/csc.System.Core.dll.aotprofile': No such file or directory.
make[7]: *** [Makefile:61: ../../class/lib/build-linux//mscorlib.dll.so] Error 1

¯\_(ツ)_/¯

Vort added a comment.Sat, Nov 7, 2:10 PM

So looks like I need to build latest Mono at Toolforge.

I made it.
Mono v6.12.0.90 is installed at /data/project/wikitasks/mono now.
Build takes 7 hours by the way.

First results are good:
Maximum amount of threads for wp_cyrlat run was 14.
Let's see how it will behave at 2:00 in the night.

Vort added a comment.Sun, Nov 8, 7:10 AM

I'm not sure if it is a better result or not.
Most likely, yes, but more data is needed:

Scanning page titles[11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][11][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][15][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16] Done
Vort added a comment.Tue, Nov 10, 5:04 AM

Today's result was very strange.
Looks like Mono not fixed problem, but hackfixed it instead:

Scanning page titles[11][11][11][11][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][15][15][15][15][15][15][15][15][17][17][17][17][17][17][17][17][17][17][17][17][17][17][17][17][17][17][17][17][17][17][17][17][17][17][17][17][17][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][19][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][22][11][10][12][12][12][12][12][12][12][12][12][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][13][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][16][10][10][10][10][10][10][10][10][10][10][11][11][11][12][12][12][12][12][12][12][12][12][12][12][12][12][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][14][15][15][15][15][16][16][16][16][16][16][16][16][16][16][10][10] Done