Page MenuHomePhabricator

Java jobs run the Stretch grid seem to require a very large memory reservation
Open, Needs TriagePublic

Description

Originally reported at https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Strange_Memory_Behavior_on_Toolforge_with_Java by @Fastily:

Hi all, when I run the following command on toolforge:
jsub -once -mem 2g -quiet -j y -o a.txt java -Xmx1G -version
And view the log file:

$ cat a.txt
Error occurred during initialization of VM
Could not allocate metaspace: 1073741824 bytes

I see an error related to (lack of) memory. Changing the memory values to something crazy seems to yield the desired output.
For example:

$ jsub -once -mem 8g -quiet -j y -o a.txt java -Xmx4G -version
$ cat a.txt

results in the desired output

openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment (build 11.0.2+9-Debian-3bpo91)
OpenJDK 64-Bit Server VM (build 11.0.2+9-Debian-3bpo91, mixed mode, sharing)

That leaves me with the following questions

  1. Is this a bug or the expected behavior?
  2. Is the latter example the best practice on toolforge these days (i.e. set crazy memory values)?

Thanks in advance.

Related Objects

Event Timeline

bd808 created this task.Mar 27 2019, 1:18 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 27 2019, 1:18 AM

102:54:06 1 ✗ zhuyifei1999@tools-sgebastion-08: ~$ (ulimit -v $(( (1<<20) * 2 )); strace -e mmap,munmap,brk -f java -Xmx1G -version)
2brk(NULL) = 0x561dc983d000
3mmap(NULL, 105680, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f12f27e3000
4mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f12f27e1000
5mmap(NULL, 2200072, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f12f23c0000
6mmap(0x7f12f25d8000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18000) = 0x7f12f25d8000
7mmap(NULL, 2212936, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f12f21a3000
8mmap(0x7f12f23ba000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7f12f23ba000
9mmap(0x7f12f23bc000, 13384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f12f23bc000
10mmap(NULL, 2159536, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f12f1f93000
11mmap(0x7f12f21a1000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xe000) = 0x7f12f21a1000
12mmap(NULL, 2109680, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f12f1d8f000
13mmap(0x7f12f1f91000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f12f1f91000
14mmap(NULL, 3795296, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f12f19f0000
15mmap(0x7f12f1d85000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x195000) = 0x7f12f1d85000
16mmap(0x7f12f1d8b000, 14688, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f12f1d8b000
17mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f12f27df000
18munmap(0x7f12f27e3000, 105680) = 0
19brk(NULL) = 0x561dc983d000
20brk(0x561dc985e000) = 0x561dc985e000
21mmap(NULL, 20741976, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f12f0628000
22mmap(0x7f12f18a4000, 1011712, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x107c000) = 0x7f12f18a4000
23mmap(0x7f12f199b000, 347992, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f12f199b000
24mmap(NULL, 105680, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f12f27e3000
25mmap(NULL, 3674720, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f12f02a6000
26mmap(0x7f12f0618000, 49152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x172000) = 0x7f12f0618000
27mmap(0x7f12f0624000, 12896, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f12f0624000
28mmap(NULL, 3158248, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f12effa2000
29mmap(0x7f12f02a4000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x102000) = 0x7f12f02a4000
30mmap(NULL, 2188336, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f12efd8b000
31mmap(0x7f12effa0000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15000) = 0x7f12effa0000
32munmap(0x7f12f27e3000, 105680) = 0
33mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f12f26de000
34strace: Process 17316 attached
35[pid 17316] mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f12e7d8b000
36[pid 17316] munmap(0x7f12e7d8b000, 2576384) = 0
37[pid 17316] munmap(0x7f12ec000000, 64532480) = 0
38[pid 17316] mmap(NULL, 105680, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f12f27e3000
39[pid 17316] mmap(NULL, 2128832, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f12efb83000
40[pid 17316] mmap(0x7f12efd89000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7f12efd89000
41[pid 17316] munmap(0x7f12f27e3000, 105680) = 0
42[pid 17316] mmap(NULL, 2154912, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f12ef974000
43[pid 17316] mmap(0x7f12efb80000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xc000) = 0x7f12efb80000
44[pid 17316] mmap(NULL, 2270984, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f12ef749000
45[pid 17316] mmap(0x7f12ef971000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x28000) = 0x7f12ef971000
46[pid 17316] mmap(NULL, 8192, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f12f27fb000
47[pid 17316] mmap(0x7f12f27fb000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f12f27fb000
48[pid 17316] mmap(NULL, 102803208, PROT_READ, MAP_SHARED, 4, 0) = 0x7f12e1df5000
49[pid 17316] mmap(NULL, 32768, PROT_READ|PROT_WRITE, MAP_SHARED, 5, 0) = 0x7f12f27f3000
50[pid 17316] mmap(0x7f12f26de000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f12f26de000
51[pid 17316] mmap(NULL, 2126264, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f12ef541000
52[pid 17316] mmap(0x7f12ef747000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7f12ef747000
53[pid 17316] mmap(NULL, 2118048, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f12ef33b000
54[pid 17316] mmap(0x7f12ef53f000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7f12ef53f000
55[pid 17316] mmap(NULL, 141001128, PROT_READ, MAP_SHARED, 3, 0) = 0x7f12d977c000
56[pid 17316] mmap(NULL, 251658240, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f12ca77c000
57[pid 17316] mmap(0x7f12ca77c000, 2555904, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f12ca77c000
58[pid 17316] mmap(NULL, 49152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f12f27e7000
59[pid 17316] mmap(0x7f12f27e7000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f12f27e7000
60[pid 17316] mmap(0x7f12cad0b000, 2555904, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f12cad0b000
61[pid 17316] mmap(NULL, 962560, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f12ef250000
62[pid 17316] mmap(0x7f12ef250000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f12ef250000
63[pid 17316] mmap(0x7f12d2243000, 2555904, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f12d2243000
64[pid 17316] mmap(NULL, 962560, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f12ef165000
65[pid 17316] mmap(0x7f12ef165000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f12ef165000
66[pid 17316] --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=NULL} ---
67[pid 17316] mmap(0xc0000000, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0xc0000000
68[pid 17316] mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f12eef64000
69[pid 17316] mmap(0x7f12ef164000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f12ef164000
70[pid 17316] mmap(0xc0000000, 5570560, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xc0000000
71[pid 17316] mmap(0x7f12eef64000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f12eef64000
72[pid 17316] mmap(0xd5550000, 11206656, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xd5550000
73[pid 17316] mmap(NULL, 1400832, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f12eee0e000
74[pid 17316] mmap(0x7f12eee0e000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f12eee0e000
75[pid 17316] mmap(0x7f12ef00e000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f12ef00e000
76[pid 17316] mmap(0x800000000, 17489920, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x800000000
77[pid 17316] mmap(0x800000000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 4, 0x1000) = 0x800000000
78[pid 17316] mmap(0x800002000, 3915776, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0x3000) = 0x800002000
79[pid 17316] mmap(0x8003be000, 7188480, PROT_READ, MAP_PRIVATE|MAP_FIXED, 4, 0x3bf000) = 0x8003be000
80[pid 17316] mmap(0x800a99000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0xa9a000) = 0x800a99000
81[pid 17316] mmap(0x800a9a000, 6373376, PROT_READ, MAP_PRIVATE|MAP_FIXED, 4, 0xa9b000) = 0x800a9a000
82[pid 17316] mmap(0x800000000, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
83[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
84[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
85[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
86[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
87[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
88[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
89[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
90[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
91[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
92[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
93[pid 17316] mmap(0x840000000, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
94[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
95[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
96[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
97[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
98[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
99[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
100[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
101[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
102[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
103[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
104[pid 17316] mmap(0x880000000, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
105[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
106[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
107[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
108[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
109[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
110[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
111[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
112[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
113[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
114[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
115[pid 17316] mmap(0x8c0000000, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
116[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
117[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
118[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
119[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
120[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
121[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
122[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
123[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
124[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
125[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
126[pid 17316] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
127Error occurred during initialization of VM
128Could not allocate metaspace: 1073741824 bytes
129[pid 17316] munmap(0x7f12d977c000, 1731944) = 0
130[pid 17316] +++ exited with 1 +++
131+++ exited with 1 +++
13202:54:22 1 ✗ zhuyifei1999@tools-sgebastion-08: ~$ (ulimit -v unlimited; strace -e mmap,munmap,brk -f java -Xmx1G -version)
133brk(NULL) = 0x55c91eea1000
134mmap(NULL, 105680, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f78355b4000
135mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f78355b2000
136mmap(NULL, 2200072, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7835191000
137mmap(0x7f78353a9000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18000) = 0x7f78353a9000
138mmap(NULL, 2212936, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7834f74000
139mmap(0x7f783518b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7f783518b000
140mmap(0x7f783518d000, 13384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f783518d000
141mmap(NULL, 2159536, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7834d64000
142mmap(0x7f7834f72000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xe000) = 0x7f7834f72000
143mmap(NULL, 2109680, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7834b60000
144mmap(0x7f7834d62000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f7834d62000
145mmap(NULL, 3795296, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f78347c1000
146mmap(0x7f7834b56000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x195000) = 0x7f7834b56000
147mmap(0x7f7834b5c000, 14688, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7834b5c000
148mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f78355b0000
149munmap(0x7f78355b4000, 105680) = 0
150brk(NULL) = 0x55c91eea1000
151brk(0x55c91eec2000) = 0x55c91eec2000
152mmap(NULL, 20741976, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f78333f9000
153mmap(0x7f7834675000, 1011712, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x107c000) = 0x7f7834675000
154mmap(0x7f783476c000, 347992, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f783476c000
155mmap(NULL, 105680, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f78355b4000
156mmap(NULL, 3674720, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7833077000
157mmap(0x7f78333e9000, 49152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x172000) = 0x7f78333e9000
158mmap(0x7f78333f5000, 12896, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f78333f5000
159mmap(NULL, 3158248, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7832d73000
160mmap(0x7f7833075000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x102000) = 0x7f7833075000
161mmap(NULL, 2188336, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7832b5c000
162mmap(0x7f7832d71000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15000) = 0x7f7832d71000
163munmap(0x7f78355b4000, 105680) = 0
164mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f78354af000
165strace: Process 17334 attached
166[pid 17334] mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f782ab5c000
167[pid 17334] munmap(0x7f782ab5c000, 21643264) = 0
168[pid 17334] munmap(0x7f7830000000, 45465600) = 0
169[pid 17334] mmap(NULL, 105680, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f78355b4000
170[pid 17334] mmap(NULL, 2128832, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7832954000
171[pid 17334] mmap(0x7f7832b5a000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7f7832b5a000
172[pid 17334] munmap(0x7f78355b4000, 105680) = 0
173[pid 17334] mmap(NULL, 2154912, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7832745000
174[pid 17334] mmap(0x7f7832951000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xc000) = 0x7f7832951000
175[pid 17334] mmap(NULL, 2270984, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f783251a000
176[pid 17334] mmap(0x7f7832742000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x28000) = 0x7f7832742000
177[pid 17334] mmap(NULL, 8192, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f78355cc000
178[pid 17334] mmap(0x7f78355cc000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f78355cc000
179[pid 17334] mmap(NULL, 102803208, PROT_READ, MAP_SHARED, 4, 0) = 0x7f7825df5000
180[pid 17334] mmap(NULL, 32768, PROT_READ|PROT_WRITE, MAP_SHARED, 5, 0) = 0x7f78355c4000
181[pid 17334] mmap(0x7f78354af000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f78354af000
182[pid 17334] mmap(NULL, 2126264, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7832312000
183[pid 17334] mmap(0x7f7832518000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7f7832518000
184[pid 17334] mmap(NULL, 2118048, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f783210c000
185[pid 17334] mmap(0x7f7832310000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7f7832310000
186[pid 17334] mmap(NULL, 141001128, PROT_READ, MAP_SHARED, 3, 0) = 0x7f781d77c000
187[pid 17334] mmap(NULL, 251658240, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f780e77c000
188[pid 17334] mmap(0x7f780e77c000, 2555904, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f780e77c000
189[pid 17334] mmap(NULL, 49152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f78355b8000
190[pid 17334] mmap(0x7f78355b8000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f78355b8000
191[pid 17334] mmap(0x7f780ed0b000, 2555904, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f780ed0b000
192[pid 17334] mmap(NULL, 962560, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f7832021000
193[pid 17334] mmap(0x7f7832021000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7832021000
194[pid 17334] mmap(0x7f7816243000, 2555904, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7816243000
195[pid 17334] mmap(NULL, 962560, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f7831f36000
196[pid 17334] mmap(0x7f7831f36000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7831f36000
197[pid 17334] --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=NULL} ---
198[pid 17334] mmap(0xc0000000, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0xc0000000
199[pid 17334] mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f7831d35000
200[pid 17334] mmap(0x7f7831f35000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7831f35000
201[pid 17334] mmap(0xc0000000, 5570560, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xc0000000
202[pid 17334] mmap(0x7f7831d35000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7831d35000
203[pid 17334] mmap(0xd5550000, 11206656, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xd5550000
204[pid 17334] mmap(NULL, 1400832, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f7831bdf000
205[pid 17334] mmap(0x7f7831bdf000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7831bdf000
206[pid 17334] mmap(0x7f7831ddf000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7831ddf000
207[pid 17334] mmap(0x800000000, 17489920, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x800000000
208[pid 17334] mmap(0x800000000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 4, 0x1000) = 0x800000000
209[pid 17334] mmap(0x800002000, 3915776, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0x3000) = 0x800002000
210[pid 17334] mmap(0x8003be000, 7188480, PROT_READ, MAP_PRIVATE|MAP_FIXED, 4, 0x3bf000) = 0x8003be000
211[pid 17334] mmap(0x800a99000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0xa9a000) = 0x800a99000
212[pid 17334] mmap(0x800a9a000, 6373376, PROT_READ, MAP_PRIVATE|MAP_FIXED, 4, 0xa9b000) = 0x800a9a000
213[pid 17334] mmap(0x800000000, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f77ce77c000
214[pid 17334] munmap(0x7f77ce77c000, 1073741824) = 0
215[pid 17334] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f77ce77c000
216[pid 17334] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f778e77c000
217[pid 17334] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f774e77c000
218[pid 17334] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f770e77c000
219[pid 17334] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f76ce77c000
220[pid 17334] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f768e77c000
221[pid 17334] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f764e77c000
222[pid 17334] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f760e77c000
223[pid 17334] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f75ce77c000
224[pid 17334] mmap(NULL, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f758e77c000
225[pid 17334] munmap(0x7f77ce77c000, 1073741824) = 0
226[pid 17334] munmap(0x7f778e77c000, 1073741824) = 0
227[pid 17334] munmap(0x7f774e77c000, 1073741824) = 0
228[pid 17334] munmap(0x7f770e77c000, 1073741824) = 0
229[pid 17334] munmap(0x7f76ce77c000, 1073741824) = 0
230[pid 17334] munmap(0x7f768e77c000, 1073741824) = 0
231[pid 17334] munmap(0x7f764e77c000, 1073741824) = 0
232[pid 17334] munmap(0x7f760e77c000, 1073741824) = 0
233[pid 17334] munmap(0x7f75ce77c000, 1073741824) = 0
234[pid 17334] munmap(0x7f758e77c000, 1073741824) = 0
235[pid 17334] mmap(0x840000000, 1073741824, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x840000000
236[pid 17334] mmap(NULL, 135168, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f783548e000
237[pid 17334] mmap(NULL, 135168, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f783546d000
238[pid 17334] mmap(NULL, 8388608, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f78313df000
239[pid 17334] mmap(NULL, 163840, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7835445000
240[pid 17334] mmap(NULL, 372736, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f78353ea000
241[pid 17334] mmap(NULL, 528384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f783135e000
242[pid 17334] mmap(0x7f78313df000, 4194304, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f78313df000
243[pid 17334] mmap(0x840000000, 393216, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x840000000
244[pid 17334] mmap(NULL, 1056768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f783125c000
245strace: Process 17335 attached
246[pid 17335] mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f780677c000
247[pid 17335] munmap(0x7f780677c000, 25706496) = 0
248[pid 17335] munmap(0x7f780c000000, 41402368) = 0
249[pid 17334] mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f783115b000
250strace: Process 17336 attached
251[pid 17336] mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f7800000000
252[pid 17336] munmap(0x7f7804000000, 67108864) = 0
253[pid 17334] mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0 <unfinished ...>
254[pid 17336] mmap(0x7f783115b000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0 <unfinished ...>
255[pid 17334] <... mmap resumed> ) = 0x7f783105a000
256[pid 17336] <... mmap resumed> ) = 0x7f783115b000
257strace: Process 17337 attached
258[pid 17337] mmap(0x7f7804000000, 67108864, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f7804000000
259[pid 17337] mmap(0x7f783105a000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f783105a000
260[pid 17334] mmap(NULL, 20301776, PROT_READ, MAP_PRIVATE, 4, 0) = 0x7f780d41f000
261[pid 17334] mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f7830f59000
262strace: Process 17338 attached
263[pid 17338] mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f77f8000000
264[pid 17338] munmap(0x7f77fc000000, 67108864) = 0
265[pid 17338] mmap(0x7f7830f59000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7830f59000
266[pid 17334] mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f7830e58000
267strace: Process 17339 attached
268[pid 17339] mmap(0x7f77fc000000, 67108864, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f77fc000000
269[pid 17334] mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f7830d57000
270[pid 17339] mmap(0x7f7830e58000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0strace: Process 17340 attached
271) = 0x7f7830e58000
272[pid 17340] mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f77f0000000
273[pid 17340] munmap(0x7f77f4000000, 67108864) = 0
274[pid 17334] mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f7830c56000
275[pid 17340] mmap(0x7f7830d57000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7830d57000
276strace: Process 17341 attached
277[pid 17341] mmap(0x7f77f4000000, 67108864, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f77f4000000
278[pid 17341] mmap(0x7f7830c56000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7830c56000
279[pid 17334] mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f7830b55000
280strace: Process 17342 attached
281[pid 17342] mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f77e8000000
282[pid 17342] munmap(0x7f77ec000000, 67108864) = 0
283[pid 17342] mmap(0x7f7830b55000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7830b55000
284[pid 17334] mmap(NULL, 1056768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f7830a53000
285strace: Process 17343 attached
286[pid 17343] mmap(0x7f77ec000000, 67108864, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f77ec000000
287openjdk version "11.0.2" 2019-01-15
288OpenJDK Runtime Environment (build 11.0.2+9-Debian-3bpo91)
289OpenJDK 64-Bit Server VM (build 11.0.2+9-Debian-3bpo91, mixed mode, sharing)
290[pid 17334] mmap(0x7f78354af000, 16384, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f78354af000
291[pid 17334] mmap(0x7f78354af000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f78354af000
292[pid 17334] mmap(0x7f78354af000, 16384, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f78354af000
293[pid 17338] mmap(0x7f7830f59000, 16384, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0 <unfinished ...>
294[pid 17343] +++ exited with 0 +++
295[pid 17338] <... mmap resumed> ) = 0x7f7830f59000
296[pid 17338] +++ exited with 0 +++
297[pid 17335] +++ exited with 0 +++
298[pid 17334] +++ exited with 0 +++
299[pid 17333] munmap(0x7f781d77c000, 1731944) = 0
300[pid 17342] +++ exited with 0 +++
301[pid 17341] +++ exited with 0 +++
302[pid 17340] +++ exited with 0 +++
303[pid 17339] +++ exited with 0 +++
304[pid 17337] +++ exited with 0 +++
305[pid 17336] +++ exited with 0 +++
306+++ exited with 0 +++
30702:54:50 0 ✓ zhuyifei1999@tools-sgebastion-08: ~$

It wants 10 GiB of heap space... but what's wrong with giving it just that? It's not like it's a consumable given that every process has their own address space.

Is the latter example the best practice on toolforge these days (i.e. set crazy memory values)?

The latter example demonstrates exactly how grid engine is nuts.

Oh and, gdb doesn't help on why it needs 10GiB of heap space.

02:54:50 0 ✓ zhuyifei1999@tools-sgebastion-08: ~$ gdb java -ex 'catch syscall write' -ex 'r -Xmx1G -version'
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
[...]
Type "apropos word" to search for commands related to "word"...
Reading symbols from java...(no debugging symbols found)...done.
Catchpoint 1 (syscall 'write' [1])
Starting program: /usr/bin/java -Xmx1G -version
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff7fd9700 (LWP 17480)]
[Switching to Thread 0x7ffff7fd9700 (LWP 17480)]

Thread 2 "java" hit Catchpoint 1 (call to syscall write), 0x00007ffff79b21cd in write () at ../sysdeps/unix/syscall-template.S:84
84	../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0  0x00007ffff79b21cd in write () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007ffff6a067e0 in ?? () from /usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so
#2  0x00007ffff6a058e3 in ?? () from /usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so
#3  0x00007ffff6628ee2 in ?? () from /usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so
#4  0x00007ffff6b5568e in ?? () from /usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so
#5  0x00007ffff66d8aa6 in JNI_CreateJavaVM () from /usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so
#6  0x00007ffff77966c4 in ?? () from /usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/jli/libjli.so
#7  0x00007ffff779ac2d in ?? () from /usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/jli/libjli.so
#8  0x00007ffff79a94a4 in start_thread (arg=0x7ffff7fd9700) at pthread_create.c:456
#9  0x00007ffff72d7d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
(gdb) up
#1  0x00007ffff6a067e0 in ?? () from /usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so
(gdb) disas
No function contains program counter for selected frame.

I spent quite a while playing with OpenJDK cli arguments and various memory limits on the grid. I really wasn't able to find any magic combination that allowed a JVM to startup with less than a 4G -mem setting (technically a -l h_vmem=4G argument to qsub). It seems that even when given lots of "extra" arguments like -XX:MaxRAM=<size>, -Xms<size>, -Xmx<size>, and -Xss<size> OpenJDK 11.0.2 still tries to allocate an amount of virtual memory that will trigger a crash with less than 4G allowed by the Son of Grid Engine access controller.

I did find some articles online that seemed to indicate that OpenJDK would probably be a bit smarter about how it allocates its initial heaps and edens if it detected that it was running under cgroup limits (for example in a Docker container). The core issue seems to be in part that the OpenJDK code is trying to be smarter than the operator and interrogating the system to decide how much ram and cpu are available globally on the host. In a shared environment these guesses end up being higher than the limits actually applied to the individual application at runtime permit.

To try and address the original questions:

  1. Is this a bug or the expected behavior?

I think there are potentially some bugs in play here, but they seem to be upstream in OpenJDK rather than in Son of Grid Engine or the configured Toolforge deployment of it. Specifically

  1. Is the latter example the best practice on toolforge these days (i.e. set crazy memory values)?

I hesitate to call it a "best practice", but for OpenJDK jobs it may be necessary to ask for a fairly large reservation. Certainly the -mem granted to a job will need to be significantly larger than any -Xmx value that is used. -Xmx sets the maximum heap size, but this is only a part of the memory allocated by the JVM at startup. Other memory includes the per-thread stack space, "native" memory (off-heap allocations), Eden size, shared class space, etc:

$ java -Xms32M -Xmx128M -Xmn2M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -version
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment (build 11.0.2+9-Debian-3bpo91)
OpenJDK 64-Bit Server VM (build 11.0.2+9-Debian-3bpo91, mixed mode, sharing)

Native Memory Tracking:

Total: reserved=1517235KB, committed=110115KB
-                 Java Heap (reserved=131072KB, committed=34816KB)
                            (mmap: reserved=131072KB, committed=34816KB)

-                     Class (reserved=1056864KB, committed=4576KB)
                            (classes #430)
                            (  instance classes #364, array classes #66)
                            (malloc=96KB #451)
                            (mmap: reserved=1056768KB, committed=4480KB)
                            (  Metadata:   )
                            (    reserved=8192KB, committed=4096KB)
                            (    used=136KB)
                            (    free=3960KB)
                            (    waste=0KB =0.00%)
                            (  Class space:)
                            (    reserved=1048576KB, committed=384KB)
                            (    used=3KB)
                            (    free=381KB)
                            (    waste=0KB =0.00%)

-                    Thread (reserved=15461KB, committed=597KB)
                            (thread #15)
                            (stack: reserved=15392KB, committed=528KB)
                            (malloc=52KB #84)
                            (arena=18KB #28)

-                      Code (reserved=247725KB, committed=7585KB)
                            (malloc=37KB #372)
                            (mmap: reserved=247688KB, committed=7548KB)

-                        GC (reserved=46059KB, committed=42487KB)
                            (malloc=8407KB #506)
                            (mmap: reserved=37652KB, committed=34080KB)

-                  Compiler (reserved=135KB, committed=135KB)
                            (malloc=4KB #58)
                            (arena=131KB #5)

-                  Internal (reserved=543KB, committed=543KB)
                            (malloc=503KB #1089)
                            (mmap: reserved=40KB, committed=40KB)

-                    Symbol (reserved=1038KB, committed=1038KB)
                            (malloc=678KB #11)
                            (arena=360KB #1)

-    Native Memory Tracking (reserved=74KB, committed=74KB)
                            (malloc=5KB #66)
                            (tracking overhead=69KB)

-        Shared class space (reserved=17080KB, committed=17080KB)
                            (mmap: reserved=17080KB, committed=17080KB)

-               Arena Chunk (reserved=1104KB, committed=1104KB)
                            (malloc=1104KB)

-                   Logging (reserved=4KB, committed=4KB)
                            (malloc=4KB #185)

-                 Arguments (reserved=17KB, committed=17KB)
                            (malloc=17KB #465)

-                    Module (reserved=58KB, committed=58KB)
                            (malloc=58KB #1024)
jberkel added a subscriber: jberkel.Apr 9 2019, 9:21 PM

Sorry if this isn't directly related to the reported bug, but is there any way to find out why a process got sigkilled on the grid? I have a Java process getting terminated without traces of hserr*.log, and I suspect it is related to some native libraries allocating too much off-heap memory. I'd like to confirm that it's due to the OOM killer (or perhaps quota related?)

bd808 added a comment.Jul 1 2019, 8:53 PM

Sorry if this isn't directly related to the reported bug, but is there any way to find out why a process got sigkilled on the grid? I have a Java process getting terminated without traces of hserr*.log, and I suspect it is related to some native libraries allocating too much off-heap memory. I'd like to confirm that it's due to the OOM killer (or perhaps quota related?)

https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid#Returning_the_status_of_a_particular_job

thanks, with qacct -j jobid I get:

...
failed       37  : qmaster enforced h_rt, h_cpu, or h_vmem limit
maxvmem      16.558GB
category     -q task -l h_vmem=16777216k

maxvmem is greater than h_vmem so the process was probably killed because of that. Strange thing is that the JVM is limited to -Xmx8GB, so this must be off-heap memory.