Page MenuHomePhabricator

PID not expanded in heap dumps
Closed, ResolvedPublic

Description

Earlier today, a heap dump was generated on restbase1007.eqiad.wmnet, and the %p format specifier, which was supposed to be expanded to $PID... wasn't.

$ sudo ls -lh /var/lib/cassandra/
total 3.1G
drwxr-x---   2 cassandra cassandra  28K Oct 27 19:02 commitlog
drwxr-x--- 161 cassandra cassandra  16K Oct 23 09:14 data
-rw-------   1 cassandra cassandra 3.1G Oct 27 11:14 java_%p.hprof
drwxr-x---   2 cassandra cassandra  20K Oct 27 19:24 saved_caches

Event Timeline

Eevans raised the priority of this task from to Needs Triage.
Eevans updated the task description. (Show Details)
Eevans added projects: SRE, RESTBase-Cassandra.
Eevans added subscribers: Eevans, fgiunchedi.

Change 249419 had a related patch set uploaded (by Filippo Giunchedi):
cassandra: fix HeapDumpPath and ErrorFile settings

https://gerrit.wikimedia.org/r/249419

Change 249419 merged by Filippo Giunchedi:
cassandra: fix HeapDumpPath and ErrorFile settings

https://gerrit.wikimedia.org/r/249419

chasemp set Security to None.

change has been merged, pending rolling restart

fgiunchedi closed this task as Resolved.EditedDec 4 2015, 12:40 PM

completed, cassandra has been rolled restarted following openjdk security advisory

cassand+ 3001 457 35.5 42478052 23376880 ? SLl Dec02 13624:55 java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms12g -Xmx12g -XX:+HeapDum OnOutOfMemoryError -XX:HeapDumpPath=/var/lib/cassandra/ -XX:ErrorFile=/var/lib/cassandra//hs_err_pid%p.log -Xss256k