Page MenuHomePhabricator

Run jstack / jmap / etc... with PrivateTmp=true
Closed, ResolvedPublic

Description

Our elasticsearch systemd unit sets PrivateTmp=true, which is overall a good thing. But this prevents jstack / jmap / etc... from connecting to the JVM as they expect a socket in the temp directory.

The easy (but less secure) workaround is to disable PrivateTmp. A better solution should be to wrap jstack / jmap / etc... and give them the access to that same temp dir. Not sure if there is something similar to JoinsNamespaceOf that we could use in this context.

Event Timeline

Gehel created this task.Aug 20 2019, 8:27 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 20 2019, 8:27 AM

If you just want to run a command manually that's already possible using nsenter, e.g. on my Buster workstation I have systemd-timesynd running with PrivateTmp:

jmm@ziegler:~$ systemctl status systemd-timesyncd | grep "Main PID"
 Main PID: 745 (systemd-timesyn)
jmm@ziegler:~$ sudo nsenter -t 745 -m
root@ziegler:/# ls /tmp/

I'm not sure how exactly jstack is being invoked, but something like "sudo nsenter -t $ELASTICPID -m $JSTACK_CMD " should do the trick. If you leave out the command name, it opens a shell, which is probably even more useful.

Gehel added a comment.Aug 20 2019, 8:59 AM

Nice! I did not know that one.

So as an example, jstack can be called with:

gehel@elastic2050:~$ sudo systemctl status elasticsearch_6@production-search-codfw.service | grep "Main PID"
 Main PID: 1110 (java)
gehel@elastic2050:~$ sudo nsenter -t 1110 -m sudo -u elasticsearch jstack 1110
2019-08-20 08:58:53
Full thread dump OpenJDK 64-Bit Server VM (25.222-b10 mixed mode):

"Attach Listener" #9275 daemon prio=9 os_prio=0 tid=0x00007f9c48002000 nid=0x31f81 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"elasticsearch[elastic2050-production-search-codfw][fetch_shard_store][T#15]" #7796 daemon prio=5 os_prio=0 tid=0x00007f9b58025000 nid=0x2a438 waiting on condition [0x00007f8a64a5e000]
   java.lang.Thread.State: WAITING (parking)
[...]
Joe added a comment.Aug 20 2019, 9:25 AM

You can automate the process with a simple alias:

systemctl-jstack() {
if [ -z $1 ] {
   echo "Please provide a service name."
   echo "Usage: systemctl-jstack  elasticsearch_6@production-search-codfw.service
  exit 1
}
PID=$(systemctl show $1  -p MainPID | sed -e 's/^MainPID=//')
USER=$(systemctl show $1 -p User | sed - e 's/^User=//')
sudo nsenter -t $PID -m sudo -u $USER jstack $PID
}

and so on for other tools.

BTW, nsenter can also be used for other namespace types (e.g. network namespaces), one can print all namespaces currently in use via "sudo lsns" (or without lsns it only prints the namespaces created under the current user session).

Gehel claimed this task.Aug 20 2019, 5:27 PM
Joe triaged this task as Normal priority.Aug 21 2019, 6:12 AM
debt closed this task as Resolved.Sep 5 2019, 6:26 PM