Increasing the sysctl "kernel.pid_max" to 4M would probably work as a temporary hack, and would help to mitigate similar errors in other programs. Ideally puppet would check /proc/$pid/cmdline or something to make sure it is killing the right thing.

Unsure why this didn't trigger sooner since the orphan pid file timestamp was months old and mysqld had been running for three days.

The number in the PID file did not correspond to mysqld's PID, or else Puppet would have killed it sooner. It must have corresponded to the TID of one of mysqld's threads at the time Puppet ran.

Puppet checked the service status by running /etc/init.d/puppet status, which called /lib/lsb/init-function's status_of_proc, which confirmed that the service was running by ensuring that kill -0 succeeded when invoked with the number from the PID file. It then killed the process using start-stop-daemon, which called kill(2) with the same number as its argument.

[[ http://manpages.courier-mta.org/htmlman1/kill.1.html#kill-1_sect3 | Both kill(2) and kill(1) can be called with a TID instead of a PID ]]. When start-stop-daemon attempted to send a TERM signal to the TID in Puppet's pidfile, the signal was sent to the entire thread group, causing mysqld to shut down.

I think this can be prevented now by changing /etc/init.d/puppet to use --name and possibly --user in addition to --pidfile with start-stop-daemon.

In a future where we switch to Debian and thus systemd AFAIK this would not happen with a native service (it uses a cgroup per service for tracking and killing it) instead of a shell init script.

But to completely prevent anything like it we would need to change this for all services running on one system.