I can reproduce locally using upstream version of phabricator not wmf.

By that i mean i can reproduce on wmf and upstream too.

It basically means that the bin/phd stop command does nothing and thinking it has could lead to more daemons being added with bin/phd start.

I think we need to change the systemd unit file to Type=forking
as in https://secure.phabricator.com/T4181#204679

@mmodell it is already type=forking also i tested locally without any systemd files as a mac does not support systemd. SO i ran manually bin/phd

though on my local machine it seems to take like 10+ for it shut down.

Investigating it further and have found when you try to stop phd daemons it grows the logs by as much as 99% in 20-30mins.

See https://phabricator.wikimedia.org/P5049

If this result happened on a system not even running systemd, AND you tested the systemd unit file on labs before we merged this.. ist it even relevant?

How is this even possible since you tested starting and stopping before we converted it and i also did it myself after deploying.

I tested it but i didn't check the daemon status on pabricator's ui. I used the systemd status to tell me if it stopped it or started it.

I upgraded phabricator test instance yesturday to match prod's update

I have an idea on what's causing that error in the logs, this https://github.com/wikimedia/phabricator-libphutil/commit/be546154255ccfcd5db10eabdabcadf1d8ea73b7

I see this in the log now

[10-Mar-2017 17:58:14 UTC] [2017-03-10 17:58:14] ERROR 2: posix_isatty(): 2 is not a valid stream resource at [/srv/deployment/phabricator/deployment/libphutil/src/console/PhutilConsoleFormatter.php:39]
[10-Mar-2017 17:58:14 UTC] arcanist(), phabricator(), phutil(), security(), sprint(), wmf-ext-misc(head=wmf/stable, ref.wmf/stable=ef53ff4b0a99)
[10-Mar-2017 17:58:14 UTC] #0 posix_isatty(unknown type) called at [<phutil>/src/console/PhutilConsoleFormatter.php:39]
[10-Mar-2017 17:58:14 UTC] #1 PhutilConsoleFormatter::getDisableANSI() called at [<phutil>/src/console/PhutilConsoleFormatter.php:75]
[10-Mar-2017 17:58:14 UTC] #2 PhutilConsoleFormatter::interpretFormat(string) called at [<phutil>/src/console/PhutilConsoleFormatter.php:50]
[10-Mar-2017 17:58:14 UTC] #3 PhutilConsoleFormatter::formatString(string, string, integer, string, string)
[10-Mar-2017 17:58:14 UTC] #4 call_user_func_array(array, array) called at [<phutil>/src/console/format.php:7]
[10-Mar-2017 17:58:14 UTC] #5 phutil_console_format(string, string, integer, string, string)
[10-Mar-2017 17:58:14 UTC] #6 call_user_func_array(string, array) called at [<phutil>/src/console/PhutilConsoleServer.php:154]
[10-Mar-2017 17:58:14 UTC] #7 PhutilConsoleServer::writeText(unknown type, array) called at [<phutil>/src/console/PhutilConsoleServer.php:39]
[10-Mar-2017 17:58:14 UTC] #8 PhutilConsoleServer::handleMessage(PhutilConsoleMessage) called at [<phutil>/src/console/PhutilConsole.php:204]
[10-Mar-2017 17:58:14 UTC] #9 PhutilConsole::writeMessage(PhutilConsoleMessage) called at [<phutil>/src/console/PhutilConsole.php:187]
[10-Mar-2017 17:58:14 UTC] #10 PhutilConsole::writeTextMessage(string, array) called at [<phutil>/src/console/PhutilConsole.php:153]
[10-Mar-2017 17:58:14 UTC] #11 PhutilConsole::writeLog(string, string, integer, string, string) called at [<phutil>/src/serviceprofiler/PhutilServiceProfiler.php:175]
[10-Mar-2017 17:58:14 UTC] #12 PhutilServiceProfiler::echoListener(string, integer, array)
[10-Mar-2017 17:58:14 UTC] #13 call_user_func(array, string, integer, array) called at [<phutil>/src/serviceprofiler/PhutilServiceProfiler.php:33]
[10-Mar-2017 17:58:14 UTC] #14 PhutilServiceProfiler::beginServiceCall(array) called at [<phutil>/src/events/PhutilEventEngine.php:55]
[10-Mar-2017 17:58:14 UTC] #15 PhutilEventEngine::dispatchEvent(PhutilEvent) called at [<phutil>/src/daemon/PhutilDaemonHandle.php:297]
[10-Mar-2017 17:58:14 UTC] #16 PhutilDaemonHandle::dispatchEvent(string, array) called at [<phutil>/src/daemon/PhutilDaemonHandle.php:98]
[10-Mar-2017 17:58:14 UTC] #17 PhutilDaemonHandle::didLaunch() called at [<phutil>/src/daemon/PhutilDaemonPool.php:100]
[10-Mar-2017 17:58:14 UTC] #18 PhutilDaemonPool::newDaemon() called at [<phutil>/src/daemon/PhutilDaemonPool.php:287]
[10-Mar-2017 17:58:14 UTC] #19 PhutilDaemonPool::updateAutoscale() called at [<phutil>/src/daemon/PhutilDaemonPool.php:192]
[10-Mar-2017 17:58:14 UTC] #20 PhutilDaemonPool::updatePool() called at [<phutil>/src/daemon/PhutilDaemonOverseer.php:177]
[10-Mar-2017 17:58:14 UTC] #21 PhutilDaemonOverseer::run() called at [<phabricator>/scripts/daemon/launch_daemon.php:24]
[10-Mar-2017 17:58:14 UTC] [2017-03-10 17:58:14] ERROR 2: posix_isatty(): expects argument 1 to be a valid stream resource at [/srv/deployment/phabricator/deployment/libphutil/src/console/PhutilConsoleFormatter.php:39]
[10-Mar-2017 17:58:14 UTC] arcanist(), phabricator(), phutil(), security(), sprint(), wmf-ext-misc(head=wmf/stable, ref.wmf/stable=ef53ff4b0a99)
[10-Mar-2017 17:58:14 UTC] #0 posix_isatty(unknown type) called at [<phutil>/src/console/PhutilConsoleFormatter.php:39]
[10-Mar-2017 17:58:14 UTC] #1 PhutilConsoleFormatter::getDisableANSI() called at [<phutil>/src/console/PhutilConsoleFormatter.php:75]
[10-Mar-2017 17:58:14 UTC] #2 PhutilConsoleFormatter::interpretFormat(string) called at [<phutil>/src/console/PhutilConsoleFormatter.php:50]
[10-Mar-2017 17:58:14 UTC] #3 PhutilConsoleFormatter::formatString(string, string, integer, string, string)
[10-Mar-2017 17:58:14 UTC] #4 call_user_func_array(array, array) called at [<phutil>/src/console/format.php:7]
[10-Mar-2017 17:58:14 UTC] #5 phutil_console_format(string, string, integer, string, string)
[10-Mar-2017 17:58:14 UTC] #6 call_user_func_array(string, array) called at [<phutil>/src/console/PhutilConsoleServer.php:154]
[10-Mar-2017 17:58:14 UTC] #7 PhutilConsoleServer::writeText(unknown type, array) called at [<phutil>/src/console/PhutilConsoleServer.php:39]
[10-Mar-2017 17:58:14 UTC] #8 PhutilConsoleServer::handleMessage(PhutilConsoleMessage) called at [<phutil>/src/console/PhutilConsole.php:204]
[10-Mar-2017 17:58:14 UTC] #9 PhutilConsole::writeMessage(PhutilConsoleMessage) called at [<phutil>/src/console/PhutilConsole.php:187]
[10-Mar-2017 17:58:14 UTC] #10 PhutilConsole::writeTextMessage(string, array) called at [<phutil>/src/console/PhutilConsole.php:153]
[10-Mar-2017 17:58:14 UTC] #11 PhutilConsole::writeLog(string, string, integer, string, string) called at [<phutil>/src/serviceprofiler/PhutilServiceProfiler.php:175]
[10-Mar-2017 17:58:14 UTC] #12 PhutilServiceProfiler::echoListener(string, integer, array)
[10-Mar-2017 17:58:14 UTC] #13 call_user_func(array, string, integer, array) called at [<phutil>/src/serviceprofiler/PhutilServiceProfiler.php:33]
[10-Mar-2017 17:58:14 UTC] #14 PhutilServiceProfiler::beginServiceCall(array) called at [<phutil>/src/events/PhutilEventEngine.php:55]
[10-Mar-2017 17:58:14 UTC] #15 PhutilEventEngine::dispatchEvent(PhutilEvent) called at [<phutil>/src/daemon/PhutilDaemonHandle.php:297]
[10-Mar-2017 17:58:14 UTC] #16 PhutilDaemonHandle::dispatchEvent(string, array) called at [<phutil>/src/daemon/PhutilDaemonHandle.php:98]
[10-Mar-2017 17:58:14 UTC] #17 PhutilDaemonHandle::didLaunch() called at [<phutil>/src/daemon/PhutilDaemonPool.php:100]
[10-Mar-2017 17:58:14 UTC] #18 PhutilDaemonPool::newDaemon() called at [<phutil>/src/daemon/PhutilDaemonPool.php:287]
[10-Mar-2017 17:58:14 UTC] #19 PhutilDaemonPool::updateAutoscale() called at [<phutil>/src/daemon/PhutilDaemonPool.php:192]
[10-Mar-2017 17:58:14 UTC] #20 PhutilDaemonPool::updatePool() called at [<phutil>/src/daemon/PhutilDaemonOverseer.php:177]
[10-Mar-2017 17:58:14 UTC] #21 PhutilDaemonOverseer::run() called at [<phabricator>/scripts/daemon/launch_daemon.php:24]

In T160167#3091390, @Paladox wrote:

I tested it but i didn't check the daemon status on pabricator's ui. I used the systemd status to tell me if it stopped it or started it.

So you are saying systemd does not say the truth when it claims phd is stopped but the UI tells the truth??

Yes it seems so, but it seems systemd only checks that the pid is deleted on stop which it is.

It's this https://secure.phabricator.com/rPHU0625e4d28b16cba8b0154e49dc31845559b4e704 commit that caused the problem as checking out the one previous to that one works with shutting down the daemons instantly.