Phabricator on phab1001 started to return 503s today from 16:50 UTC (icinga alarming time), this is the related stacktrace:
[Tue Feb 06 16:56:52.502222 2018] [:error] [pid 28700] [client 94.174.160.93:26097] [2018-02-06 16:56:52] EXCEPTION: (Exception) Failed to `proc_open()`: proc_open() expects parameter 2 to be array, unknown given at [<phutil>/src/future/exec/ExecFuture.php:678] [Tue Feb 06 16:56:52.502884 2018] [:error] [pid 28700] [client 94.174.160.93:26097] arcanist(), phabricator(), phabricator-translations(), phutil(), security(), sprint(), wmf-ext-misc() [Tue Feb 06 16:56:52.502899 2018] [:error] [pid 28700] [client 94.174.160.93:26097] #0 <#2> ExecFuture::isReady() called at [<phutil>/src/future/Future.php:37] [Tue Feb 06 16:56:52.502904 2018] [:error] [pid 28700] [client 94.174.160.93:26097] #1 <#2> Future::resolve() called at [<phutil>/src/future/exec/execx.php:37] [Tue Feb 06 16:56:52.502909 2018] [:error] [pid 28700] [client 94.174.160.93:26097] #2 <#2> exec_manual(string, string) called at [<phutil>/src/filesystem/Filesystem.php:993] [Tue Feb 06 16:56:52.502913 2018] [:error] [pid 28700] [client 94.174.160.93:26097] #3 <#2> Filesystem::resolveBinary(string) called at [<phutil>/src/filesystem/Filesystem.php:962] [Tue Feb 06 16:56:52.502918 2018] [:error] [pid 28700] [client 94.174.160.93:26097] #4 <#2> Filesystem::binaryExists(string) called at [<phabricator>/src/applications/config/check/PhabricatorPygmentSetupCheck.php:13] [Tue Feb 06 16:56:52.502923 2018] [:error] [pid 28700] [client 94.174.160.93:26097] #5 <#2> PhabricatorPygmentSetupCheck::executeChecks() called at [<phabricator>/src/applications/config/check/PhabricatorSetupCheck.php:63] [Tue Feb 06 16:56:52.502928 2018] [:error] [pid 28700] [client 94.174.160.93:26097] #6 <#2> PhabricatorSetupCheck::runSetupChecks() called at [<phabricator>/src/applications/config/check/PhabricatorSetupCheck.php:258] [Tue Feb 06 16:56:52.502932 2018] [:error] [pid 28700] [client 94.174.160.93:26097] #7 <#2> PhabricatorSetupCheck::runNormalChecks() called at [<phabricator>/src/applications/config/engine/PhabricatorSetupEngine.php:26] [Tue Feb 06 16:56:52.502937 2018] [:error] [pid 28700] [client 94.174.160.93:26097] #8 <#2> PhabricatorSetupEngine::execute() called at [<phabricator>/src/applications/config/check/PhabricatorSetupCheck.php:194] [Tue Feb 06 16:56:52.502941 2018] [:error] [pid 28700] [client 94.174.160.93:26097] #9 <#2> PhabricatorSetupCheck::willProcessRequest() called at [<phabricator>/src/aphront/configuration/AphrontApplicationConfiguration.php:137] [Tue Feb 06 16:56:52.502946 2018] [:error] [pid 28700] [client 94.174.160.93:26097] #10 <#2> AphrontApplicationConfiguration::runHTTPRequest(AphrontPHPHTTPSink) called at [<phabricator>/webroot/index.php:17] [Tue Feb 06 16:56:52.502950 2018] [:error] [pid 28700] [client 94.174.160.93:26097] #11 phlog(Exception) called at [<phabricator>/src/aphront/response/AphrontUnhandledExceptionResponse.php:20] [Tue Feb 06 16:56:52.502955 2018] [:error] [pid 28700] [client 94.174.160.93:26097] #12 AphrontUnhandledExceptionResponse::setException(Exception) called at [<phabricator>/webroot/index.php:21]
The first occurrence of this error in the phabricator apache error log is at:
[Tue Feb 06 16:14:27.176766 2018] [:error] [pid 18114] [client 220.181.125.124:10109] [2018-02-06 16:14:27] EXCEPTION: (Exception) Failed to `proc_open()`: proc_open() expects parameter 2 to be array, integer given {>} (Exception) Failed to `proc_open()`: proc_open() expects parameter 2 to be array, integer given at [<phutil>/src/future/exec/ExecFuture.php:678]
This issue was "fixed" with an apache2 restart.