Going to https://phabricator.wmcloud.org/ says
Warning: mysqli::__construct(): (HY000/1698): Access denied for user 'app_user'@'localhost' in /srv/deployment/phabricator/deployment-cache/revs/c1262ac47371502b1946696930feb211b9a73f9d/phabricator/support/preamble.php on line 31
Here is the full log:
[Mon Mar 03 08:56:08.724529 2025] [proxy_fcgi:error] [pid 2950329:tid 2950521] [client 172.16.5.238:48680] AH01071: Got error 'PHP message: PHP Warning: mysqli::__construct(): (HY000/1698): Access denied for user 'app_user'@'localhost' in /srv/deployment/phabricator/deployment-cache/revs/c1262ac47371502b1946696930feb211b9a73f9d/phabricator/support/preamble.php on line 31'
[Mon Mar 03 08:56:08.724977 2025] [proxy_fcgi:error] [pid 2950329:tid 2950521] [client 172.16.5.238:48680] AH01071: Got error 'PHP message: redirector.php: Connect Error (1698) Access denied for user 'app_user'@'localhost'PHP message: [2025-03-03 08:56:08] EXCEPTION: (RuntimeException) Cannot modify header information - headers already sent by (output started at /srv/deployment/phabricator/deployment-cache/revs/c1262ac47371502b1946696930feb211b9a73f9d/phabricator/support/preamble.php:31) at [<arcanist>/src/error/PhutilErrorHandler.php:273]PHP message: arcanist(), ava(), phorge(), translations(), wmf-ext-misc()PHP message: #0 <#2> PhutilErrorHandler::handleError(integer, string, string, integer, array) called at [<arcanist>/src/error/PhutilErrorHandler.php:273]PHP message: #1 <#2> header(string) called at [<phorge>/src/aphront/sink/AphrontPHPHTTPSink.php:14]PHP message: #2 <#2> AphrontPHPHTTPSink::emitHTTPStatus(integer, string) called at [<phorge>/src/aphront/sink/AphrontHTTPSink.php:40]PHP message: #3 <#2> AphrontHTTPSink::writeHTTPStatus(integer, string) called at [<phorge>/src/aphront/sink/AphrontHTTPSink.php:131]PHP message: #4 <#2> AphrontHTTPSink::writeResponse(PhabricatorConfigResponse) called at [<phorge>/src/aphront/configuration/AphrontApplicationConfiguration.php:380]PHP message: #5 <#2> AphrontApplicationConfiguration::writeResponse(AphrontPHPHTTPSink, PhabricatorConfigResponse) called at [<phorge>/src/aphront/configuration/AphrontApplicationConfiguration.php:130]PHP message: #6 <#2> AphrontApplicationConfiguration::runHTTPRequest(AphrontPHPHTTPSink) called at [<phorge>/webroot/index.php:35]PHP message: #7 phlog(RuntimeException) called at [<phorge>/src/aphront/response/AphrontUnhandledExceptionResponse.php:32]PHP message: #8 AphrontUnhandledExceptionResponse::setException(RuntimeException) called at [<phorge>/webroot/index.php:46]PHP message: [2025-03-03 08:56:08] EXCEPTION: (RuntimeException) Cannot modify header information - headers already sent by (output started at /srv/deployment/phabricator/deployment-cache/revs/c1262ac47371502b1946696930feb211b9a73f9d/phabricator/support/preamble.php:31) at [<arcanist>/src/error/PhutilErrorHandler.php:273]PHP message: arcanist(), ava(), phorge(), translations(), wmf-ext-misc()PHP message: #0 PhutilErrorHandler::handleError(integer, string, string, integer, array)PHP message: #1 header(string, boolean, integer) called at [<phorge>/support/startup/PhabricatorStartup.php:369]PHP message: #2 PhabricatorStartup::didFatal(string, string) called at [<phorge>/support/startup/PhabricatorStartup.php:333]PHP message: #3 PhabricatorStartup::didEncounterFatalException(string, RuntimeException, boolean) called at [<phorge>/webroot/index.php:75]'
==> phabricator_access.log <==
@cee: {"client.ip":"172.16.5.238", "ecs.version":"1.7.0", "event.category":["network", "web"], "event.dataset":"apache.access", "event.duration":"20254", "event.kind":"event", "event.outcome":"unknown", "event.type":["access", "connection"], "http.request.headers.accept_language":"en-US,en;q=0.5", "http.request.headers.x_analytics":"-", "http.request.headers.x_client_ip":"-", "http.request.headers.x_forwarded_for":"-", "http.request.headers.x_request_id":"-", "http.request.method":"GET", "http.request.referrer":"-", "http.response.bytes":"220", "http.response.headers.content_type":"text/html", "http.response.headers.user":"-", "http.response.status_code":"200", "labels.handler":"proxy:unix:/run/php/fpm-www.sock|fcgi://localhost/", "labels.httpd_server_name":"phabricator.wmcloud.org", "server.ip":"172.16.4.197", "service.type":"apache2", "source.ip":"172.16.5.238", "timestamp":"2025-03-03T08:56:08", "url.domain":"phabricator.wmcloud.org", "url.full":"http://phabricator.wmcloud.org/?__path__=%2f", "url.path":"/", "url.query":"?__path__=%2f", "user.name":"-", "user_agent.original":"Mozilla/5.0 (X11; Linux x86_64; rv:135.0) Gecko/20100101 Firefox/135.0"}From what I saw on the machine, the local MariaDB instance has no user nor databases that relates to phabricator:
MariaDB [(none)]> select user,host from mysql.user; +-------------+-----------+ | User | Host | +-------------+-----------+ | mariadb.sys | localhost | | mysql | localhost | | root | localhost | +-------------+-----------+ 3 rows in set (0.007 sec) MariaDB [(none)]> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | +--------------------+ 3 rows in set (0.005 sec)
Mentioned in SAL (#wikimedia-cloud) [2025-03-03T19:44:58Z] <mutante> editing /etc/mysql/mariadb.conf.d/50-server.cnf to point mariab datadir to /srv/sqldata ; systemctl restart mariadb --> fixes test Phabricator (T387619)
What happened is mariadb got restarted on March 1st and the datadir was configured as the default /var/lib/mysql when the actual data dir is /srv/sqldata.
(Previously we had related issues when a non-default data dir was used.)
I edited /etc/mysql/mariadb.conf.d/50-server.cnf to change the datadir line to /srv/sqldata and then systemctl restart mariadb and the Phab UI is back.
Running puppet did NOT change the config file...
Still need to debug why puppet does not manage the file.. it used to.. maybe this is caused by a change to the mariadb.conf.db snippets in a previous version upgrade?
The phabricator-bullseye instance uses (as it should) the regular role::phabricator puppet class like in production.
This includes profile::phabricator::main and in there we have this code to set up the database server IF IN CLOUD.
# in cloud, use a local db server
if $::realm == 'labs' {
class { 'profile::mariadb::generic_server':
datadir => $database_datadir,
}
}Such "realm checks" are somewhat discouraged in some places but at the same time it's much better than not using the normal production puppet role and having some special "cloud only" role as some other services do.
In Hiera data in Horizon we have profile::phabricator::main::database_datadir: /srv/sqldata.
If this Hiera key was NOT set then the default value for the datadir would be /var/lib/mysql, set in profile::phabricator::main.
So what this all means is .. datadir config was only written once when the instance originally got the role. Back then was set to default dir, incorrectly.
This did not cause an issue until much later the mariadb server was restarted (for unrelated reasons).
datadir path in config and in puppet code now matches. puppet does not revert the change. service was restarted.
basically one forgotten restart as part of T329571 in the past, afaict.
this should be all