May need a ./bin/phd restart

In T397626#10958631, @Aklapper wrote:

May need a ./bin/phd restart

If that's what's responsible for populating activity feeds, then that would make sense; as one of the Unresolved Setup Issues is "Daemons Are Not Running":

The daemons are not running, background processing (including sending email, rebuilding search indexes, importing commits, cleaning up old data, and running builds) can not be performed.

It suggests running the command ./bin/phd start.

The phd service is masked on the instance (for some reason tbd). It can't be started unless it gets unmasked.

root@phabricator-bullseye:/home/dzahn# systemctl status phd
● phd.service
     Loaded: masked (Reason: Unit phd.service is masked.)

the reason for being masked is that it's masked.. which is of course very helpful.

    if $::fqdn == $active_server {
...
        $phd_service_ensure = 'running'
...
    } else {
...
        systemd::mask { 'phd.service': }

^ This is because phd can't run on the inactive server in production because it fails if it can't write to the database.

But the $active_server is set in cloud VPS devtools project context:

phabricator_active_server: phabricator-bullseye

And the FQDN is phabricator-bullseye.devtools.eqiad1.wikimedia.cloud still.

This makes me think maybe the host name lookup changed in cloud VPS at some point, making this not match anymore.

Change #1165600 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] cloudvps/devtools: fix FQDN for Phabricator test instance

https://gerrit.wikimedia.org/r/1165600

Change #1165600 merged by Dzahn:

[operations/puppet@production] cloudvps/devtools: fix FQDN for Phabricator test instance

https://gerrit.wikimedia.org/r/1165600

@A_smart_kitten The phd service is now running on the instance. Did this fix it?

It looks like it did, ty!

Interesting to know — it seems like the Phabricator activity feed seemingly lists events as having happened at the time the daemon processes them, rather than necessarily at the time they actually happened.
(ref. https://phabricator.wmcloud.org/feed/query/all/ saying that everything that's occurred so far on the test instance happened between 21:05 & 21:10 UTC today)

Thanks for the fix @Dzahn :D

I think the instance('s database) was wiped/reset at some point (xref T265360#10598095). I'd guess that it's been broken at least since then; but I wouldn't know about before that