for the deployment server an issue is also T257317 (also see latest comments there where Raine runs into it in production)

I confirmed with RelEng that gerrit-bullseye can be safely deleted.

Thanks! Done. The gerrit instance has been deleted.

I re-created the phabricator.wmcloud.org proxy to point to the new instance but still doesn't work despite changing phabricator_domain in hiera, running puppet and restarting apache on the host. If this doesn't work I'll just go back to using phabricator-new as the proxy name.

Can you confirm that the Hiera change actually arrives in the Phabricator config itself in the file system?

What error do you get?

@Dzahn Yes, after changing Hiera and running puppet, that Apache config changes to the value specified in that domain.

The error was Site Not Found iirc.

Anyways, I'll blame caching or something since I switched backed to phabricator-new and it works fine. I'll leave it as is for now.

I've shutdown phabricator-bullseye and will delete it.

If the error comes from the haproxy that wmcs operates we should report this as a bug to them.

The error sounds like it does. But maybe we can verify it's not from the local apache via log files.

If the URL changes we need to inform the users somehow.

Yeah, I think the error might be coming from the proxy and not the host. The apache log files did not show anything at the time.

Maybe it's an issue with re-using proxy names. I'll reach out to WMCS team.

I might have ran into something similar before where if you delete a proxy and recreate it it took a while or was cached. But not sure anymore about the details. There was also something like this with instance names, if you delete one and then try to recreate it again under the exact same name.

Thanks for reaching out.

I had a chat with @dcaro and re-created the phabricator.wmcloud.org proxy and pointed it to the new host. It does looks like the proxy is doing what it's supposed to be doing as traffic is visible on tcpdump and apache log file (I had checked the incorrect file initially i.e. access.log instead of phabricator_access.log). It seems that the initially configured external URI cannot be changed or is hard-coded into the config and not changing even after altering phabricator_domain and running puppet.

I did grep -r "phabricator-new" * . in /srv and I found a setting still set to the initial value used for phabricator_domain.

@Arnoldokoth I think you need a fresh scap deployment. There is some config in puppet and some in deployment repo and then scap looks at both and combines them.

@Dzahn Oooh. That was indeed the problem. It works fine now. :)

I can now delete the old host.

:) yay, great to hear