@Lucas_Werkmeister_WMDE I merged the fix. Feel free to attempt re-deploying.

I believe this should be good to go. @ARamirez_WMF Could you kindly test and confirm?

Adding @KFrancis to verify NDA.

Created silence ID 725be722-c9d2-47ff-8248-52c8906f8001 for 3 days.

Thank you. I'll resolve this.

o/ Following up if there's been any improvements on this @AntiCompositeNumber @JJMC89 We are now currently on version 6.5.22.

@Dzahn Oooh. That was indeed the problem. It works fine now. :)

I had a chat with @dcaro and re-created the phabricator.wmcloud.org proxy and pointed it to the new host. It does looks like the proxy is doing what it's supposed to be doing as traffic is visible on tcpdump and apache log file (I had checked the incorrect file initially i.e. access.log instead of phabricator_access.log). It seems that the initially configured external URI cannot be changed or is hard-coded into the config and not changing even after altering phabricator_domain and running puppet.

Yeah, I think the error might be coming from the proxy and not the host. The apache log files did not show anything at the time.

@Dzahn Yes, after changing Hiera and running puppet, that Apache config changes to the value specified in that domain.

I re-created the phabricator.wmcloud.org proxy to point to the new instance but still doesn't work despite changing phabricator_domain in hiera, running puppet and restarting apache on the host. If this doesn't work I'll just go back to using phabricator-new as the proxy name.

Actually, the dump might have caused file integrity issues on the new instance. It looks like when you upload a new avatar on the host it renders as expected from the usercontent domain.

There is one issue though. Avatars are not loading on the new instance. On Firefox, it says A resource is blocked by OpaqueResponseBlocking, please check browser console for details and NS_BINDING_ABORTED. Then on Chrome it says (failed)net::ERR_BLOCKED_BY_ORB. So some kind of CORS issue since these are being served from a different domain.

I also dumped the the database from phabricator-bullseye to the new host and looks okay afaict. We should be good to go with getting rid of the old host.

I got this from @brennen https://www.mediawiki.org/wiki/Phabricator/Local_Dev_Environment#Set_up_MySQL/MariaDB_database_privileges

The scap deploy was successful after jumping through some hurdles specifically setting up keyholder and checking out the wmf/stable branch. Currently troubleshooting a connection issue on MariaDB.

I've setup phabricato-trixie and the last merge with the php8.4 template resulted in a mostly successful puppet run. The current issue is that phab isn't deployed yet so working on that next.

@Marostegui Yes, those can be removed.

I terminated this host so should not be an issue now. I'll mark this ticket as resolved.

Created silence ID a0c0746a-28ce-4310-a2e2-f4e5aaa980cb for 7 days

I disabled puppet on phab2002 and silenced it to prevent alerting noise. phab2003 has been running as the passive server for almost 24 hours now and no issues so far. I'll extend the silence till next week and begin decommissioning on Monday.

@Marostegui Thank you for clarifying this.

Since we are unblocked on the deployment, I'll proceed with promoting phab2003 as the passive_server and begin decom of the old host.

Nice, thank you @Marostegui. I think the grants now match. I tried the above on phab2002 and it works and the grants are similar.

@Marostegui Got it. Yeah, I also tried going through the proxy and had no issues. I think the issue is the config then: cc: @brennen @Dzahn

root@phab2003:~# mysql -h m3-slave.codfw.wmnet -u phabricatorphd -P 3323 -p
Enter password: 
ERROR 1045 (28000): Access denied for user 'phabricatorphd'@'10.192.27.12' (using password: YES)

@Marostegui We ran into an issue with one of the deployment scripts run by scap related to database permissions. Could you configure the same permissions for the phabricatorphd user from the same host?

@Hyphen Did you manage to file an issue upstream?

Looks similar to T369443

I've applied the production role to phab2003 and puppet runs fine and the phd service is disabled as expected. There is a strange issue with running a scap deploy related to database grants which doesn't occur when deploying to phab2002 but I'm guessing this is probably influenced by the phabricator_passive_server hiera value which is still set to phab2002. I'll proceed with altering this to point to the new host.

This is currently blocked since the role phabricator::migration is missing the config file and preventing scap from deploying. I'll try the production role.

I've copied the error output here.

As far as I can tell, T425959 resolved this issue so I'll close the ticket.

Both host are up-to-date:

(1) vrts1003.eqiad.wmnet                                                                                                                                                                      
----- OUTPUT for command #1: 'ls -l /opt' -----                                                                                                                                               
total 403476                                                                                                                                                                                  
lrwxrwxrwx 1 root root           17 May 13 17:04 otrs -> /opt/znuny-6.5.20

There is an issue running scap from the deploy host in devtools for the phabricator-bookworm instance. The issue is related to the publickey. cc: @brennen @Dzahn

The new hardware has a disk controller not supported on bullseye so we can only use bookworm or trixie on this host. Thanks @elukey for assisting with this. cc: @Dzahn @brennen

@Dzahn The quota looks healthy. I think it would be enough for this test unless we need floating IPs.

I followed up with Znuny. The fix will be in the next release hopefully this week.

Thanks for reporting @Xaosflux This is what seems to be the issue.

Apr 07 14:43:57 vrts1003 OTRS-CGI-10[4134825]: [Error][Kernel::Modules::AgentTicketQueue::Run][Line:271]: Invalid Filter: 'Unlocked'!
Apr 07 14:44:33 vrts1003 OTRS-CGI-10[4134825]: [Error][Kernel::Modules::AgentTicketQueue::Run][Line:271]: Invalid Filter: 'Unlocked'!

For the average number of requests for each planet sub-domain in the last 30 days: https://w.wiki/HywH

Expected due to T419712.

Test instance is back up. I did the following as recommended here https://gitlab.com/gitlab-org/gitlab/-/issues/590848#note_3100211428

aokoth@gitlab-1002:~$  sudo gitlab-psql -t -c "UPDATE pool_repositories
SET organization_id = (
  SELECT organization_id
  FROM projects
  WHERE projects.id = pool_repositories.source_project_id
)
WHERE organization_id IS NULL;
"

then:

sudo gitlab-ctl upgrade
sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart

Started this on the test instance but it failed due to a migration error:

Manually deleted the timer and unit files from the host. So hopefully this does not fire again.

Resolving this since we already followed steps shared by Znuny for resolution.

@elukey Ack. Thank you.

Thanks @elukey

@Johannnes89 Np. Do we need to amend anything or this can be resolved?

@Ottomata / @Milimetric / @Ahoelzl Kindly approve.

@Johannnes89 Could you clear your cache and cookies and retry?