Here is a doc with "the plan" (tm) how to move forward with the upgrades:

https://docs.google.com/document/d/1b-K6JUluT3c5I08mhcxsuPDTaismPDh01qnYBRQmMGA/edit?usp=sharing

@Arnoldokoth Could you comment on the quota situation (asking since you already checked in the meeting).

@Dzahn The quota looks healthy. I think it would be enough for this test unless we need floating IPs.

Thank you @Arnoldokoth ! Sounds good. I don't think we need floating IPs for this one.

instance phabricator-bullseye already exists. We created it 2 years and 4 months ago :)

role::phabricator is also already applied.

https://horizon.wikimedia.org/project/instances/c2b2b2a1-140b-402f-be7a-50749c78bfc7/

I guess it just needs a web proxy!

@brennen Wanna try to deploy latest to phabricator-bullseye.devtools.eqiad1.wikimedia.cloud?

Ignore the latest comments, mixed up with the even older "phab-prod-1001" again.

Creating new instance.

phabricator-bookworm:

https://horizon.wikimedia.org/project/instances/be09795d-5194-47fc-9983-aa17d75005af/

phabricator-trixie:

https://horizon.wikimedia.org/project/instances/adb1ad24-4427-41bf-973f-b8a15c951ec4/

Mentioned in SAL (#wikimedia-cloud) [2026-04-21T18:43:47Z] <mutante> created instances phabricator-bookworm and phabricator-trixie (T424055)

created a proxy to for the bookworm VM:

https://phabricator-bookworm.wmcloud.org

There is an issue running scap from the deploy host in devtools for the phabricator-bookworm instance. The issue is related to the publickey. cc: @brennen @Dzahn

P92512

I've copied the error output here.

Change #1287023 had a related patch set uploaded (by Thcipriani; author: Thcipriani):

[operations/puppet@production] Phabricator: require config before scap

https://gerrit.wikimedia.org/r/1287023

I left a a bunch of comments I should have put here on the ticket rather than the paste bin.

Let's transclude the paste here to continue.

--> comments at and after P92512#375819

{P92512}

Change #1287023 merged by Dzahn:

[operations/puppet@production] Phabricator: require config before scap

https://gerrit.wikimedia.org/r/1287023

Change #1287447 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator::migration: require config before scap

https://gerrit.wikimedia.org/r/1287447

Change #1287447 abandoned by Dzahn:

[operations/puppet@production] phabricator::migration: require config before scap

Reason:

not working as intended

https://gerrit.wikimedia.org/r/1287447

The bookworm instance has been deleted in T428069#12042082

Change #1308690 had a related patch set uploaded (by AOkoth; author: AOkoth):

[operations/puppet@production] hiera: add host specific values for new phab host

https://gerrit.wikimedia.org/r/1308690

Change #1308690 merged by AOkoth:

[operations/puppet@production] hiera: add host specific values for new phab host

https://gerrit.wikimedia.org/r/1308690

Change #1308695 had a related patch set uploaded (by AOkoth; author: AOkoth):

[operations/puppet@production] fpm: add templates for version 8.4

https://gerrit.wikimedia.org/r/1308695

Change #1308695 merged by AOkoth:

[operations/puppet@production] fpm: add templates for version 8.4

https://gerrit.wikimedia.org/r/1308695

I've setup phabricato-trixie and the last merge with the php8.4 template resulted in a mostly successful puppet run. The current issue is that phab isn't deployed yet so working on that next.

The scap deploy was successful after jumping through some hurdles specifically setting up keyholder and checking out the wmf/stable branch. Currently troubleshooting a connection issue on MariaDB.

Jul 13 20:15:31 phabricator-trixie systemd[1]: Starting phd.service - phabricator-phd...
Jul 13 20:15:31 phabricator-trixie phd[560406]: [2026-07-13 20:15:31] EXCEPTION: (PhabricatorClusterStrandedException) Unable to establish a connection to any database host (while trying "phabricator_config"). >
Jul 13 20:15:31 phabricator-trixie phd[560406]: AphrontConnectionQueryException: Attempt to connect to app_user@127.0.0.1 failed with error #1130: Host '127.0.0.1' is not allowed to connect to this MariaDB serv>
Jul 13 20:15:31 phabricator-trixie phd[560406]: arcanist(), ava(), phorge(), translations(), wmf-ext-misc()

@Arnoldokoth I looked at this and wanted to fix the database grants. But when I looked at mysql console I saw there are no databases in it yet. But if no database has been created yet then what would we grant access to.

This makes me think we first need to run some initial script that creates the databases.

I found this one:

https://www.jimby.name/techbits/recent/phorge/

which includes these steps:

# mysql -u root -p    (use the password from the initialization step above if needed)
  root@localhost [(none)]> create user 'phorge'@'localhost' identified by 'MyPhorgeSecretPw';
  root@localhost [(none)]> create database phorge character set utf8mb4 collate utf8mb4_general_ci;
  root@localhost [(none)]> grant all privileges on `phabricator\_%`.* to 'phorge'@'localhost';   NOTE THE USE OF BACKTICKS HERE. 
  root@localhost [(none)]> grant all privileges on phorge.* to 'phorge'@'localhost';
  root@localhost [(none)]> flush privileges;
  root@localhost [(none)]> quit;

That should fix it.

Thanks @Dzahn

I got this from @brennen https://www.mediawiki.org/wiki/Phabricator/Local_Dev_Environment#Set_up_MySQL/MariaDB_database_privileges

Though one of the fixes was disabling skip-name-resolve in the MariaDB config.

It's working now: https://phabricator-new.wmcloud.org/

I also dumped the the database from phabricator-bullseye to the new host and looks okay afaict. We should be good to go with getting rid of the old host.

cool! congrats getting it to work!

There is one issue though. Avatars are not loading on the new instance. On Firefox, it says A resource is blocked by OpaqueResponseBlocking, please check browser console for details and NS_BINDING_ABORTED. Then on Chrome it says (failed)net::ERR_BLOCKED_BY_ORB. So some kind of CORS issue since these are being served from a different domain.

@brennen and I are trying to find a fix for this.

Actually, the dump might have caused file integrity issues on the new instance. It looks like when you upload a new avatar on the host it renders as expected from the usercontent domain.