Page MenuHomePhabricator

Migrate phabricator.wikimedia.org to Phorge as upstream
Closed, ResolvedPublic

Assigned To
Authored By
brennen
Apr 3 2023, 8:53 PM
Referenced Files
None
Tokens
"Orange Medal" token, awarded by Aklapper."Love" token, awarded by DAlangi_WMF."Barnstar" token, awarded by Jdforrester-WMF."Love" token, awarded by Dzahn."Love" token, awarded by valerio.bozzolan."Love" token, awarded by Remagoxer."Barnstar" token, awarded by greg.

Description

Overview

Phorge is the community fork of Phabricator. WMF's plan is to migrate to using this as an upstream. (T302528)

Phorge has https://we.phorge.it/w/installation_and_setup/update_from_phabricator/ as documentation for this process.

In rough outline, the plan here is mostly:

  • Switch upstream remotes in the deployment repos
  • Pull in changes from the Phorge project
  • Test both locally and in the devtools project
  • Run a typical scap deploy to production.

Related stuff:

Upstream changes

A list of commits for both Phorge's stable branch and master, where development is happening:

phorge commits on stable:
113b17ceca Promote 2022.37 to stable
377ac059d6 Update default wordmark and logo
dbf7cd2b4a Rebrand: Rename library and update "version" logic
c3f0c0b0f7 Merge Phacility/master into phorge
9426765a2c Flatten "RemarkupValue" objects when setting field defaults for custom forms
1fc04e238f Fix repository URI generation for SVN commit identifiers represented with integer types
944b257d5d Fix a policy issue where permissions were not properly checked when disabling global builtin queries
3052ed1484 Remove obsolete, policy-violating "owners.query" API method
835cd7f270 Improve remarkup metadata robustness against debris in the database
7d4357683a Hide the blurb of a user when that user is disabled
b293e6ffed Make Datepicker translatable
fd1922c763 Update Install and Update Scripts to the new Paths and Repository links
ce31289228 (PhabricatorENV) update doclinks to link to we.phorge.it instead of secure.phabricator.com
a4948ec800 Add conduit endpoints for querying legalpad
69cb760921 Updated .arclint in Phorge to exclude package-lock.json from Aphlict setup
e7740c8669 Add HarbormasterHookController as an entry point for all Harbormaster hooks
2258ba8535 Add package.json for aphlict
af36da4741 Rename example sshd files
1ddb953b9f Update Diviner documentation to reference Phorge instead of Phabricator.
7157dd96b3 T15006: Update .arcconfig to point to we.phorge.it
fda1a55450 Update Readme

arcanist commits on stable:
42a51e15 Promote 2022.37 to stable
0c0b9644 Rebrand: Change Server name
9b4bcc83 Merge Phacility/master into phorge
42302929 Fix incorrect quoting of author in 'arc patch'
5407e5e5 Fix PhutilLibraryMapBuilder to call the right function in log()
7498c102 Update arcanist readme to reference Phorge
737bd0d4 Update Diviner documentation to reference Phorge instead of Phabricator for Arcanist.
cf4b14fc Arcanist Update .arcconfig to point to we.phorge.it

-------

phorge commits on master:
1b9da964e9 Add setup check to avoid git version 2.5.0 and below (May 2015)
9c759fe23a Fix preamble-support
a7b472284b Mobile: show Homepage contents and avoid duplicate menus
0a82ccb8a4 AphrontFileResponse: avoid alien usages of setDownload()
d46aab8321 Add GitHub mirror to list of known mirrors
b587865ce7 People: add a permalink to list Administrators
dad1b15a9c User Badges: better integrate with Removed Comments
3a8ee271d9 Render all Removed Comments a little more docile
524579fe64 Diffusion: fix missing mobile "Actions" from Manage Repo
787a84969f Phriction: clarify its search results as "Wiki page"
306ce1c0b7 Fix "Search" Application description
96ae4ba13a PHP 8.1: fixes for strlen() not accepting NULL anymore, part 2
d25d630fe5 PHP 8.2: fixes for strlen() not accepting NULL anymore, part 1
fc9bbb9949 Workboard: fix regression caused by tags= prefiller
821df3364e Hide the "hidden" fields on custom form previews
6036079e88 PHP 8.2: fix deprecated use of "parent" in callables
5eac69b79d Ponder: strike code of resolved questions in query results
780e86acf8 Herald Rule Creation Page: fix Back button in some cases
f0618d54e8 Installation Guide: mention that git 2.5.0 (2015) is needed server-side
38e3692d7b Workboard: in some cases, increase menu opacity (accessibility)
5e2b367715 Workboard: improve the Create Task link to mention the Project slug
d98c47041f PHP 8.2: fix deprecated creation of dynamic properties
03c9bf575e PHP 8.2: fixes for deprecated use of ${var} in strings
42a8b73adb Feed Differential retitled: do not repeat the title twice
9bc3c16b6e Fix information page about Read-only status
c6f56b8221 Workboard: fix CTRL+click on "Create Task" and "Edit Task"
b33e373503 Drag & Drop: set a link as external
4f356838be FileUpload: show File page instead of binary file
5cba56182f Diviner: fix "Javascript" -> "JavaScript" and minor change
3487ee444a Fix regression in new confirmation Dialog
9623e66745 replace usage of each() with foreach() for smtp
66192a5b83 Fix "Undefined index: icon" when visiting Search Servers using MySQL
53c31b7b13 Fix middle-click, CTRL+click, right-click etc. on Typehead search results
36dba82224 Show the API summary on the top of each Conduit API page
20fb93d1a4 Add small documentation details
83672ba949 Allow to prefill name and description fields of a new Passphrase
6a563a68b5 Celerify
02a4f8b0c8 Fix editing a Calendar import ICS URI
d5040f9a8f updating twitch to latest api (Helix)
dc558b5538 Fix NULL pointer exception in some circumstances from Calendar's homepage
a8a38835e1 Fix broken file PHID extraction that causes Pholio uploads to crash
abb693962d Update sprite sheets
4aedc206c7 Update PassphraseCredential to support Mentions
f8ffa393c4 Added cross-platform default fonts
69b2710af9 Prevent Line Numbers in Diffusion being copied as Tabs
a16e8773f0 Show confirmation dialog when closing a modal if form contents have been changed
377ac059d6 Update default wordmark and logo
dbf7cd2b4a Rebrand: Rename library and update "version" logic
c3f0c0b0f7 Merge Phacility/master into phorge
9426765a2c Flatten "RemarkupValue" objects when setting field defaults for custom forms
1fc04e238f Fix repository URI generation for SVN commit identifiers represented with integer types
944b257d5d Fix a policy issue where permissions were not properly checked when disabling global builtin queries
3052ed1484 Remove obsolete, policy-violating "owners.query" API method
835cd7f270 Improve remarkup metadata robustness against debris in the database
7d4357683a Hide the blurb of a user when that user is disabled
b293e6ffed Make Datepicker translatable
fd1922c763 Update Install and Update Scripts to the new Paths and Repository links
ce31289228 (PhabricatorENV) update doclinks to link to we.phorge.it instead of secure.phabricator.com
a4948ec800 Add conduit endpoints for querying legalpad
69cb760921 Updated .arclint in Phorge to exclude package-lock.json from Aphlict setup
e7740c8669 Add HarbormasterHookController as an entry point for all Harbormaster hooks
2258ba8535 Add package.json for aphlict
af36da4741 Rename example sshd files
1ddb953b9f Update Diviner documentation to reference Phorge instead of Phabricator.
7157dd96b3 T15006: Update .arcconfig to point to we.phorge.it
fda1a55450 Update Readme

arcanist commits on master:
08dfffd5 Replace function utf8_decode() - deprecated since PHP 8.2
9e1bb955 updating twitch to latest api (Helix)
0c0b9644 Rebrand: Change Server name
9b4bcc83 Merge Phacility/master into phorge
42302929 Fix incorrect quoting of author in 'arc patch'
5407e5e5 Fix PhutilLibraryMapBuilder to call the right function in log()
7498c102 Update arcanist readme to reference Phorge
737bd0d4 Update Diviner documentation to reference Phorge instead of Phabricator for Arcanist.
cf4b14fc Arcanist Update .arcconfig to point to we.phorge.it

There are some conflicts for Phorge:

8:30:03 brennen@metaphor:~/code/wmf/phabricator/deployment/phabricator (wmf/migrate-phorge) ❦ git merge phorge/stable
Removing webroot/rsrc/image/logo/light-eye.png
Auto-merging src/view/form/control/AphrontFormDateControl.php
Auto-merging src/infrastructure/env/PhabricatorEnv.php
Removing src/docs/user/feedback.diviner
Removing src/docs/flavor/about_flavor_text.diviner
Removing src/docs/contributor/cla.diviner
Removing src/applications/owners/conduit/OwnersQueryConduitAPIMethod.php
Auto-merging src/applications/legalpad/storage/LegalpadDocumentSignature.php
Auto-merging src/applications/legalpad/query/LegalpadDocumentSignatureSearchEngine.php
Auto-merging src/applications/legalpad/query/LegalpadDocumentSignatureQuery.php
CONFLICT (content): Merge conflict in src/applications/legalpad/query/LegalpadDocumentSignatureQuery.php
CONFLICT (add/add): Merge conflict in src/applications/legalpad/conduit/LegalpadSignatureSearchConduitAPIMethod.php
Auto-merging src/applications/legalpad/conduit/LegalpadSignatureSearchConduitAPIMethod.php
Auto-merging src/__phutil_library_map__.php
Auto-merging resources/celerity/map.php
CONFLICT (content): Merge conflict in resources/celerity/map.php
Auto-merging .arclint
Auto-merging .arcconfig
CONFLICT (content): Merge conflict in .arcconfig
Automatic merge failed; fix conflicts and then commit the result.
zsh: exit 1     git merge phorge/stable

So a first step is working through those.


T302528 - is where WMF originally considered this and made a decision

T335080 - is a subtask how to handle the database part of the migration

T334519 - is an (relatively unrelated) task to upgrade phabricator (or phorge) servers to bullseye

T328595 - is a previous task in which we created a test instance for phorge, running in the devtools project in cloud VPS

Details

ReferenceSource BranchDest BranchAuthorTitle
repos/phabricator/deployment!16work/phorge-migration-2023-07-11wmf/stablebrennenmerge updated arcanist and phabricator submodules from phorge
repos/phabricator/deployment!10work/phorge-migrationwmf/stablebrennenDraft: Merge updated arcanist and phabricator submodules from Phorge
repos/phabricator/deployment!9work/phorge-migrationwmf/stablebrennenTentative merge of phorge repos
Customize query in GitLab

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

T302528 - is where WMF originally considered this and made a decision

T335080 - is a subtask how to handle the database part of the migration

T334519 - is an (relatively unrelated) task to upgrade phabricator (or phorge) servers to bullseye

T328595 - is a previous task in which we created a test instance for phorge, running in the devtools project in cloud VPS. For it we created the puppet class profile::phorge which is separate from current/historic phabricator setup which can likely be simplified (removal of the whole LVS service for git-ssh etc)

Note that if we can slowly reduce the number of local patches needed in Wikimedia Phabricator, this is a good starting point upstream:

https://we.phorge.it/T15081

I don't see a sub-task for Phorge PHP 8.1 support downstream here, so I would like to mention here that @Aklapper contributed so damn much in the last 6 days to Phorge: in this moment 28+ patches were proposed by aklapper, reviewed, and landed on Phorge's main branch.

To have an idea about this nice work, have a look at the sub-tasks here:

https://we.phorge.it/T15064

So feel free to follow that Task when Debian bookworm (now still in testing) will be needed.

Thanks. In my understanding, for the Wikimedia migration we will not target Debian Bookworm (PHP 8.2) but Debian Bullseye which ships PHP 7.4.

Thanks. In my understanding, for the Wikimedia migration we will not target Debian Bookworm (PHP 8.2) but Debian Bullseye which ships PHP 7.4.

✅ Nice. And I strongly recommend to follow this ↑ at least for this year (to avoid too many "little surprises").

Maybe this is already under control but I noticed some configurations that - at the moment - are probably database-only, like additional maniphest.statuses (In Progress / Stalled etc.) and maybe other things.

Probably these conf could also be pasted here for reproducibility reasons:

https://www.mediawiki.org/wiki/Phabricator/Local_Dev_Environment

(We can discuss this specific thing maybe here: https://www.mediawiki.org/wiki/Talk:Phabricator/Local_Dev_Environment)

https://phab.wmflabs.org/ is now running a merge of https://we.phorge.it/rPaa27cccc8713540a496b4f62429615531be74add into wmf/stable (see above draft MR). I need to clean up the state of the merge on our side a bit - missed some badly automerged stuff in the Legalpad code on my first pass - but I think this is at least testable.

Great news and work !:) thank you!

https://phab.wmflabs.org/ is now running a merge of https://we.phorge.it/rPaa27cccc8713540a496b4f62429615531be74add into wmf/stable (see above draft MR). I need to clean up the state of the merge on our side a bit - missed some badly automerged stuff in the Legalpad code on my first pass - but I think this is at least testable.

Really nice!

To simplify the work of testers, I tried to visually clarify a bit the new features in the current stable:

https://we.phorge.it/w/changelog/2023.17/

Enjoy testing! ✨

brennen changed the task status from Open to In Progress.Aug 8 2023, 9:28 PM
brennen added a subscriber: Jelto.

@Jelto @eoghan - would someone be available this coming Monday, 2023-08-14 @ 16:00 UTC to downtime alerts for this deploy?

@brennen Yep, I'll be around and can give you a hand. I'll stick some time in my calendar and we can coordinate what you need.

Please see: https://phabricator.wikimedia.org/T335080#9079148
There will be no DBAs around and most of EU will be off the next day (and on batphone oncall).

brennen changed the task status from In Progress to Stalled.Aug 16 2023, 4:49 AM

Deployment of this had been scheduled for tomorrow. Unfortunately I'm dealing with a set of family emergencies this week, and will have to defer.

Take care <3 - we will be ready for this when you are

Ok, I'd like to re-schedule this for this Wednesday, 2023-08-23 @ 15:00 UTC (08:00 PDT).

Any objections? I know this collides with the MW on k8s sync, but I'm afraid this is probably the last practical morning before I'm out for a while.

brennen changed the task status from Stalled to In Progress.Aug 23 2023, 2:46 PM

Planning to go ahead in ~15 minutes.

Deployment notes: https://etherpad.wikimedia.org/p/2023-08-23-phabricator

Icinga downtime and Alertmanager silence (ID=00df6f2a-2b2a-4d40-81db-d959750d0622) set by jelto@cumin1001 for 1:00:00 on 1 host(s) and their services with reason: Switch Phabricator to Phorge

phab1004.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-08-23T15:21:04Z] <brennen@deploy1002> Started deploy [phabricator/deployment@82e8e76]: update phabricator to phorge (T333885)

Mentioned in SAL (#wikimedia-operations) [2023-08-23T15:21:43Z] <brennen@deploy1002> Finished deploy [phabricator/deployment@82e8e76]: update phabricator to phorge (T333885) (duration: 00m 38s)

Copying from Etherpad: Notes for the 2023-08-23 Phabricator → Phorge deploy

Background:

Checklist - pre-deploy:

  • Merge relevant branches {{done}}
  • Checkout latest wmf/stable on deploy1002.eqiad.wmnet, ensure correct state {{done}}
  • Ensure database backup has been done {{done}}

Checklist - deploy:

  • Downtime phab* for 1 hour
  • sudo cookbook sre.hosts.downtime -r "Switch Phabricator to Phorge" -t T333885 -M 60 phab1004.eqiad.wmnet {{done}}
  • alertmanager (alerts.wikimedia.org) create new silence (top right corner) for label 'instance' = phab1004.eqiad.wmnet
  • deploy1002: scap deploy -v -l 'phab1004.eqiad.wmnet' 'update phabricator to phorge (T333885)'
  • above but for phab2002 {{done}}

In case of rollback:

  • Last scap tag: scap/sync/2023-06-27/0001

Potential takeaways:

One update re: rollbacks: As best I could tell from a quick revert of code on phab.wmflabs.org to a pre-Phorge state, legalpad functionality seems unaffected by the changed schema.

Optimistically calling this resolved after an uneventful day.

f8ffa393c4 caused a visible and unexpected change for me, especially as the update wasn’t really announced beforehand (so I had no idea why the fonts suddenly changed). It’s not necessarily a bad change, but something that’s worth being mentioned.

Yeah to announce this maybe we can publish the post stub in T340223 (but I do not have enough permissions)

I'm also still confused about Tech News but maybe it's not off-topic there https://meta.wikimedia.org/wiki/Talk:Tech/News/2023/35

BTW I'm sorry Tacsipacsi for this problem on your device. Please join https://we.phorge.it/T15049 and let's see what we can do about it.

There it would be nice to share your current browser and operating system. Also a cute screenshot would be awesome.

I have a Debian 11 with Cinnamon and Firefox. I don’t have Segoe UI or Helvetica/Helvetica Neue installed, but I did install Arial years ago for some reason. However, the default UI font is not Arial, but rather something the settings app calls “Sans Regular” (maybe DejaVu Sans?), so when you prepended system-ui to the font specification, it started to use that “Sans Regular” instead of Arial. Since it’s me who broke things in some sense (by installing Arial), I don’t think you need to fix it, it’s just worth being mentioned that others with custom font setups can also experience it. (I replied here because I don’t have a Phorge account and I don’t think it’s worth creating one, as I don’t expect you to follow up on this in Phorge.)

Icinga downtime and Alertmanager silence (ID=98876a84-36a0-444b-8998-f6e6693c8597) set by jelto@cumin1001 for 0:30:00 on 1 host(s) and their services with reason: Phabricator maintenance

phab1004.eqiad.wmnet