Page MenuHomePhabricator
Create Task
Maniphest T328595

Create proof of concept Phorge instance in devtools project in wmcs (phorge.wmcloud.org)
Closed, ResolvedPublic
Actions

Description

We want to create an instance of Phorge, the Phabricator community fork, in the devtools project in wmcs.

This will be as a proof of concept, to test it and see if we can import tickets from Phabricator.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Dzahn created this task.Feb 1 2023, 7:11 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 1 2023, 7:11 PM
Dzahn mentioned this in T302528: Consider migrating from Phabricator to Phorge once that new upstream is ready for such a migration.Feb 1 2023, 7:12 PM
Dzahn added a subscriber: MacFan4000.

also see https://we.phorge.it/w/installation_and_setup/update_from_phabricator/ from T302528#8244434 . thanks to @MacFan4000

Dzahn claimed this task.Feb 1 2023, 7:15 PM
Aklapper added a subtask: T302528: Consider migrating from Phabricator to Phorge once that new upstream is ready for such a migration.Feb 1 2023, 7:19 PM
Jdforrester-WMF subscribed.Feb 1 2023, 8:33 PM

Should there be a new project for Phab->Phorge, or are single tasks/trees sufficient?

Dzahn added a comment.Feb 1 2023, 10:07 PM

Project in Phabricator? I don't think that's needed. It should be sufficient to have tasks/trees.

Lens0021 subscribed.Feb 2 2023, 1:30 AM
Ltrlg subscribed.Feb 2 2023, 9:41 AM
Dzahn moved this task from Incoming to Backlog on the collaboration-services board.Feb 3 2023, 10:17 PM
Dzahn added a parent task: T328815: devtools: quota increase request.
LSobanski moved this task from Backlog to Work in Progress on the collaboration-services board.Feb 6 2023, 4:32 PM
Dzahn added a comment.EditedFeb 7 2023, 12:35 AM

Our quota increase request was granted. :)

I could then create a new instance "phorge-1001" in the devtools project.

I am now applying the "role::simplelamp2" to let puppet install a basic LAMP.. before getting to the Phorge part itself.

The installation guide is https://we.phorge.it/book/phorge/article/installation_guide/ and says you need a LAMP, unsurprisingly.

Aklapper added a parent task: Restricted Task.Feb 7 2023, 12:45 AM
gerritbot added a comment.Feb 7 2023, 6:42 PM

Change 887379 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phorge: create role/profile to install LAMP and git clone phorge

https://gerrit.wikimedia.org/r/887379

gerritbot added a project: Patch-For-Review.Feb 7 2023, 6:42 PM
gerritbot added a comment.Feb 7 2023, 7:07 PM

Change 887379 merged by Dzahn:

[operations/puppet@production] phorge: create role/profile to install LAMP and git clone phorge

https://gerrit.wikimedia.org/r/887379

Maintenance_bot removed a project: Patch-For-Review.Feb 7 2023, 7:29 PM
gerritbot added a comment.Feb 7 2023, 7:46 PM

Change 887392 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phorge: list of apache modules needs to be an array

https://gerrit.wikimedia.org/r/887392

gerritbot added a project: Patch-For-Review.Feb 7 2023, 7:46 PM

Change 887392 merged by Dzahn:

[operations/puppet@production] phorge: list of apache modules needs to be an array

https://gerrit.wikimedia.org/r/887392

gerritbot added a comment.Feb 7 2023, 8:23 PM

Change 887397 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phorge: add minimal apache site, add class parameters for docroot et al

https://gerrit.wikimedia.org/r/887397

gerritbot added a comment.Feb 7 2023, 8:30 PM

Change 887397 merged by Dzahn:

[operations/puppet@production] phorge: add minimal apache site, add class parameters for docroot et al

https://gerrit.wikimedia.org/r/887397

Maintenance_bot removed a project: Patch-For-Review.Feb 7 2023, 8:30 PM
gerritbot added a comment.Feb 7 2023, 8:57 PM

Change 887410 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phorge: add httpd Directory snippet, git clone arcanist

https://gerrit.wikimedia.org/r/887410

gerritbot added a project: Patch-For-Review.Feb 7 2023, 8:57 PM
gerritbot added a comment.Feb 7 2023, 9:12 PM

Change 887410 merged by Dzahn:

[operations/puppet@production] phorge: add httpd Directory snippet, git clone arcanist

https://gerrit.wikimedia.org/r/887410

Maintenance_bot removed a project: Patch-For-Review.Feb 7 2023, 9:30 PM
gerritbot added a comment.Feb 7 2023, 11:45 PM

Change 887429 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phorge: separate install pathes for phorge and arcanist

https://gerrit.wikimedia.org/r/887429

gerritbot added a project: Patch-For-Review.Feb 7 2023, 11:45 PM
gerritbot added a comment.Feb 7 2023, 11:57 PM

Change 887429 merged by Dzahn:

[operations/puppet@production] phorge: separate install pathes for phorge and arcanist

https://gerrit.wikimedia.org/r/887429

gerritbot added a comment.Feb 7 2023, 11:57 PM

Change 887431 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phorge: git clone arcanist also from we.phorge.it, not Phacility

https://gerrit.wikimedia.org/r/887431

gerritbot added a comment.Feb 8 2023, 12:00 AM

Change 887431 merged by Dzahn:

[operations/puppet@production] phorge: git clone arcanist also from we.phorge.it, not Phacility

https://gerrit.wikimedia.org/r/887431

gerritbot added a comment.Feb 8 2023, 12:09 AM

Change 887432 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phorge: mvove httpd setup to profile and don't call it apache

https://gerrit.wikimedia.org/r/887432

gerritbot added a comment.Feb 8 2023, 12:13 AM

Change 887433 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phorge: install php-mbstring, php-curl and php-mysql modules

https://gerrit.wikimedia.org/r/887433

gerritbot added a comment.Feb 8 2023, 7:30 PM

Change 887432 merged by Dzahn:

[operations/puppet@production] phorge: move httpd setup to profile and don't call it apache

https://gerrit.wikimedia.org/r/887432

gerritbot added a comment.Feb 8 2023, 7:34 PM

Change 887820 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phorge: renamed apache.conf.erb to httpd.conf.erb

https://gerrit.wikimedia.org/r/887820

gerritbot added a comment.Feb 8 2023, 7:41 PM

Change 887433 merged by Dzahn:

[operations/puppet@production] phorge: install php-mbstring, php-curl and php-mysql modules

https://gerrit.wikimedia.org/r/887433

gerritbot added a comment.Feb 8 2023, 7:41 PM

Change 887820 merged by Dzahn:

[operations/puppet@production] phorge: rename apache.conf.erb to httpd.conf.erb

https://gerrit.wikimedia.org/r/887820

Maintenance_bot removed a project: Patch-For-Review.Feb 8 2023, 8:30 PM
gerritbot added a comment.Feb 18 2023, 2:08 AM

Change 890131 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] devtools: set mariadb datadir path for phorge-1001 instance

https://gerrit.wikimedia.org/r/890131

gerritbot added a project: Patch-For-Review.Feb 18 2023, 2:08 AM
gerritbot added a comment.Feb 18 2023, 2:12 AM

Change 890131 merged by Dzahn:

[operations/puppet@production] devtools: set mariadb datadir path for phorge-1001 instance

https://gerrit.wikimedia.org/r/890131

gerritbot added a comment.Feb 18 2023, 2:17 AM

Change 890132 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phorge: install php-zip and php-gd packages

https://gerrit.wikimedia.org/r/890132

gerritbot added a comment.Feb 18 2023, 2:20 AM

Change 890132 merged by Dzahn:

[operations/puppet@production] phorge: install php-zip and php-gd packages

https://gerrit.wikimedia.org/r/890132

gerritbot added a comment.Feb 18 2023, 2:29 AM

Change 890133 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phorge: install php-apcu and python3-pygments

https://gerrit.wikimedia.org/r/890133

gerritbot added a comment.Feb 18 2023, 2:32 AM

Change 890133 merged by Dzahn:

[operations/puppet@production] phorge: install php-apcu and python3-pygments

https://gerrit.wikimedia.org/r/890133

gerritbot added a comment.Feb 18 2023, 2:44 AM

Change 890134 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phorge: add parameter and value for the repo path

https://gerrit.wikimedia.org/r/890134

gerritbot added a comment.Feb 18 2023, 2:46 AM

Change 890134 merged by Dzahn:

[operations/puppet@production] phorge: add parameter and value for the repo path

https://gerrit.wikimedia.org/r/890134

Dzahn added a comment.EditedFeb 18 2023, 2:51 AM

This config value could not be set in the web UI, it was locked. This had to be used to set it instead;

root@phorge-1001:/srv/phorge# ./bin/config set repository.default-local-path /srv/repo
 DONE  Wrote configuration key "repository.default-local-path" to local storage (in file "conf/local/local.json").
Maintenance_bot removed a project: Patch-For-Review.Feb 18 2023, 3:10 AM
Dzahn added a comment.Mar 7 2023, 11:59 PM

After further debugging and solving a couple setup issues, this works now!

https://phorge.wmcloud.org/

the main fix needed was to replace "http" with "https" in the base-uri setting in the main config file.

This is used in src pathes to CSS and JS and it seems like when there is http in there those get blocked (CSP added by domainproxy?, they can be loaded from external).

Just changing that to https made it work.

Other fixes were mysql/mariadb related. The following made 3 "unresolved setup issues" go away:

23:31 < mutante> !log devtools - phorge-1001 - MariaDB [(none)]> SET GLOBAL sql_mode = "STRICT_ALL_TABLES,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION";

23:33 < mutante> !log devtools - phorge-1001 - MariaDB [(none)]> SET GLOBAL local_infile=0;

23:34 < mutante> !log devtools - phorge-1001 - MariaDB [(none)]> SET GLOBAL max_allowed_packet=33554432;

Dzahn added a comment.Mar 8 2023, 12:05 AM

@MacFan4000 Your user has been approved now.

Stashbot added a comment.Mar 11 2023, 12:20 AM

Mentioned in SAL (#wikimedia-cloud) [2023-03-11T00:20:27Z] <mutante> - on phorge1001, enable general query log in mysql (mariadb), to learn about database scheme, don't forget to turn that off so VM doesn't run out of disk (SET GLOBAL general_log=1;) T328595

Dzahn added a comment.EditedMar 11 2023, 12:40 AM

I looked at using the public dump file to import tasks but that does not seem to be possible. It's not a mysql dump file at all. (as also was mentioned years ago in old ticket T108199 it is lacking some data).

The script that creates that dump is in /srv/deployment/phabricator/deployment/tools/public_task_dump.py. There does not seem to be an import script.

Then I looked more at the actual database scheme, trying to export a single task in production to then import it here.

Turns out that is also not easy at all... "The schema design of Phabricator relies extensively on relational ids and there appears to be no easy way to export individual data sets as "open data" with object metadata descriptors."

P45729 phabricator selecting a maniphest task - mysql log
1 102952 Query SELECT `task`.* FROM `maniphest_task` task WHERE (task.id in (1)) ORDER BY `task`.`id` DESC
2 102952 Query SELECT edge.* FROM `edge` edge WHERE ((edge.src IN ('PHID-TASK-qrwadcbqndxztsztatld')) AND (edge.type IN ('21'))) ORDER BY edge.dateCreated DESC, edge.seq DESC
3 102952 Query SELECT edge.* FROM `edge` edge WHERE ((edge.src IN ('PHID-TASK-qrwadcbqndxztsztatld')) AND (edge.type IN ('1', '11', '38', '51', '52', '62'))) ORDER BY edge.dateCreated DESC, edge.seq DESC
4 102952 Query SELECT `x`.* FROM `maniphest_transaction` `x` WHERE (x.objectPHID IN ('PHID-TASK-qrwadcbqndxztsztatld')) ORDER BY `x`.`id` DESC LIMIT 101
5 102952 Query SELECT `xcomment`.* FROM `maniphest_transaction_comment` `xcomment` WHERE (`xcomment`.phid IN ('PHID-XCMT-gkjfkmmcbygfhhbvuzpl')) ORDER BY `xcomment`.`id` DESC
6 102952 Query SELECT `task`.* FROM `maniphest_task` task WHERE ((`task`.spacePHID IS NULL) AND (task.phid in ('PHID-TASK-qrwadcbqndxztsztatld'))) ORDER BY `task`.`id` DESC
7 102950 Query SELECT `user`.* FROM `user` `user` WHERE (user.phid IN ('PHID-USER-iwt6im6hqz2lpzwbid5o')) ORDER BY `user`.`id` DESC
8 102950 Query SELECT * FROM `phabricator_user`.`user_profile` WHERE userPHID IN ('PHID-USER-iwt6im6hqz2lpzwbid5o')
9 102950 Query SELECT cacheKey, userPHID, cacheData, cacheType FROM `user_cache`
10 WHERE cacheIndex IN ('y_FPXujT93U3') AND userPHID IN ('PHID-USER-iwt6im6hqz2lpzwbid5o')
11 102952 Query SELECT `task`.* FROM `maniphest_task` task WHERE ((`task`.spacePHID IS NULL) AND (task.phid in ('PHID-TASK-qrwadcbqndxztsztatld'))) ORDER BY `task`.`id` DESC
12 102953 Connect root@localhost on phabricator_search using Socket
13 102953 Query SET NAMES utf8mb4
14 102953 Query SELECT * FROM `search_editengineconfiguration` WHERE (engineKey IN ('maniphest.task')) ORDER BY `id` DESC
15 102949 Query SELECT * FROM `phabricator_cache`.`cache_markupcache` WHERE cacheKey IN ('remarkup(cSnfXpiJCQzN, GsQsT4XwhXMx)@21@7VwBk7PApozc')
16 102954 Connect root@localhost on phabricator_file using Socket
17 102954 Query SET NAMES utf8mb4
18 102954 Query SELECT `attachments`.* FROM `file_attachment` `attachments` WHERE (attachments.objectPHID IN ('PHID-TASK-qrwadcbqndxztsztatld')) ORDER BY `attachments`.`id` DESC LIMIT 101
19 102952 Query SELECT edge.* FROM `edge` edge WHERE ((edge.src IN ('PHID-TASK-qrwadcbqndxztsztatld')) AND (edge.type IN ('41'))) ORDER BY edge.dateCreated DESC, edge.seq DESC
20 102952 Query SELECT edge.* FROM `edge` edge WHERE ((edge.src IN ('PHID-TASK-qrwadcbqndxztsztatld')) AND (edge.type IN ('21'))) ORDER BY edge.dateCreated DESC, edge.seq DESC
21 102950 Query SELECT `user`.* FROM `user` `user` WHERE (user.phid IN ('PHID-USER-iwt6im6hqz2lpzwbid5o')) ORDER BY `user`.`id` DESC
22 102955 Connect root@localhost on phabricator_token using Socket
23 102955 Query SET NAMES utf8mb4
24 102955 Query SELECT * FROM `token_given` WHERE (objectPHID IN ('PHID-TASK-qrwadcbqndxztsztatld')) ORDER BY `id` DESC
25 102953 Query SELECT * FROM `search_editengineconfiguration` WHERE ((engineKey IN ('maniphest.task')) AND (subtype IN ('default'))) ORDER BY `id` DESC
26 102953 Query SELECT * FROM `search_editengineconfiguration` WHERE ((engineKey IN ('maniphest.task')) AND (subtype IN ('default'))) ORDER BY `id` DESC
27 102956 Connect root@localhost on phabricator_draft using Socket
28 102956 Query SET NAMES utf8mb4
29 102956 Query SELECT * FROM `phabricator_draft`.`draft_versioneddraft` WHERE objectPHID = 'PHID-TASK-qrwadcbqndxztsztatld' AND authorPHID = 'PHID-USER-iwt6im6hqz2lpzwbid5o' ORDER BY version DESC LIMIT 1
30 102952 Query SELECT id AS version FROM `maniphest_transaction`
31 WHERE objectPHID = 'PHID-TASK-qrwadcbqndxztsztatld' AND authorPHID = 'PHID-USER-iwt6im6hqz2lpzwbid5o'
32 ORDER BY id DESC LIMIT 1
33 102952 Query SELECT edge.* FROM `edge` edge WHERE ((edge.src IN ('PHID-TASK-qrwadcbqndxztsztatld')) AND (edge.type IN ('41'))) ORDER BY edge.dateCreated DESC, edge.seq DESC
34 102952 Query SELECT edge.* FROM `edge` edge WHERE ((edge.src IN ('PHID-TASK-qrwadcbqndxztsztatld')) AND (edge.type IN ('4', '3', '1'))) ORDER BY edge.dateCreated DESC, edge.seq DESC
35 102957 Connect root@localhost on phabricator_project using Socket
36 102957 Query SET NAMES utf8mb4
37 102957 Query SELECT `project`.* FROM `project` `project` JOIN `edge` e ON e.src = project.phid AND e.type = 60 WHERE ((`project`.spacePHID IS NULL) AND (project.status IN ('0')) AND (e.dst IN ('PHID-USER-iwt6im6hqz2lpzwbid5o')) AND (project.milestoneNumber IS NULL)) GROUP BY `project`.phid ORDER BY `project`.`name` ASC LIMIT 5
38 102950 Query SELECT `user`.* FROM `user` `user` WHERE (user.phid IN ('PHID-USER-iwt6im6hqz2lpzwbid5o')) ORDER BY `user`.`id` DESC
39 102950 Query SELECT * FROM `phabricator_user`.`user_profile` WHERE userPHID IN ('PHID-USER-iwt6im6hqz2lpzwbid5o')

The scenario that Christopher warned about in 2015 seems be coming true (T108199 - Evaluate Phabricator for its ability to export and import data) was exactly right but that ticket was declined years ago :/.

The only good news is that it seems like the database scheme is still the same in Phorge.

So in production we should be able to just replace phab with phorge keeping the same DB.. but importing tickets here (without importing the entire database which is a no-go because of private tasks in cloud projects).. is neither easy nor very useful for testing.

Switching to a completely different bug tracker would therefore also be a problem.

Dzahn closed this task as Resolved.Mar 11 2023, 12:53 AM

Regardless of that outcome, this ticket is resolved because all the boxes are checked now:

  • We want to create an instance of Phorge, the Phabricator community fork, in the devtools project in wmcs. - done
  • This will be as a proof of concept, to test it and see if we can import tickets from Phabricator. - we saw if we could and found out not really
  • .. but this should not stop us from replacing phabricator with phorge in production.
valerio.bozzolan subscribed.Mar 13 2023, 5:46 PM
valerio.bozzolan added a comment.Mar 13 2023, 5:49 PM

Hoping to be useful I registered a test volunteer account called valerio.bozz. Waiting for approval

Dzahn added a comment.Mar 13 2023, 5:51 PM

Thanks. Approved!

valerio.bozzolan awarded a token.Mar 13 2023, 5:52 PM
Dzahn mentioned this in P43914 phorge install example.Mar 27 2023, 4:59 PM
Dzahn mentioned this in T333885: Migrate phabricator.wikimedia.org to Phorge as upstream.Apr 20 2023, 12:39 AM
Dzahn added a parent task: T333885: Migrate phabricator.wikimedia.org to Phorge as upstream.
Aklapper renamed this task from create a phorge instance in the devtools project in wmcs to Create proof of concept Phorge instance in devtools project in wmcs (phorge.wmcloud.org).Jun 8 2023, 1:38 PM
Stashbot added a comment.Nov 21 2023, 9:00 PM

Mentioned in SAL (#wikimedia-cloud) [2023-11-21T21:00:34Z] <mutante> - deleted instance phorge-1001 to get quota back and allow for creting new phabricator-on-bullseye instance T328595 T327068

Stashbot mentioned this in T327068: Bullseye upgrade for remaining Collab hosts.Nov 21 2023, 9:00 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL