Converting a top level project to a sub-project is magic that @mmodell has an admin script for somewhere I believe. I know he did the needful for the projects that we moved around this way before.

Anyone with CLI access (e.g. me) could do that, see https://wikitech.wikimedia.org/wiki/Phabricator#Converting_a_parent_project_into_a_subproject

Is this still wanted? If yes, which project tag (URL) should become a subproject under which project tag (URL)?

Flagging for discussion by the WMCS team.

Some context for that discussion:

• As far as I know, LabsDB-Auditor is a dead project
• PAWS and Quarry could be sub-projects of the Cloud-Services umbrella project.
• Wikibugs can be a top level project as it is today, or could be made a sub-project of Tools

Ah, usual disclaimer: I highly recommend checking the "Here are some warnings, to avoid wrong expectations" section in https://www.mediawiki.org/wiki/Phabricator/Project_management#Parent_Projects,_Subprojects_and_Milestones first. :)

Discussed in the 2020-09-30 WMCS team meeting. Our consensus:

• We should move PAWS and Quarry to be sub-projects of Cloud-Services
• @Krenair should be asked if archiving LabsDB-Auditor is reasonable
• @Legoktm and @valhallasw should be asked if they would like Wikibugs made into a sub-project of Tools. When the WMCS team created the Tools umbrella project we stated that it was optional for tool related projects to be placed there. That still seems reasonable to us today.

@Aklapper if you can do the needful for PAWS and Quarry that would be great.

Urgh, looking at https://wikitech.wikimedia.org/wiki/Phabricator#Converting_a_parent_project_into_a_subproject I am wondering if The workboard seems to get nuked so you'd have to locally keep data which task is in which column and manually fix that up is still true (I really hope not?)

Also, as I'd first need to manually remove the future-parent project #cloud-services from all #PAWS and #Quarry tasks to avoid DB corruption, Phab's "Advanced" search for those tasks being in both parent projects (cloud-services && (paws || quarry)) also lists tasks which are in subprojects of #cloud-services, making identifying the actual tasks a bit more cumbersome. Links for myself when proceeding here:

• https://phabricator.wikimedia.org/maniphest/query/XoMmF9kFbxWr/#R (PAWS and CS, need to remove CS only from those task which are actually in CS)
• https://phabricator.wikimedia.org/maniphest/query/y7jhxusFtXdP/#R (Quarry and CS, need to remove CS only from those task which are actually in CS)

When I discovered VPS subproject list my expectation has been that the full structure of the cloud architecture will be reflected.

• Wikidata-Query-Service is a VPS as well but not yet under any umbrella.

Minus 2:

• https://fountain.wmflabs.org/ does not work, no VPS for now.
• dawiki not yet configured, at least not as VPS.

All VPS subdomains but no other should be derived from Cloud-Services > VPS-Projects rather than from Projects root.

Active things under toolforge.org should be in Tools. That might improve navigation and understanding, and avoid confusion. Cloud-Services is branching into Tools and VPS and this is quite intuitive. Together with Data-Services as umbrella and administrative tasks like Toolforge etc. that makes structure obvious. Dead projects do not need to be moved around.

In T177787#6581535, @PerfektesChaos wrote:

• Wikidata-Query-Service is a VPS as well but not yet under any umbrella.

This is incorrect. WDQS has some testing environment(s) in Cloud VPS projects, but WDQS itself (https://query.wikidata.org/) is hosted in the Wikimedia Foundation's "production" network on dedicated hardware.

Maybe; one reason more to avoid confusion and attempt clear structures.

I guess I've said before that this is a PITA. And buggy. And complex. I tried and I failed, see below.

In T177787#6506238, @bd808 wrote:

• We should move PAWS and Quarry to be sub-projects of Cloud-Services

• Check for Herald rules which mention Quarry or PAWS in their name:
  • SELECT ruleId FROM phabricator_herald.herald_condition WHERE fieldName = "projects" AND (value LIKE "%PHID-PROJ-yh2h2e536o6uq6dvw3bu%" OR value LIKE "%PHID-PROJ-5gpxds34uk5hmw2zvgc4%"); lists my rule H9
  • SELECT ruleId FROM phabricator_herald.herald_action WHERE action LIKE "%projects%" AND (target LIKE "%PHID-PROJ-yh2h2e536o6uq6dvw3bu%" OR target LIKE "%PHID-PROJ-5gpxds34uk5hmw2zvgc4%"); lists no results
  • Update Herald rules accordingly
• Check for milestone collisions on the same hierarchy level only (!), as tasks in milestones can only be in 1 milestone having the same parent project.
  • Check soon-to-be-parent project tag if it has any milestones at https://phabricator.wikimedia.org/project/subprojects/832/ - none directly in this case but some subprojects have milestones (doing so as tasks in milestones can only be in one Milestone and it is unclear to me if a task can be in a milestone of a parent project and a milestone of a subproject of the parent project at the same time, or if a task can be in a milestone of a subproject of the parent project and a different milestone of a different subproject of the parent project at the same time)
  • Check soon-to-be-subproject tag for milestones: Quarry has one milestone and zero subprojects: https://phabricator.wikimedia.org/project/subprojects/800/
  • Check soon-to-be-subproject tag for milestones: PAWS has three milestones and zero subprojects: https://phabricator.wikimedia.org/project/subprojects/1648/
• Manually check if #Cloud-Services soon-to-be-parent project tag itself (!) is set on any of the soon-to-be-subproject tasks; silently mass-remove if that's the case, but manually do ignore any subprojects of the soon-to-be-parent project tag which will also be listed in the query results, as it is fine when a task is associated with more than one subproject of the same parent project.
  • PAWS: https://phabricator.wikimedia.org/maniphest/query/7itrd8VPAKzO/#R - silently mass-removed for 132196, 296870, 160114, 183142, 178722, 167086, 158685, 146061, 132025
  • Quarry: https://phabricator.wikimedia.org/maniphest/query/swEcStTW_SAX/#R - silently mass-removed for 283773, 165625, 141818, 121502, 104308, 251719, 242355, 207677, 107627, 152212
• Open PAWS workboard at https://phabricator.wikimedia.org/project/board/1648/?filter=all&hidden=true in case something goes wrong and we need to restore columns and column positions
• Open Quarry workboard at https://phabricator.wikimedia.org/project/board/800/?filter=all&hidden=true in case something goes wrong and we need to restore columns and column positions
• Run CLI script to convert these [2 projects] into subprojects of #cloud-services:
  • Quarry: sudo /srv/phab/phabricator/bin/move_project --subproject --child 800 --parent 832 --keep-members child
    • FAIL: ...and this is where we lost the Quarry milestone "FY2021/2022-Q1" with its four tasks at https://phabricator.wikimedia.org/project/view/5469/ : "This object has an unknown or invalid policy setting ("")." https://phabricator.wikimedia.org/project/edit/5469/ cannot be reached either.
  • PAWS: sudo /srv/phab/phabricator/bin/move_project --subproject --child 1648 --parent 832 --keep-members child
• Because "If a project has no subprojects, but then one is added, the existing Project becomes a “parent” Project and all of its members shift to that first Subproject", potentially remove these members again from the first subproject created - not applicable
• Check and update project tag descriptions for potential invalid links - nothing to update

FAIL: ...and this is where we lost the Quarry milestone "FY2021/2022-Q1" with its four tasks at https://phabricator.wikimedia.org/project/view/5469/ : "This object has an unknown or invalid policy setting ("")." https://phabricator.wikimedia.org/project/edit/5469/ cannot be reached either.

Quarry at https://phabricator.wikimedia.org/project/subprojects/800/ incorrectly shows no milestones.

https://phabricator.wikimedia.org/maniphest/query/Cc_iNCoVId1j/#R also shows no tasks in that milestone and after entering the milestone name and running the query, it converts its displayed name to "Restricted Project".

mysql:phstats@m3-slave.eqiad.wmnet [phabricator_project]> SELECT id, name, phid, viewPolicy, editPolicy, hasMilestones, hasSubprojects, parentProjectPHID, milestoneNumber FROM phabricator_project.project WHERE (id = 5469 OR id = 800 OR id = 832);
+------+----------------+--------------------------------+------------+--------------------------------+---------------+----------------+--------------------------------+-----------------+
| id   | name           | phid                           | viewPolicy | editPolicy                     | hasMilestones | hasSubprojects | parentProjectPHID              | milestoneNumber |
+------+----------------+--------------------------------+------------+--------------------------------+---------------+----------------+--------------------------------+-----------------+
|  832 | Cloud-Services | PHID-PROJ-msyn2z45n7mw45bfuscb | public     | PHID-PLCY-gsth6m2as2637wjel4ti |             0 |              1 | NULL                           |            NULL |
|  800 | Quarry         | PHID-PROJ-yh2h2e536o6uq6dvw3bu | public     | PHID-PLCY-gsth6m2as2637wjel4ti |             1 |              0 | PHID-PROJ-msyn2z45n7mw45bfuscb |            NULL |
| 5469 | FY2021/2022-Q1 | PHID-PROJ-zmu45phc7kizekpym7uo | public     | users                          |             0 |              0 | PHID-PROJ-yh2h2e536o6uq6dvw3bu |               1 |
+------+----------------+--------------------------------+------------+--------------------------------+---------------+----------------+--------------------------------+-----------------+
3 rows in set (0.001 sec)

mysql:phstats@m3-slave.eqiad.wmnet [phabricator_maniphest]> SELECT CONCAT("https://phabricator.wikimedia.org/T", t.id) AS affectedTasks FROM phabricator_maniphest.edge e INNER JOIN phabricator_maniphest.maniphest_task t WHERE e.type = 41 AND e.src = t.phid AND e.dst = "PHID-PROJ-zmu45phc7kizekpym7uo"; 
+-------------------------------------------+
| affectedTasks                             |
+-------------------------------------------+
| https://phabricator.wikimedia.org/T288528 |
| https://phabricator.wikimedia.org/T289503 |
| https://phabricator.wikimedia.org/T287471 |
| https://phabricator.wikimedia.org/T71037  |
+-------------------------------------------+
4 rows in set (0.001 sec)

$:acko\> echo '{"objectIdentifier":"5469", "transactions":[{"type":"view","value":"public"}]}' | /var/www/html/phorge/arcanist/bin/arc call-conduit --conduit-uri https://phabricator.wikimedia.org --conduit-token "api-xxxxxxxxxxxxxxxxxxxxxxxxxxxx" project.edit --
{
  "error": "ERR-CONDUIT-CORE",
  "errorMessage": "ERR-CONDUIT-CORE: <project.edit> [Access Denied: Restricted Project] (Can View) You do not have permission to view this object. // This object has an unknown or invalid policy setting (\"\"). Members of a project can always view it.",
  "response": null
}

This is why I'd like to entirely refrain from trying to convert parent projects to subprojects until someone spends a lot of time fixing and testing that script.

No big damage done here. Still not the way how to do things.

In T177787#9023514, @Aklapper wrote:

I guess I've said before that this is a PITA. And buggy. And complex. I tried and I failed, see below.

In T177787#6506238, @bd808 wrote:

• We should move PAWS and Quarry to be sub-projects of Cloud-Services

I disagree that these should be a sub-project of cloud-services. Sounds like that is difficult to facilitate as well. These are community projects. I believe some general perspective on this has changed in the three years since previous consensus was reached. I would ask that these not be designated a sub-project. And it sounds like it is a terrible thing from a phabricator perspective to do, so bonus?

Eh, sorry.
I could manually revert this if wanted by creating a new Quarry Phab project, moving the tasks, and archive the old one. Though will trigger some notifications.

In T177787#9032078, @Aklapper wrote:

Eh, sorry.
I could manually revert this if wanted by creating a new Quarry Phab project, moving the tasks, and archive the old one. Though will trigger some notifications.

I probably misread the above. If this is already done, there is no need to revert it.