Page MenuHomePhabricator

Create application to manage Diffusion repositories for a Tool Labs project
Closed, ResolvedPublic

Description

Forking task from outline in T117071#2201228.

Requirements

Create a web application that uses the various conduit APIs to:

  • Create a Diffusion repository based on an existing Tool Labs project
  • Automatically set-up the "Editable By" and "Can Push" policies for the repository created through the tool, and also provide a UI to allow project owners to manage that

To keep things simple, all repos managed by the app will have some uniform characteristics:

  • Repo name is "tool-<service group name>" (e.g. "tool-verisons" for tools.versions)
  • Repo callsign is "TOOL<gid of service group>" (e.g. "TOOL52937" for tools.versions) See T133252#2290871
  • "Dangerous Changes" flag (force push and branch delete) enabled
  • All members of the tool having a Phabricator account at the time the repo is created are granted edit and push rights. (We can live with Phabricator users needing to be added manually in the app if we can't find a method to map from LDAP users to Phabricator users automatically.)

Some additional simplifying assumptions for the initial tool:

  • A repo is associated with a single shared tool account (service group)
  • Management of the repo via the tool is granted to all members of the tool account (maintainers)
  • Only one repo per tool is allowed
    • We can add support for multiple repos as a later improvement
  • The tool will not manage mirrors, staging, automation, or any other advanced diffusion settings. These features can be handled manually via Phabricator tickets by members of the repository-admins group.

This set of functions provides the bare minimum needed to publish the code for a tool, namely a web accessible version control system. By adding an .arcconfig file to the repository this bare minimum can be expanded to allow code submission as well via arcanist/differential.

Constraints

The application needs access to LDAP data on service group membership for authorization (only managers of a tool should be able to modify the settings for a repo) and an authentication (login) method that can be correlated with the group members. The authz information needed is available within Labs/Tool Labs, but currently there is no suitable authn solution. SUL OAuth is available to tools today, but SUL accounts have no correlation with wikitech accounts and/or LDAP data.

If we want to write new data into the LDAP directory, the application can not be hosted in either Tool Labs or Labs generally. Access to the credentials needed to modify LDAP cannot be properly secured inside the Labs environment. This means that the app will need to be hosted in the production side of the WMF network.

Related Objects

Event Timeline

As I see it, there are 4 options for how to build this application:

  • Add this functionality to wikitech as a MediaWiki extension. This would be either additional special page(s) for MediaWiki-extensions-OpenStackManager or a separate extension that depends on OSM and/or the LDAP auth used in wikitech.
  • Build a Horizon dashboard that implements the desired functionality.
  • Build a standalone application
  • Build a standalone application that can be extended in the future to manage additional workflows for Toolforge

I prefer the fourth for these reasons:

Assuming that a standalone application is chosen, python and django would be a reasonable choice for the development work:

  • Horizon is a django app and there are potential opportunities to share libraries or django apps between the two solutions.
  • Django and python provide many off-the-shelf components and libraries that can be leveraged to reduce development effort.
  • Python is a well liked language by the Operations team in general and the Cloud-Services ops team in particular. This will be useful for future additions where sharing libraries for workflows such as managing Tools-Kubernetes containers will be possible.
Restricted Application added a project: User-bd808. · View Herald TranscriptApr 21 2016, 3:06 AM
scfc added a subscriber: scfc.Apr 24 2016, 4:06 AM

Without having thought this through: Why not a Phabricator "application" for this and similar tasks? I think more users are acquainted with it than with Horizon or a stand-alone application. It is meant to be the central hub for everything development-related, so if a user could do his administration needs in it without having to contact wikitech, Horizon or something else, that would be preferable IMHO.

(This question probably belongs better to another task; feel free to move and answer there.)

I don't think something like a Horizon dashboard is appropriate because it doesn't fit in with the workflow for Tool Labs. The whole point of Horizon is to manage virtual machines and Tool Labs doesn't have virtual machines. Horizon also needs 2FA, which may be a barrier for some people.

If a standalone application is used, then it should be extendable, so additional functionality can be added in the future, like T128158.

It would be nice to do everything in Phabricator, as per @scfc's idea, because it means the user doesn't have to learn yet another interface to do things with.

Without having thought this through: Why not a Phabricator "application" for this and similar tasks? I think more users are acquainted with it than with Horizon or a stand-alone application. It is meant to be the central hub for everything development-related, so if a user could do his administration needs in it without having to contact wikitech, Horizon or something else, that would be preferable IMHO.
(This question probably belongs better to another task; feel free to move and answer there.)

Phabricator would satisfy some of the necessary and desired points:

  • SUL integration via OAuth
  • Hosted outside of Labs which allows read/write LDAP communication
  • "Home" of Diffusion which should allow deep integration for configuration needs

Cons of Phabricator as the home for a Tool Labs feature:

  • Not currently required for any Tool Labs workflows (but neither is a standalone tool)
  • PHP implementation language not widely used by techops teams (but well understood by MediaWiki contributors)
  • PHP libraries not sharable with Horizon or cli tools developed for Tool Labs
  • Guided workflows will be more difficult in a shared audience application. Adding to the navigational chrome of the site will be especially controversial.
  • Small number of developers have familiarity with the Phabricator codebase (and none of them are assigned to this project or Tool Labs in general).

If a standalone application is used, then it should be extendable, so additional functionality can be added in the future, like T128158.

This would indeed by my intention. The larger story arc for Tool Labs improvements which was partially inspired by T128158: Tools web interface for tool authors (Brainstorming ticket) describes a "console.wmflabs.org" application that would eventually manage:

  • Associating SUL and LDAP user accounts
  • Git repository management
  • Creation of new LDAP accounts tied to existing SUL accounts
  • Tool (service group) creation and membership management
  • Metadata management for Tools

This initial git management story would implement concrete solutions for the first 2 bullets. The additional features would be added later as either one or two follow on projects.

It would be nice to do everything in Phabricator, as per @scfc's idea, because it means the user doesn't have to learn yet another interface to do things with.

While I agree it would be nice, there are a bunch of practical problems with it - primarily,

Third-party application development is something we plan to support eventually, but it is not supported today (May 2015), and support is very far away

from https://secure.phabricator.com/T5447

So if we do any application development on top of phabricator, it requires that we basically maintain our own fork of Phabricator for the foreseeable future. It would also mess with Phabricator upgrades, since we'll have to basically look for any breaking changes (which won't be easily available or such, since upstream specifically says they do not support it) and fix them in our custom code. I don't really think we've the manpower to do that.

Hence, I don't think writing a custom Phab app is a viable option.

A small bit of pile on due to the subject matter :)

We have a very small amount of local changes we maintain currently (to phab) and they are a big pain. We have tried as an organization to write/maintain phab apps before and it was a huge, huge time sink. Apps are fairly well architected in the phab universe but there is no internal API that has any amount of historical or future consistency. It is an extremely fast moving project, see the comment from one of our maintainers here https://secure.phabricator.com/T5447#124128. Phab is an excellent software forge that is really tempting to integrate with for a case like this, and it's even a reasonable idea if we are willing to commit nearly a full time person to maintain it. There are a few apps I would like to see created but it's a hard road to take.

kaldari updated the task description. (Show Details)Apr 28 2016, 10:13 PM

A small bit of pile on due to the subject matter :)
We have a very small amount of local changes we maintain currently (to phab) and they are a big pain. We have tried as an organization to write/maintain phab apps before and it was a huge, huge time sink. Apps are fairly well architected in the phab universe but there is no internal API that has any amount of historical or future consistency. It is an extremely fast moving project, see the comment from one of our maintainers here https://secure.phabricator.com/T5447#124128. Phab is an excellent software forge that is really tempting to integrate with for a case like this, and it's even a reasonable idea if we are willing to commit nearly a full time person to maintain it. There are a few apps I would like to see created but it's a hard road to take.

@chasemp: I think the situation is improving, especially for straightforward integrations. There are a few fairly stable integration points and much of the conduit api is stabilizing as well. So it really depends on what integrations want to do, some things could live with quite minimal maintenance as long as they don't touch many internal APIs or schemas.

Anyway my original offer still stands - it would not be much work to create an API method that you can call to fetch the SUL -> LDAP associations from phabricator.

@mmodell The custom API that would be useful in the near term would be one that lets me lookup a user entry like user.query does using the LDAP username associated with an account as the query input. That would let me map the list of maintainers for a given tool that are stored in LDAP with a list of phids that I will need to ensure in the "Editable By" and "Can Push" policies of an associated Diffusion repo.

The next piece we will need after that is a way to actually edit the "Editable By" and "Can Push" policies. I'm not sure that there is Conduit access to that yet either. There is no repository.edit api endpoint to do things like project.edit's members.[add|remove|set] transaction types allow.

@bd808: Upstream Task T10748: looks relevant: "Implement diffusion.repository.edit, for creating and editing repositories via the API"

mmodell added a comment.EditedMay 3 2016, 9:37 PM

@bd808:

From https://secure.phabricator.com/phame/post/view/771/development_notes_2016_week_18/

"Repository APIs
Changes are coming to repositories soon to make them fully queryable and editable via the API. You can find details in https://secure.phabricator.com/T10748.
Additionally, these changes will make repositories much more flexible about how they handle the URIs they fetch from, mirror to, and offer to users as clone URIs. You can find details about the URI changes in https://secure.phabricator.com/T10366.
These changes exist today in a new set of "shadow" interfaces, but aren't yet accessible from the UI. We'll cut over to make them authoritative shortly, likely in the next week."

heh @mmodell your relative T#'s got mangled in that quote to obscure bugzilla things :)

but seems pretty cool

heh @mmodell your relative T#'s got mangled in that quote to obscure bugzilla things :)

Whoops. Fixed

mmodell added a comment.EditedMay 5 2016, 1:08 AM

Also, the custom api method "user.ldapquery" is now live on this instance.

Also, the custom api method "user.ldapquery" is now live on this instance.

Thanks for your help with this @mmodell. It looks like I just need to get down to the fun work of figuring out how to use all of this now.

It looks like I just need to get down to the fun work of figuring out how to use all of this now.

Upstream documentation: https://secure.phabricator.com/book/phabricator/article/diffusion_api/

It looks like setting up a mirror of an external public repo may be possible with this as well.

bd808 moved this task from To Do to In Dev/Progress on the User-bd808 board.May 12 2016, 3:57 PM

Repo callsign is "TOOL<gid of service group>" (e.g. "TOOL52937" for tools.versions

This won't work. Manual testing on https://phab-03.wmflabs.org resulted in the error message "Callsigns may only contain UPPERCASE letters.". Callsigns are actually optional, so maybe we can just leave that undefined. The alternative would be to try and come up with some stable, unique mapping from a service group to a "short" alphabetic suffix.

Let's ignore callsigns if we can!

bd808 updated the task description. (Show Details)May 12 2016, 9:36 PM

If you leave callsign undefined it just names the repositories with a sequential number based on the order of creation, like rEWIS extension-Wikispeech

I've been doing some testing on https://phab-03.wmflabs.org/. I can create repos via the API there, but there are a few things about them that aren't quite right yet. The biggest problem at the moment is that local hosting isn't enabled for the new repos and there is no explicit way to enable that outside of the web UI.

Comparing the source we have in local version control against the upstream source shows that I need to wait for rP29d1115 to land locally. So close to working!

There doesn't seem to be a conduit api for creating/editing/searching policies. We will need that in order to set the push policy for the newly created repo to something other than the default "anyone can push" value. Policies are widely used as parameters in the new edit apis, so maybe I'm just missing something? The format of the messages passed back and forth for the ajax calls that power policy editing via the web UI look a whole lot like the new style transactional edit apis, so naively it doesn't seem like this would be incredibility difficult to expose via conduit.

@bd808: I can deploy newer upstream code for you.

With regard to policies, I've had considerable experience working with those. I can help write a custom conduit method that will generate the exact policy you need and return the phid for use with the repository edit api.

Is the content of the policy straightforward? I imagine it would be something like:

{
  "projects": "#repository-admins",
  "users": ["list", "usernames"]
}

Something straightforward like that isn't difficult at all to write.

@bd808: I can deploy newer upstream code for you.

That would be great to keep the proof of concept work moving forward.

With regard to policies, I've had considerable experience working with those. I can help write a custom conduit method that will generate the exact policy you need and return the phid for use with the repository edit api.
Is the content of the policy straightforward? I imagine it would be something like:

{
  "projects": "#repository-admins",
  "users": ["list", "usernames"]
}

Something straightforward like that isn't difficult at all to write.

That would also be most excellent. When the web gui is used the sort of policy I'm imagining looks a bit like this in the POST to Phab:

{
  "default": "deny",
  "rules": [
    {
      "action":"allow",
      "rule":"PhabricatorProjectsPolicyRule",
      "value":[
        "PHID-PROJ-74bt3nlwd4hl2ofmw77h"
      ]
    },
    {
      "action":"allow",
      "rule":"PhabricatorUsersPolicyRule",
      "value":[
        "PHID-USER-wzvm7msgcojlqmymu3vc",
        "PHID-USER-sikaw4dhyejwbfg2wujb"
      ]
    }
  ]
}

It's easy enough to look up the phids for things from their symbolic names.

And the response looks like:

{
  "error":null,
  "payload":{
    "phid":"PHID-PLCY-nsfusm6s3fmvw55ytam6",
    "info":{
      "name":"Custom Policy",
      "full":"Custom Policy",
      "icon":"fa-certificate"
    }
  }
}

Should we fork a subtask to track working on something like that?

@bd808: I merged upstream/master on phab-03 so that you can make more progress without waiting for prod to update.

bd808 moved this task from Triage to In Progress on the Toolforge board.

Basic testing environment deployed at http://striker.wmflabs.org/. See http://devwiki-striker.wmflabs.org/wiki/VagrantRoleStriker for some documentation of the test setup. This is using a stand-alone LDAP directory, wiki farm, and Phabricator instance.

bd808 moved this task from Backlog to Doing on the Striker board.
bd808 moved this task from Doing to Done on the Striker board.Mar 4 2017, 3:57 AM