As I see it, there are 4 options for how to build this application:

• Add this functionality to wikitech as a MediaWiki extension. This would be either additional special page(s) for MediaWiki-extensions-OpenStackManager or a separate extension that depends on OSM and/or the LDAP auth used in wikitech.
• Build a Horizon dashboard that implements the desired functionality.
• Build a standalone application
• Build a standalone application that can be extended in the future to manage additional workflows for Toolforge

I prefer the fourth for these reasons:

• @Andrew is actively working on replacing MediaWiki-extensions-OpenStackManager with Horizon. (T87279: Make OpenStack Horizon useful for production labs)
• Horizon is an OpenStack upstream project focused on management of virtual machines rather than the workflows of Tool Labs.
• There are other projects described by the Tool Labs vision that would benefit from having authn/authz correlated with the LDAP directory and the ability to write information to LDAP.

Assuming that a standalone application is chosen, python and django would be a reasonable choice for the development work:

• Horizon is a django app and there are potential opportunities to share libraries or django apps between the two solutions.
• Django and python provide many off-the-shelf components and libraries that can be leveraged to reduce development effort.
• Python is a well liked language by the SRE team in general and the Cloud-Services ops team in particular. This will be useful for future additions where sharing libraries for workflows such as managing Tools-Kubernetes containers will be possible.

Without having thought this through: Why not a Phabricator "application" for this and similar tasks? I think more users are acquainted with it than with Horizon or a stand-alone application. It is meant to be the central hub for everything development-related, so if a user could do his administration needs in it without having to contact wikitech, Horizon or something else, that would be preferable IMHO.

(This question probably belongs better to another task; feel free to move and answer there.)

I don't think something like a Horizon dashboard is appropriate because it doesn't fit in with the workflow for Tool Labs. The whole point of Horizon is to manage virtual machines and Tool Labs doesn't have virtual machines. Horizon also needs 2FA, which may be a barrier for some people.

If a standalone application is used, then it should be extendable, so additional functionality can be added in the future, like T128158.

It would be nice to do everything in Phabricator, as per @scfc's idea, because it means the user doesn't have to learn yet another interface to do things with.

In T133252#2232927, @scfc wrote:

Without having thought this through: Why not a Phabricator "application" for this and similar tasks? I think more users are acquainted with it than with Horizon or a stand-alone application. It is meant to be the central hub for everything development-related, so if a user could do his administration needs in it without having to contact wikitech, Horizon or something else, that would be preferable IMHO.

(This question probably belongs better to another task; feel free to move and answer there.)

Phabricator would satisfy some of the necessary and desired points:

• SUL integration via OAuth
• Hosted outside of Labs which allows read/write LDAP communication
• "Home" of Diffusion which should allow deep integration for configuration needs

Cons of Phabricator as the home for a Tool Labs feature:

• Not currently required for any Tool Labs workflows (but neither is a standalone tool)
• PHP implementation language not widely used by techops teams (but well understood by MediaWiki contributors)
• PHP libraries not sharable with Horizon or cli tools developed for Tool Labs
• Guided workflows will be more difficult in a shared audience application. Adding to the navigational chrome of the site will be especially controversial.
• Small number of developers have familiarity with the Phabricator codebase (and none of them are assigned to this project or Tool Labs in general).

In T133252#2233064, @tom29739 wrote:

If a standalone application is used, then it should be extendable, so additional functionality can be added in the future, like T128158.

This would indeed by my intention. The larger story arc for Tool Labs improvements which was partially inspired by T128158: Tools web interface for tool authors (Brainstorming ticket) describes a "console.wmflabs.org" application that would eventually manage:

• Associating SUL and LDAP user accounts
• Git repository management
• Creation of new LDAP accounts tied to existing SUL accounts
• Tool (service group) creation and membership management
• Metadata management for Tools

This initial git management story would implement concrete solutions for the first 2 bullets. The additional features would be added later as either one or two follow on projects.

In T133252#2233064, @tom29739 wrote:

It would be nice to do everything in Phabricator, as per @scfc's idea, because it means the user doesn't have to learn yet another interface to do things with.

While I agree it would be nice, there are a bunch of practical problems with it - primarily,

Third-party application development is something we plan to support eventually, but it is not supported today (May 2015), and support is very far away

from https://secure.phabricator.com/T5447

So if we do any application development on top of phabricator, it requires that we basically maintain our own fork of Phabricator for the foreseeable future. It would also mess with Phabricator upgrades, since we'll have to basically look for any breaking changes (which won't be easily available or such, since upstream specifically says they do not support it) and fix them in our custom code. I don't really think we've the manpower to do that.

Hence, I don't think writing a custom Phab app is a viable option.

A small bit of pile on due to the subject matter :)

We have a very small amount of local changes we maintain currently (to phab) and they are a big pain. We have tried as an organization to write/maintain phab apps before and it was a huge, huge time sink. Apps are fairly well architected in the phab universe but there is no internal API that has any amount of historical or future consistency. It is an extremely fast moving project, see the comment from one of our maintainers here https://secure.phabricator.com/T5447#124128. Phab is an excellent software forge that is really tempting to integrate with for a case like this, and it's even a reasonable idea if we are willing to commit nearly a full time person to maintain it. There are a few apps I would like to see created but it's a hard road to take.

In T133252#2235001, @chasemp wrote:

A small bit of pile on due to the subject matter :)

We have a very small amount of local changes we maintain currently (to phab) and they are a big pain. We have tried as an organization to write/maintain phab apps before and it was a huge, huge time sink. Apps are fairly well architected in the phab universe but there is no internal API that has any amount of historical or future consistency. It is an extremely fast moving project, see the comment from one of our maintainers here https://secure.phabricator.com/T5447#124128. Phab is an excellent software forge that is really tempting to integrate with for a case like this, and it's even a reasonable idea if we are willing to commit nearly a full time person to maintain it. There are a few apps I would like to see created but it's a hard road to take.

@chasemp: I think the situation is improving, especially for straightforward integrations. There are a few fairly stable integration points and much of the conduit api is stabilizing as well. So it really depends on what integrations want to do, some things could live with quite minimal maintenance as long as they don't touch many internal APIs or schemas.

In T134188#2257751, @mmodell wrote:

Anyway my original offer still stands - it would not be much work to create an API method that you can call to fetch the SUL -> LDAP associations from phabricator.

@mmodell The custom API that would be useful in the near term would be one that lets me lookup a user entry like user.query does using the LDAP username associated with an account as the query input. That would let me map the list of maintainers for a given tool that are stored in LDAP with a list of phids that I will need to ensure in the "Editable By" and "Can Push" policies of an associated Diffusion repo.

The next piece we will need after that is a way to actually edit the "Editable By" and "Can Push" policies. I'm not sure that there is Conduit access to that yet either. There is no repository.edit api endpoint to do things like project.edit's members.[add|remove|set] transaction types allow.

@bd808: Upstream Task T10748: looks relevant: "Implement diffusion.repository.edit, for creating and editing repositories via the API"

@bd808:

From https://secure.phabricator.com/phame/post/view/771/development_notes_2016_week_18/

"Repository APIs

Changes are coming to repositories soon to make them fully queryable and editable via the API. You can find details in https://secure.phabricator.com/T10748.

Additionally, these changes will make repositories much more flexible about how they handle the URIs they fetch from, mirror to, and offer to users as clone URIs. You can find details about the URI changes in https://secure.phabricator.com/T10366.

These changes exist today in a new set of "shadow" interfaces, but aren't yet accessible from the UI. We'll cut over to make them authoritative shortly, likely in the next week."

heh @mmodell your relative T#'s got mangled in that quote to obscure bugzilla things :)

but seems pretty cool

In T133252#2261987, @chasemp wrote:

heh @mmodell your relative T#'s got mangled in that quote to obscure bugzilla things :)

Whoops. Fixed

Looks like the conduit API is in place now: https://phabricator.wikimedia.org/conduit/method/diffusion.repository.edit/

Also, the custom api method "user.ldapquery" is now live on this instance.

In T133252#2266279, @mmodell wrote:

Looks like the conduit API is in place now: https://phabricator.wikimedia.org/conduit/method/diffusion.repository.edit/

In T133252#2266280, @mmodell wrote:

Also, the custom api method "user.ldapquery" is now live on this instance.

Thanks for your help with this @mmodell. It looks like I just need to get down to the fun work of figuring out how to use all of this now.

In T133252#2266309, @bd808 wrote:

It looks like I just need to get down to the fun work of figuring out how to use all of this now.

Upstream documentation: https://secure.phabricator.com/book/phabricator/article/diffusion_api/

It looks like setting up a mirror of an external public repo may be possible with this as well.

Repo callsign is "TOOL<gid of service group>" (e.g. "TOOL52937" for tools.versions

This won't work. Manual testing on https://phab-03.wmflabs.org resulted in the error message "Callsigns may only contain UPPERCASE letters.". Callsigns are actually optional, so maybe we can just leave that undefined. The alternative would be to try and come up with some stable, unique mapping from a service group to a "short" alphabetic suffix.

Let's ignore callsigns if we can!

If you leave callsign undefined it just names the repositories with a sequential number based on the order of creation, like rEWIS extension-Wikispeech

I've been doing some testing on https://phab-03.wmflabs.org/. I can create repos via the API there, but there are a few things about them that aren't quite right yet. The biggest problem at the moment is that local hosting isn't enabled for the new repos and there is no explicit way to enable that outside of the web UI.

Comparing the source we have in local version control against the upstream source shows that I need to wait for rP29d1115 to land locally. So close to working!

There doesn't seem to be a conduit api for creating/editing/searching policies. We will need that in order to set the push policy for the newly created repo to something other than the default "anyone can push" value. Policies are widely used as parameters in the new edit apis, so maybe I'm just missing something? The format of the messages passed back and forth for the ajax calls that power policy editing via the web UI look a whole lot like the new style transactional edit apis, so naively it doesn't seem like this would be incredibility difficult to expose via conduit.

@bd808: I can deploy newer upstream code for you.

With regard to policies, I've had considerable experience working with those. I can help write a custom conduit method that will generate the exact policy you need and return the phid for use with the repository edit api.

Is the content of the policy straightforward? I imagine it would be something like:

{
  "projects": "#repository-admins",
  "users": ["list", "usernames"]
}

Something straightforward like that isn't difficult at all to write.

In T133252#2291840, @mmodell wrote:

@bd808: I can deploy newer upstream code for you.

That would be great to keep the proof of concept work moving forward.

With regard to policies, I've had considerable experience working with those. I can help write a custom conduit method that will generate the exact policy you need and return the phid for use with the repository edit api.

Is the content of the policy straightforward? I imagine it would be something like:

{
  "projects": "#repository-admins",
  "users": ["list", "usernames"]
}

Something straightforward like that isn't difficult at all to write.

That would also be most excellent. When the web gui is used the sort of policy I'm imagining looks a bit like this in the POST to Phab:

{
  "default": "deny",
  "rules": [
    {
      "action":"allow",
      "rule":"PhabricatorProjectsPolicyRule",
      "value":[
        "PHID-PROJ-74bt3nlwd4hl2ofmw77h"
      ]
    },
    {
      "action":"allow",
      "rule":"PhabricatorUsersPolicyRule",
      "value":[
        "PHID-USER-wzvm7msgcojlqmymu3vc",
        "PHID-USER-sikaw4dhyejwbfg2wujb"
      ]
    }
  ]
}

It's easy enough to look up the phids for things from their symbolic names.

And the response looks like:

{
  "error":null,
  "payload":{
    "phid":"PHID-PLCY-nsfusm6s3fmvw55ytam6",
    "info":{
      "name":"Custom Policy",
      "full":"Custom Policy",
      "icon":"fa-certificate"
    }
  }
}

Should we fork a subtask to track working on something like that?

Should we fork a subtask to track working on something like that?

done.

@bd808: I merged upstream/master on phab-03 so that you can make more progress without waiting for prod to update.

Basic testing environment deployed at http://striker.wmflabs.org/. See http://devwiki-striker.wmflabs.org/wiki/VagrantRoleStriker for some documentation of the test setup. This is using a stand-alone LDAP directory, wiki farm, and Phabricator instance.