Page MenuHomePhabricator

Answer questions about ongoing maintenance of phabricator customizations/extensions
Closed, ResolvedPublic

Description

We've had a lot of requests for customization of phabricator which we have so far rejected as being to costly to maintain. This task is partially to explain why we are discouraging any modifications to phabricator that are not 100% necessary, and also to develop a plan for what modifications we will undertake and how we will deal with maintenance.

So one of the main questions is, who is going to take responsibility for maintaining changes when they inevitably break or cause merge conflicts with upstream? We don't have a dedicated phabricator team going forward as the current phabricator team is essentially on loan from other departments. We all have other responsibilities than phabricator. For example, any time that I (@mmodell) spend on phabricator is going to be essentially volunteer time since it will be in addition to my release engineering work. Up until now I've been granted a lot of leeway to essentially slack on other work because we all agreed that phabricator was the best use for my time and the migration was an important project for the entire organization.

I think the issue of who is responsible for phabricator going forward, and how much time we each dedicate to it, needs further discussion and perhaps an official management decision. Until that happens we are going to request that anyone who wants some customization in phabricator needs to volunteer their own time to maintain it or find some other way to ensure that the customization doesn't break or cause problems for ops when doing regular upgrades to phabricator.

When the custom functionality is confined to a phabricator extension that lives outside of the phabricator repository, the merge conflict issue is avoided making the code inherently easier to maintain. Other maintenance costs can be mostly mitigated with clean code and a few unit tests.

To help with some of these issues, I intend to write up a "contributing to wikimedia's phabricator" guide that will suggest best-practices for upstreaming functionality, testing and future-proofing code, and so on.

I'd love to find out how others feel about the issues raised here. I'm especially curious how we will deal with allocation of time to ongoing phabricator maintenance after the various migrations are complete, when phabricator is no longer a top priority project.

Event Timeline

mmodell created this task.Dec 13 2014, 9:02 AM
mmodell claimed this task.
mmodell raised the priority of this task from to Low.
mmodell updated the task description. (Show Details)
mmodell changed Security from none to None.
mmodell added subscribers: mmodell, Qgil, MZMcBride and 2 others.
fbstj awarded a token.Dec 13 2014, 9:51 AM
Qgil added a subscriber: Dzahn.Dec 13 2014, 10:44 AM

Thank you! All this is food for https://www.mediawiki.org/wiki/Phabricator/Code. Let's concentrate all the documentation there.

One big change between Bugzilla upgrades and Phabricator upgrades is that we are defaulting to a weekly maintenance window, that we might use or not. The pace of development of Phabricator is impressive, and that includes fixes to problems that we have reported and patches that we are upstreaming. This means that we will upgrade at least once a month, possibly twice, maybe more if needed. This process only works when our local patches are kept to a minimum, and when we have each change properly maintained and in sync for each upgrade.

Meanwhile, in Bugzilla... upgrading was done basically when we had no other chance, maybe once a year or so. New Bugzilla releases came from time to time (Phabricator doesn't have releases, their master branch IS the release). Patching locally was basically the only option left to implement the fixes we needed, because upstream was slow, and our instance was definitely not among their priorities. I remember that the last big upgrade took a significant work to @Dzahn and @Aklapper.

All this to say that the Bugzilla and Phabricator games are very different. Both are third party tools used by many others, but you can see why we don't want to maintain a big set of local patches to be synced every other week.

Qgil awarded a token.Dec 13 2014, 10:44 AM

Thank you for writing this.

I find the view that we're outright rejecting customizations to Phabricator to be simply unacceptable and completely unaligned with the reason that we chose to use Phabricator.

As I said on T69, a key selling point to switching to Phabricator was that (unlike, say, Gerrit), it's written in PHP and other familiar Web languages and we could easily make changes to it. If this isn't the case, then we need to seriously re-evaluate the decision to switch to Phabricator.

We cannot and we will not be held hostage by an external, upstream development team. This is the antithesis of free and open source software. If there are theme or functionality changes that we want, we should be able to implement those. If customizations fundamentally aren't possible using Phabricator, then Phabricator is the wrong tool for us.

scfc added a subscriber: scfc.Dec 13 2014, 9:57 PM

If we discard the weekly updates and adopt a schedule similar to our Bugzilla instance, the maintenance cost will be much lower.

Thanks for laying all of this out, Mukunda.

I think it's ok for us to have a temporary policy of generally postponing local customizations. We haven't yet clearly spelled out our maintenance strategy for Phabricator, so it's smart for us to be conservative about incurring long-term maintenance obligations until we do. Note that I say "temporary" there. By the end of March 2015, we need to state what our long term policy is. I suspect we're going to have to have some customizations like we had to have for Bugzilla and Gerrit and pretty much every other tool that is as central to our development process as Phabricator is, but let's spend some time letting the dust settle on this migration and letting people get used to the largely uncustomized version before insisting on a bunch of changes.

We should be careful about "rejecting" these changes purely on the grounds that upstream doesn't currently look likely to take it up. We should evaluate these things based on our needs, and only truly reject those things that we can internalize a clear rationale for, independent of upstream's rationale. For those things we aren't willing to give up on that seem stalled on upstream, we need to minimally keep a list of those things, and come up with our strategy for eventually getting those things addressed.

What tasks are in this category of problem? @MZMcBride cites T69, which seems like one we should treat with some level of seriousness (I guess I should comment there directly). Is that an exceptional case, or are all of the issues like that? Is it useful to make a tracking task (or use this one) as a means of compiling a list so that we can have a concrete discussion over a list of concerns?

@RobLa-WMF: so far most of the requests have been via IRC, and I'm sure a few are being tracked in phabricator but I'll have to search to find them. I just wrote this task to help avoid repeating the response, and to give people a place to hash out the issues.

@MZMcBride: I don't think anyone is saying that we refuse to customize phabricator or that it's not fundamentally customizable, just that we need to balance the need for customization against the ongoing costs of maintaining them, and that we can't just say anything goes without regard for the people who have to deal with the consequences when our local hacks break because of a lack of forethought, or due to the inevitable conflicts with upstream.

Here are a few other customization suggestions/requests that we've either declined or set aside for future consideration or possibly pushing upstream for consideration by the phab team:

T81 T96 T129 T234 T257 T78357

Most of these are found under the #phabricator.org project tag, which is being used for collecting tasks that most likely need to be handled upstream rather than working on them ourself.

We should be careful about "rejecting" these changes purely on the grounds that upstream doesn't currently look likely to take it up. We should evaluate these things based on our needs, and only truly reject those things that we can internalize a clear rationale for, independent of upstream's rationale. For those things we aren't willing to give up on that seem stalled on upstream, we need to minimally keep a list of those things, and come up with our strategy for eventually getting those things addressed.

I strongly agree with all of this.

We haven't yet clearly spelled out our maintenance strategy for Phabricator

Related: T175

In T78464#845649, @Qgil wrote:

New Bugzilla releases came from time to time (Phabricator doesn't have releases, their master branch IS the release). [...] I remember that the last big upgrade took a significant work to @Dzahn and @Aklapper.

I suspect we're going to have to have some customizations like we had to have for Bugzilla

Our customizations for Bugzilla feel rather small and focused on very specific areas, compared to the numerous ideas and requests that have been posted about Phabricator itself already (note that I am not judging the quality of those requests) - see https://git.wikimedia.org/tree/wikimedia%2Fbugzilla%2Fmodifications.git .
Still it took several weeks (in parallel to my usual ongoing tasks) to adjust our custom code and CSS changes in Bugzilla and test all those changes before we upgraded from Bugzilla 4.2 to 4.4 approx. a year ago.

Dzahn added a comment.Dec 15 2014, 1:09 PM

We don't have a dedicated phabricator team going

just to be clear about it, this is _not T175 ?

Dzahn added a comment.Dec 15 2014, 1:10 PM

key selling point to switching to Phabricator was that (unlike, say, Gerrit), it's written in PHP and other familiar Web languages and we could easily make changes to it.

This has been my thought as well.

mark added a subscriber: mark.Dec 15 2014, 3:54 PM

@Dzahn:

just to be clear about it, this is _not T175 ?

I simply meant to say that the phabricator team is not dedicated full-time to phabricator, at least not indefinitely.

Qgil added a comment.Dec 16 2014, 3:25 PM

key selling point to switching to Phabricator was that (unlike, say, Gerrit), it's written in PHP and other familiar Web languages and we could easily make changes to it.

This has been my thought as well.

The key selling point you refer to was and is the possibility to easily contribute upstream. This has been our emphasis since the beginning. Keeping our local patches at a minimum has been another explicit emphasis, also since the beginning.

So here is a recent example:

  1. @Krenair creates T78250 here, and after a short discussion he is encouraged to file the task upstream.
  2. He then creates https://secure.phabricator.com/T6757
  3. In 21 minutes (!) upstream sets priority to Low and gives specific instructions of the work needed.

Now we have to options. Wait until someone fixes that problem upstream, or contribute a patch ourselves. When a situation like this happened to Bugzilla or Gerrit, we basically relied in one or two people able to submit a patch. Today, we have dozens of developers that, given a special interest, could scratch that itch if they really want.

A few of those itches are more complex but also more relevant. While we focus our resources on those few tasks and we try to work as close with upstream as possible (getting a lot of work done by them directly), many others can step in to solve other problems. While this situation is not perfect and we all wish we would have more resources (a common theme in most software projects), the current situation is way better than anything we had in the past.

demon removed a subscriber: demon.Dec 16 2014, 3:57 PM
Qgil moved this task from To Triage to Need discussion on the Phabricator board.Dec 21 2014, 10:02 PM
Dzahn removed a subscriber: Dzahn.Dec 22 2014, 10:11 AM
Qgil added a comment.Jan 8 2015, 10:13 PM

Is there any action pending in this task?

This is ultimately a matter of priorities and resources, just like in any other ongoing project at the WMF. It has a component up aligning upstream vs maintaining local patches or forking, which again has nothing special for a non-Wikimedia OSS project that we are using, and goes again down to priorities and resources.

I think we can close this task as Resolved.

Qgil closed this task as Resolved.Jan 16 2015, 10:55 AM

Yesterday we discussed about Wikimedia Phabricator maintenance plans in the context of the Engineering Community team quarterly review (slides, meeting notes to be published soon).

A summary:

  • Wikimedia Phabricator is entering maintenance phase.
  • @Aklapper continues to work almost full time on Phabricator, and he has specific goals for consolidating Phabricator as project management tool during this quarter.
  • We have requested about 20% time from @chasemp and @mmodell. Note that there are other developers like Chad, Christopher, dzahn, Legoktm, Valhalla, etc, volunteering on specific areas.
  • We are committed to keep upstreaming requests and upgrading regularly. Being in sync with upstream is the only sensible way to go in the near future.
  • Maintaining our own extensions is onerous. We have good reasons to push for Wikimedia SUL, Security, and Sprint, but any new extension will require deep scrutiny and a resourcing plan.
  • The code review migration is a different story. As soon as we are ready to start planning for it, it will be treated as an own project, like the Bugzilla and RT migrations.

We are in sync with WMF engineering management. We can always modify this plan if better ideas are proposed.

And with this, I think we can resolve this task. Thank you everybody for the good discussion.

greg moved this task from INBOX to Done on the Release-Engineering-Team board.Jan 28 2015, 10:15 PM
Restricted Application added a subscriber: TerraCodes. · View Herald TranscriptMay 23 2016, 6:04 PM