Thank you! All this is food for https://www.mediawiki.org/wiki/Phabricator/Code. Let's concentrate all the documentation there.

One big change between Bugzilla upgrades and Phabricator upgrades is that we are defaulting to a weekly maintenance window, that we might use or not. The pace of development of Phabricator is impressive, and that includes fixes to problems that we have reported and patches that we are upstreaming. This means that we will upgrade at least once a month, possibly twice, maybe more if needed. This process only works when our local patches are kept to a minimum, and when we have each change properly maintained and in sync for each upgrade.

Meanwhile, in Bugzilla... upgrading was done basically when we had no other chance, maybe once a year or so. New Bugzilla releases came from time to time (Phabricator doesn't have releases, their master branch IS the release). Patching locally was basically the only option left to implement the fixes we needed, because upstream was slow, and our instance was definitely not among their priorities. I remember that the last big upgrade took a significant work to @Dzahn and @Aklapper.

All this to say that the Bugzilla and Phabricator games are very different. Both are third party tools used by many others, but you can see why we don't want to maintain a big set of local patches to be synced every other week.

Thank you for writing this.

I find the view that we're outright rejecting customizations to Phabricator to be simply unacceptable and completely unaligned with the reason that we chose to use Phabricator.

As I said on T69, a key selling point to switching to Phabricator was that (unlike, say, Gerrit), it's written in PHP and other familiar Web languages and we could easily make changes to it. If this isn't the case, then we need to seriously re-evaluate the decision to switch to Phabricator.

We cannot and we will not be held hostage by an external, upstream development team. This is the antithesis of free and open source software. If there are theme or functionality changes that we want, we should be able to implement those. If customizations fundamentally aren't possible using Phabricator, then Phabricator is the wrong tool for us.

If we discard the weekly updates and adopt a schedule similar to our Bugzilla instance, the maintenance cost will be much lower.

Thanks for laying all of this out, Mukunda.

I think it's ok for us to have a temporary policy of generally postponing local customizations. We haven't yet clearly spelled out our maintenance strategy for Phabricator, so it's smart for us to be conservative about incurring long-term maintenance obligations until we do. Note that I say "temporary" there. By the end of March 2015, we need to state what our long term policy is. I suspect we're going to have to have some customizations like we had to have for Bugzilla and Gerrit and pretty much every other tool that is as central to our development process as Phabricator is, but let's spend some time letting the dust settle on this migration and letting people get used to the largely uncustomized version before insisting on a bunch of changes.

We should be careful about "rejecting" these changes purely on the grounds that upstream doesn't currently look likely to take it up. We should evaluate these things based on our needs, and only truly reject those things that we can internalize a clear rationale for, independent of upstream's rationale. For those things we aren't willing to give up on that seem stalled on upstream, we need to minimally keep a list of those things, and come up with our strategy for eventually getting those things addressed.

What tasks are in this category of problem? @MZMcBride cites T69, which seems like one we should treat with some level of seriousness (I guess I should comment there directly). Is that an exceptional case, or are all of the issues like that? Is it useful to make a tracking task (or use this one) as a means of compiling a list so that we can have a concrete discussion over a list of concerns?

@RobLa-WMF: so far most of the requests have been via IRC, and I'm sure a few are being tracked in phabricator but I'll have to search to find them. I just wrote this task to help avoid repeating the response, and to give people a place to hash out the issues.

@MZMcBride: I don't think anyone is saying that we refuse to customize phabricator or that it's not fundamentally customizable, just that we need to balance the need for customization against the ongoing costs of maintaining them, and that we can't just say anything goes without regard for the people who have to deal with the consequences when our local hacks break because of a lack of forethought, or due to the inevitable conflicts with upstream.

Here are a few other customization suggestions/requests that we've either declined or set aside for future consideration or possibly pushing upstream for consideration by the phab team:

T81 T96 T129 T234 T257 T78357

Most of these are found under the #phabricator.org project tag, which is being used for collecting tasks that most likely need to be handled upstream rather than working on them ourself.

In T78464#846549, @RobLa-WMF wrote:

We should be careful about "rejecting" these changes purely on the grounds that upstream doesn't currently look likely to take it up. We should evaluate these things based on our needs, and only truly reject those things that we can internalize a clear rationale for, independent of upstream's rationale. For those things we aren't willing to give up on that seem stalled on upstream, we need to minimally keep a list of those things, and come up with our strategy for eventually getting those things addressed.

I strongly agree with all of this.

In T78464#846549, @RobLa-WMF wrote:

We haven't yet clearly spelled out our maintenance strategy for Phabricator

Related: T175

In T78464#845649, @Qgil wrote:

New Bugzilla releases came from time to time (Phabricator doesn't have releases, their master branch IS the release). [...] I remember that the last big upgrade took a significant work to @Dzahn and @Aklapper.

In T78464#846549, @RobLa-WMF wrote:

I suspect we're going to have to have some customizations like we had to have for Bugzilla

Our customizations for Bugzilla feel rather small and focused on very specific areas, compared to the numerous ideas and requests that have been posted about Phabricator itself already (note that I am not judging the quality of those requests) - see https://git.wikimedia.org/tree/wikimedia%2Fbugzilla%2Fmodifications.git .
Still it took several weeks (in parallel to my usual ongoing tasks) to adjust our custom code and CSS changes in Bugzilla and test all those changes before we upgraded from Bugzilla 4.2 to 4.4 approx. a year ago.

We don't have a dedicated phabricator team going

just to be clear about it, this is _not T175 ?

In T78464#846063, @MZMcBride wrote:

key selling point to switching to Phabricator was that (unlike, say, Gerrit), it's written in PHP and other familiar Web languages and we could easily make changes to it.

This has been my thought as well.

@Dzahn:

just to be clear about it, this is _not T175 ?

I simply meant to say that the phabricator team is not dedicated full-time to phabricator, at least not indefinitely.

In T78464#847841, @Dzahn wrote:

In T78464#846063, @MZMcBride wrote:

key selling point to switching to Phabricator was that (unlike, say, Gerrit), it's written in PHP and other familiar Web languages and we could easily make changes to it.

This has been my thought as well.

The key selling point you refer to was and is the possibility to easily contribute upstream. This has been our emphasis since the beginning. Keeping our local patches at a minimum has been another explicit emphasis, also since the beginning.

So here is a recent example:

1. @Krenair creates T78250 here, and after a short discussion he is encouraged to file the task upstream.
2. He then creates https://secure.phabricator.com/T6757
3. In 21 minutes (!) upstream sets priority to Low and gives specific instructions of the work needed.

Now we have to options. Wait until someone fixes that problem upstream, or contribute a patch ourselves. When a situation like this happened to Bugzilla or Gerrit, we basically relied in one or two people able to submit a patch. Today, we have dozens of developers that, given a special interest, could scratch that itch if they really want.

A few of those itches are more complex but also more relevant. While we focus our resources on those few tasks and we try to work as close with upstream as possible (getting a lot of work done by them directly), many others can step in to solve other problems. While this situation is not perfect and we all wish we would have more resources (a common theme in most software projects), the current situation is way better than anything we had in the past.

Is there any action pending in this task?

This is ultimately a matter of priorities and resources, just like in any other ongoing project at the WMF. It has a component up aligning upstream vs maintaining local patches or forking, which again has nothing special for a non-Wikimedia OSS project that we are using, and goes again down to priorities and resources.

I think we can close this task as Resolved.