Page MenuHomePhabricator

Transitioning Responsibility for MediaWiki Releases
Open, Needs TriagePublic

Description

WHAT?

What is the problem or opportunity?

Wikimedia Foundation staff invest time and resources into preparing twice yearly MediaWiki releases for third parties and are often hampered in rapidly evolving the MediaWiki codebase by being tied to those releases.

What does the future look like if this is achieved?

If responsibility for delivering MediaWiki third party releases were transitioned to an organization whose primary responsibility is serving third-party users, releases would become more predictable and would better satisfy the release consumers. Releases could be delivered in the form that best meets the users’ needs. Wikimedia Foundation time and resources dedicated to preparing the releases would be freed up. A more productive channel of communication would be created between core MediaWiki developers at the Wikimedia Foundation and the third party MediaWiki community.

What happens if we do nothing?

The Wikimedia Foundation would continue to invest time and resources into delivering third party MediaWiki releases. These releases will not evolve to better meet the consumers’ needs. Opportunity will be lost to tailor the deprecation policy to better accommodate rapid evolution of the code base.

WHY?

Identify the value(s) this problem/opportunity provides.

The timing of MediaWiki releases is currently unpredictable. There are few dedicated Wikimedia Foundation resources for producing the third party MediaWiki releases. Staff participate in the release process largely as a side effort, not an integral part of their assigned role. Thus, releases are generated as time permits.

Some core code changes must currently be done phased in over time. Per the Stable Interface Policy deprecation process, certain changes may not be made immediately but must go through a deprecation process tied to releases. Delays in MediaWiki releases push the timeline of deprecation back. This proposal would allow core MediaWiki developers to move more aggressively in some areas of the MediaWiki core codebase as more active involvement from the third party MediaWiki community and extension developers serving that community would provide feedback on critical changes, reducing unnecessary delays.

Third party users of MediaWiki would like a more predictable release schedule:

  • WMDE (to base releases of Wikibase on MediaWIki third party releases)
  • Extension developers and developers of MediaWiki-focused administration tools (to better plan compatibility releases)
  • Administrators of MediaWiki sites (to better predict and resource their upgrade schedules)

The MediaWiki Stakeholders’ Group is willing to take on the responsibility to prepare the releases as part of its mission to support the MediaWiki third party community. From mediawiki.org / mwstake.org, the MediaWiki Stakeholders’ Group is “a MediaWiki user group consisting of MediaWiki developers, system administrators, users, consultants, and hosting providers who cooperate in order to improve the software and advocate the needs of MediaWiki users outside the Wikimedia Foundation (WMF) and its projects.” It was recognized by the Affiliations Committee on November 11, 2014 and incorporated in 2020 (charter / bylaws / board). It was founded by @MarkAHershberger and @Mglaser, who at the time were contracted by the Wikimedia Foundation to deliver the third party MediaWiki releases, so they have a track record of providing quality MediaWiki releases on a predictable schedule. The MediaWiki Stakeholders’ Group hosts monthly membership meetings and, with incorporation and the commencement of collecting dues from members, planning initiatives to benefit third party MediaWiki community. It also maintains a repository pointing to third-party MediaWiki extensions and skins that are indexed by codesearch.

Why are you bringing this decision to the Technical Forum?

There are a lot of details involved in making a MediaWiki release. Care has been taken to identify the key activities and pain points, but it is important to make sure that we have identified all of the relevant stakeholders and any subtle details that need to be considered.

Event Timeline

@CCicalese_WMF could you clarify if this is just for stable releases, e.g. 1.37.0, or also security/maintenance releases? And is this about switching the responsibility for organizing the release (shepherding and backporting patches), or the actual technical release (running make-release2.py) or both?

@CCicalese_WMF could you clarify if this is just for stable releases, e.g. 1.37.0, or also security/maintenance releases?

Certain aspects of security releases would still need to be done in house, but the goal would be to transition as much as possible of the mechanics.

And is this about switching the responsibility for organizing the release (shepherding and backporting patches), or the actual technical release (running make-release2.py) or both?

Both.

There are myriad details that will need to be worked out and documented. That would be the next phase of this process. Members of the MediaWiki Stakeholders' Group plan to participate in the upcoming MediaWiki 1.37 release in order to gather enough information to generate a detailed task list with assignment of responsibilities that would form the basis of a Memorandum of Understanding.

AIUI the proposal names these goals/opportunities

  1. More predictable release schedule.
  2. A better release format(?)
  3. WMF would have more time and resources for other things.
  4. Improved communication between WMF developers and the 3rd-party MediaWiki community.
  5. MediaWiki developers would not be slowed down by the Stable Interface Policy

I can see how #4 would work (I think MWStake has already been fairly successful at facilitating better exchange of information between the WMF and some parts of the developer community), though with the caveat that MWStake has positioned itself into representing a specific part of the MediaWiki community (professional MediaWiki developers and organizations with dedicated MediaWiki staff), and doesn't have much track record representing others. (Although one could argue those are exactly the ones for whom a better release process would be important.) I'm not sure I see the connection to the other four stated goals/benefits though.

#2 and #5 aren't really related to which organization does the releases. I guess the implication here is that MWStake would rethink the release format and the associated policies and come up with something better; it's hard to evaluate that without any specifics though, and it could be done with or without transitioning release management to another org.

#3 is a matter of funding. Presumably WMF staff doing the releasing is not especially expensive; another organization would incur comparable costs by doing it. If that's covered from Wikimedia movement funds, the WMF will not have more time and resources; they will just be spent elsewhere. The alternative is MWStake doing it on a volunteer or partially volunteer basis (in which case goals like a more predictable schedule seem somewhat over-ambitious), or covering it from some other source of funding, presumably from companies using MediaWiki. The latter would be a reasonable long-term goal, but how realistic is it today? In 2014, the WMF paid $75K per year for release management; MWStake was incorporated last year, and guessing from the sponsors and fees page its current industry funding might be somewhere in the $5K-$25K range. So that seems fairly ambitious.

#1 is partily a matter of sufficient resourcing (see above), partly of organizational maturity (with all due respect to MWStake, it would not be an improvement in that regard), partly of focus. I can see how MWStake could pay more attention to release management than the WMF does currently; but I doubt releasing could be done without coordinating with developers of MediaWiki core, the most active of whom are WMF staff members. So communication and coordination would become more complicated; one problem would be traded for another.

All that is not to say I would think such a transfer of responsibilities a bad idea (having a dedicated org in the Wikimedia universe to focus on third-party MediaWiki users makes strategic sense), I'm just not sure about the benefits listed here.

As I understand this proposal -- and maybe this is naive optimism -- the idea would be that MW would still branch on a rigorous schedule (every six months, with a "LTS" every two years) but that it would be hands-off past that point; all the shepherding necessary to actually alpha/beta/rc package-and-test this would be handed-off to MWStake. I agree with @Tgr that I'm not sure this isn't just moving money and time from one pile to another, but if there are other good reasons to do this, it would indeed (mildly) accomplish goal #5 in so far as deprecations and removals could be done "like clockwork" on master after the release is branched, even if (due to reality intruding) the actual *release* of that branched code was delayed.

I will note that there are a number of "core" extensions which maintain compatibility with LTS releases, not just "current master". Translate is probably the most important of these. I would like to see responsibility for maintaining LTS compatibility for these extensions transition as well. Whenever I have to make a deprecation-policy-type-of-change (say, just renaming a method from foo to bar) it's an easy codesearch-and-replace for about 90% of the code covered by codesearch, but that last 10% is a slog: it includes both code which is covered by codesearch but has a non-WMF CI system, as well as code which maintains LTS compatibility (for which just bumping the required version in extension.json is not sufficient)... and some code which does both (maintains LTS compatibility in a non-WMF system). (I know that policy says I'm not *required* to do anything to facilitate/port most of the code which falls in that 10%, but I try to break as few things as possible.)

Thank you for the feedback, @Tgr and @cscott. Note that this task is related to the Problem Statement phase of the process. As noted at T293323#7430328, specific solutions will be addressed in the next phase. That being said, your feedback is very valuable in helping to shape the decisions.

#3 is a matter of funding. Presumably WMF staff doing the releasing is not especially expensive; another organization would incur comparable costs by doing it. If that's covered from Wikimedia movement funds, the WMF will not have more time and resources; they will just be spent elsewhere.

AIUI, this proposal does not say that the releases would be covered by WMF movement funds. That was tried in the past and rejected.

The alternative is MWStake doing it on a volunteer or partially volunteer basis (in which case goals like a more predictable schedule seem somewhat over-ambitious), or covering it from some other source of funding, presumably from companies using MediaWiki. The latter would be a reasonable long-term goal, but how realistic is it today?

MWStake has two different resources that it will use to handle releases:

  1. MWStake members who will provide their own paid staff to this task initially. That is realistic today since we have enough commitment for that now.
  2. Raising money from users of the MW distribution to support these roles. In the aspect of predictable releases, that would make MWStake accountable to users of the MW releases. Right now, there is no way to directly tie contributions to the WMF to MW releases if that is what you are interested in supporting. Moving the funding source to MWStake would provide users who want to support the software a way to do that. I have had clients ask me if there was a way to do this in the past, and this would open up a channel for people to do this.

In 2014, the WMF paid $75K per year for release management; MWStake was incorporated last year, and guessing from the sponsors and fees page its current industry funding might be somewhere in the $5K-$25K range. So that seems fairly ambitious.

MWStake has corporate members who have been users and providers of MW services for several years. It is these members who will provide the initial support for releases through personnel and financial support. Once MWStake has established itself as the distributor for MW, it expects to be able to tap into other resources.

I have two concerns:

I'm concerned with the 'optics' of WMF stepping away from producing releases for the software to the community of Enterprises and Institutions who use the software for Knowledge Management. I would think libraries, software projects, enterprises and others would be more apprehensive about using the software for any serious use without the "vendor" supplying official releases.

One of the stated friction points is the ability to rapidly evolve the codebase against the requirements of the stable interface policy. If the WMF wants to rapidly evolve the code, and isn't responsible for producing releases, doesn't this mean that MWStake would be disadvantaged in their ability to produce regular 6mo releases that do obey the stable interface policy? In other words, it seems the real issue is to evolve the stable interface policy by announcing the intention to break compatibility more often; help extension developers keep their code compatible; and 3rd-parties should be prepared for more rigorous upgrade efforts.

This question is being posed as part of the technical decision forum process which has yet to fill any community seats. Some of us try to proxy vote for what we understand technical community including the poorly defined "third party" users are interested in, but without actual representation of these groups in the TDF process this proposal seems more like an internal Foundation referendum on discontinuing tarball releases than a MediaWiki community discussion on adjusting release practices and responsible parties.

@bd808 fwiw, I'm one of the community seats on the TDF. I would love to talk to you offline about this if you want to discuss more. MediaWiki-Stakeholders-Group is not a perfect representative for 3rd party users, but we're trying.

@bd808 fwiw, I'm one of the community seats on the TDF.

This membership is not disclosed on https://www.mediawiki.org/wiki/Technical_Decision_Forum#Forum_Members

fwiw, I'm one of the community seats on the TDF

[off-topic] Huh? That page describes a public appointment process. Where did this happen? Why have I not heard anything about it?

I have two concerns:

I'm concerned with the 'optics' of WMF stepping away from producing releases for the software to the community of Enterprises and Institutions who use the software for Knowledge Management.

There is a precedent for the releases being produced outside WMF in the past. The plan is for there to be a document that will lay out the responsibilities for releases, and WMF will retain some of those responsibilities, such as providing the infrastructure and accounts. The WMF is not stepping away from producing releases but instead would be allowing an organization with more involvement in the community of non-Wikimedia MediaWiki users to do so for the benefit of that community.

I would think libraries, software projects, enterprises and others would be more apprehensive about using the software for any serious use without the "vendor" supplying official releases.

The WMF is not the vendor in a traditional sense. Their primary interest in MediaWiki is rightly related to its use for Wikimedia projects, and they are not consumers of the current tarball releases. There is a benefit to the releases being produced by an organization more aligned to the needs of the consumers of those releases. That being said, the MediaWiki Stakeholders' Group would definitely benefit from participation from a broader set of those consumers.

One of the stated friction points is the ability to rapidly evolve the codebase against the requirements of the stable interface policy. If the WMF wants to rapidly evolve the code, and isn't responsible for producing releases, doesn't this mean that MWStake would be disadvantaged in their ability to produce regular 6mo releases that do obey the stable interface policy? In other words, it seems the real issue is to evolve the stable interface policy by announcing the intention to break compatibility more often; help extension developers keep their code compatible; and 3rd-parties should be prepared for more rigorous upgrade efforts.

There will still be a stable interface policy. It would be revisted to see if there are any cases in which it could be relaxed (e.g. if there is no affected code remaining in all places noted by codesearch - which would admittedly leave out private repos, so would need discussion). This would limit the impact on third parties by ensuring that dependent code had actually been remediated already before a release. This may also result in strengthening the stable interface policy is some places (e.g. banning new deprecations within X weeks of a release being cut). And, it would provide an opportunity to revisit some current confusion and annoyances about extension/skin compatibility policies and how extension/skin branches are cut and maintained. The intention is for this to be done in a way that provides a net benefit to the consumers of the releases.

Note: I've edited the task description so that it correctly includes "users" in the make up of the MediaWiki Stakeholder's group as the front page of mwstake.org does.

Hello, thanks for the problem statement, and the discussion.

I've said it in the feedback form, but seeing discussion here, I realize that the phabricator comment might be preferred in case of the problem statement overview.
I have to admit I fail to identify what really is the problem that this problem statement aims to provide an overview of? I believe @Tgr had a good take on attempting to list few possible problems you folks might mean here, I could possible see a few more in the text. Is it just me, or would it actually make sense to be more clear about the WHAT?

Discussion in the comments above seems to get into the details of a possible solution which was rightfully pointed out is not part of the "problem statement" phase. I do see however that even the problem statement seems to be mixing up the solution and the problem(s). Even the problem statement's title "Transitioning Responsibility for MediaWiki Releases" is clearly in the solution land, isn't it?

To finish on the constructive note - I don't want to come across as a complainer, I appreciate the effort put into preparing this: WMDE clearly is a stakeholder ("C" in terms of RACI matrix I'd say) in many possible topics related to Mediawiki releases. We will be more than happy to provide input on our requirements (on the later stage), but also to help identify issues with the current Mediawiki releasing process, as we certainly could name one or two. I do not understand however if that problem statement is the call to action "there are issues with Mediawiki releases, let's identify them". If there was an interest to define the problem statement (maybe there are mutliple problems here even), I'd happily serve as a rubber duck to bounce any ideas against.

Thank you for the feedback, @WMDE-leszek. I do think that your comments about the problem statement venturing into the solution space are fair. Part of the problem is the questions which must be answered. "What does the future look like if this is achieved?" does not define the problem but rather the world in which the problem has been solved somehow. It is difficult to address that without at least hinting at a solution.

While I would prefer that the process not get stalled on revising this problem statement at this point, I would indeed be very interested to hear your thoughts on any additional issues with the current MediaWiki releasing process that should be considered. Please feel free to share them here. And, without a doubt, WMDE's input on the proposed solution will be welcome and quite valuable.

I primarily depend upon tarball MediaWiki releases in my role as Debian maintainer of the "mediawiki" package. As part of this work, I've spent a decent amount of time improving the release process by working with Chad, Reedy, and others including writing make-release2.py, to fix many problems we've had with generating the tarballs, figuring out the /vendor process, weird misc issues like T263866 and T257102 and more. Then there's stuff not directly related to the release but IMO had a significant impact like getting Postgres running in CI to prevent regressions. The Debian package has a full end-to-end installer-to-editing test suite that time permitting is run against tarballs before they're released and have caught real regressions in the past.

My main priority is ensuring the security release process remains functional and stable with minimal regressions, which is the main expectation in Debian. Things have improved significantly with the regularly scheduled quarterly releases, CVEs assigned ahead of time, a tracking table so patches don't accidentally get dropped, but we are still in need of better testing (e.g. T226945, T270452, T263883), and more people poking at this is exactly what we need.

I believe that 90% of improvements to the release process and releases themselves can be done by just having people contribute the way people already do to MediaWiki - commenting on bugs, discussing on mailing lists, and submitting patches. If MWStake wants to contribute more to releases, that's fantastic, really. Please, just start helping :) ...and then we can empower more people to do more advanced work that requires specific permissions or roles.


Some specific comments:

The timing of MediaWiki releases is currently unpredictable. There are few dedicated Wikimedia Foundation resources for producing the third party MediaWiki releases. Staff participate in the release process largely as a side effort, not an integral part of their assigned role. Thus, releases are generated as time permits.

Please provide some evidence for "unpredictable". AFAIK all security releases have been roughly on time, the only things that have slipped are stable releases. Some of those slips were pretty reasonable and understandable because of COVID, with outstanding blockers preventing a release from going out. Most MWStake members are not MediaWiki *core* developers AFAICT - what is the plan for getting those blockers fixed when they inevitably arise?

I might be too deep in the Debian mindset ("it'll be released when it's ready"), but what's the end goal if we do have predictable releases? Even then I wouldn't expect all or most extensions to immediately support the new version, the ecosystem takes time to update, just like when a new Python version comes out, it takes a bit of time for various Python libraries to publish their compatible versions.

Some core code changes must currently be done phased in over time. Per the Stable Interface Policy deprecation process, certain changes may not be made immediately but must go through a deprecation process tied to releases. Delays in MediaWiki releases push the timeline of deprecation back. This proposal would allow core MediaWiki developers to move more aggressively in some areas of the MediaWiki core codebase as more active involvement from the third party MediaWiki community and extension developers serving that community would provide feedback on critical changes, reducing unnecessary delays.

It is quite surprising to hear that MWStake wants MediaWiki core to move more aggressively in deprecating and making breaking changes - I was under the impression that the exact opposite was wanted. I also don't see how this is a problem that will be fixed here, the branching schedule is run by Release Engineering, who make the bump from e.g. 1.37.0-wmf.X to 1.38.0-wmf.1, which is what enables people to merge the next round of deprecations and removals.