Page MenuHomePhabricator

Determine a standard way of installing MediaWiki lib/extension dependencies within containers
Open, LowPublic

Description

With the near-term goal of deploying minimal MediaWiki containers alongside services in the release pipeline for the purpose of e2e testing services, and the eventual goal of testing and deploying MediaWiki and its extensions via the pipeline to production, we'll need to somehow have Blubber install MediaWiki dependencies (both libraries and extensions) in some sane manner.

We could potentially build support directly into Blubber but this coupled approach seems wrong given Blubber's current design (stateless and preferring delegation to package managers over intimate knowledge of resident applications), and it also would seem like a missed opportunity to simplify the installation of MediaWiki extensions through something standard like composer.

Some ideas so far:

  1. Try to wrangler composer into natively installing extensions in addition to aggregate library dependencies (through the merge plugin).
  2. Write a composer plugin that understands the extension.json schema and can recursively resolve and install extensions. Note that the extension.json schema does not currently allow you to specify test/dev dependencies.
  3. Write some wrapper tooling that can recursively resolve and install MW extensions and leave library dependencies to composer.

In any case, Blubber would provide configuration to properly invoke the underlying dependency manager(s) much the same way it provides configuration for Node and Python dependency managers (npm and pip respectively).

Event Timeline

dduvall created this task.May 4 2018, 1:12 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 4 2018, 1:12 AM
dduvall triaged this task as Normal priority.
dduvall updated the task description. (Show Details)May 19 2018, 11:11 AM

I don't have a full picture of what the deployment container stuff is going to look like, so please correct me if my assumptions are wrong.

and it also would seem like a missed opportunity to simplify the installation of MediaWiki extensions through something standard like composer.

My main suggestion here would be not to do that. As someone who has spent a decent amount of time trying to simplify the installation of MediaWiki extensions, (and probably spent just as much time trying to make composer work), I don't think composer fits the model of what we need out of a manager for MediaWiki extensions. Whether it be security, privacy, naming, version management, branches, etc. It could probably be made to work, and people have. But that's also sacrificed a lot, notably breaking the installer. And just generally, dependency managers and package managers are hard problems. My current plan is to take the input from the current RfC, turn it into a plan that addresses most needs, and then build something that works, has TechCom approval, and can be endorsed as "official".

And, I don't think composer even gives you what you want here - since composer really just downloads extensions. You still need to manually come up with the dependency map inside of MediaWiki (assuming you want realistic e2e tests and aren't planning to enable every extension (that won't work for other reasons)).

The "standard" way of installing MediaWiki extensions is to get library dependencies from mediawiki/vendor, and then use Git to install the version of the extension you want. You'll have to manage extension dependencies manually, but given how long we've been doing that for, I don't think it's an extremely difficult burden. composer currently has no support for integrity verification, which is why we don't use composer-merge-plugin in production. Additionally, we've started to post process the composer output before deploying it (T194646).

Sidenote: We currently don't actually have true dependency data (for what you're trying to do I think). We have hard dependencies in extension.json, and we have test dependencies in CI, but those test dependencies also include dependencies that are just needed for static analysis. But I think what you want is practical dependencies, which is what extensions you actually need to have installed to make the extension useful (closest comparison is Recommends in the Debian world).

dduvall added a comment.EditedMay 30 2018, 10:34 PM

Thanks, @Legoktm. We seem to be working at an intersection of different efforts and use cases—and a glaring absence of sufficient tooling—so there are bound to be conflicting requirements put forth as we continue work on the CD Pipeline. Your experience with composer, particularly that which relates to MediaWiki extension management, is greatly appreciated.

I don't have a full picture of what the deployment container stuff is going to look like, so please correct me if my assumptions are wrong.

Hopefully the tasks related to this one can provide more context for our requirements for extension/skin dependencies for the pipeline, particularly T193777: FY2017/18-Q4: Prove viability of testing staged service containers alongside MediaWiki extension containers.

and it also would seem like a missed opportunity to simplify the installation of MediaWiki extensions through something standard like composer.

My main suggestion here would be not to do that. As someone who has spent a decent amount of time trying to simplify the installation of MediaWiki extensions, (and probably spent just as much time trying to make composer work), I don't think composer fits the model of what we need out of a manager for MediaWiki extensions. Whether it be security, privacy, naming, version management, branches, etc. It could probably be made to work, and people have. But that's also sacrificed a lot, notably breaking the installer. And just generally, dependency managers and package managers are hard problems. My current plan is to take the input from the current RfC, turn it into a plan that addresses most needs, and then build something that works, has TechCom approval, and can be endorsed as "official".

A couple of us from Release Engineering attended the RfC hackathon session and made a suggestion for the kind of thing that might help push our work forward.

Is there a rough timeline for work from that RfC? Regardless of what we implement or adopt for the pipeline in the coming quarters, it's great to know there will be a comprehensive and official solution to extension management that we can eventually switch over to. That said, we'd rather not rely on unplanned work in setting our own goals for the pipeline.

It was probably a mistake to include the "simplify the installation of MediaWiki extensions" bit in the task's description, as it's not a chief goal of Release Engineering or a concern of the pipeline. It was stated as a purported side benefit, but that statement didn't incorporate the realities and history of work on MW extension management that you have rightly pointed out.

And, I don't think composer even gives you what you want here - since composer really just downloads extensions. You still need to manually come up with the dependency map inside of MediaWiki (assuming you want realistic e2e tests and aren't planning to enable every extension (that won't work for other reasons)).

Composer's ability to recursively resolve a complete dependency graph certainly does seem limited. The experiments I was running used its vcs repo support to resolve/install extensions from our git repos, but then I noticed that Composer can't recursively resolve dependencies from repos so that approach seems like a bust.

My next thought was that it might be possible to build in support for recursion into repos, but that brings me to a question for you: Was upstream responsive to feature requests or patches?

The "standard" way of installing MediaWiki extensions is to get library dependencies from mediawiki/vendor, and then use Git to install the version of the extension you want. You'll have to manage extension dependencies manually, but given how long we've been doing that for, I don't think it's an extremely difficult burden.

Right. It doesn't seem like a difficult burden when done manually. However a manual process of any complexity is incompatible with the pipeline as we currently envision it.

To satisfy our requirement for automated integration testing of a change to a MediaWiki extension alongside its complementary or required services, we think we need a system that can take only a patchset to core or to an extension (e.g. VE) as input, produce a working container image (e.g. of core + VE + its lib and hard extension/skin dependencies), deploy that image to an isolated k8s namespace along with services (e.g. Parsoid, RESTBase), and perform limited but broad e2e testing. The current submodule and vendor-repo approach does not fit well with this design because:

  1. extension management via submodules is only a thing for core, and only for manually maintained release branches, not master
  2. the external and tightly coupled mapping in integration/config to manage test-level dependencies is a pattern we'd rather avoid as it prohibits the repo-authoritative model mentioned above
  3. the vendor repo is larger than necessary and works against keeping image sizes small, and it also prohibits the repo-authoritative model mentioned above

Sidenote: We currently don't actually have true dependency data (for what you're trying to do I think). We have hard dependencies in extension.json, and we have test dependencies in CI, but those test dependencies also include dependencies that are just needed for static analysis. But I think what you want is practical dependencies, which is what extensions you actually need to have installed to make the extension useful (closest comparison is Recommends in the Debian world).

During our previous offsite, Release Engineering also (briefly) looked at the extension.json schema (mentioned in the task description) and we noticed that same lack of test-/dev-level dependencies and suggests/recommends functionality. Do you see that schema being extended at any point? Do you have any thoughts/advice on ideas 2 and 3 mentioned in the task, either building something integrated with composer that understands extension.json or a completely separate tool?

Vvjjkkii renamed this task from Determine a standard way of installing MediaWiki lib/extension dependencies within containers to bndaaaaaaa.Jul 1 2018, 1:12 AM
Vvjjkkii raised the priority of this task from Normal to High.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: Aklapper.
thcipriani updated the task description. (Show Details)Jul 1 2018, 6:41 PM
thcipriani renamed this task from bndaaaaaaa to Determine a standard way of installing MediaWiki lib/extension dependencies within containers.
CommunityTechBot lowered the priority of this task from High to Normal.Jul 5 2018, 6:36 PM
thcipriani lowered the priority of this task from Normal to Low.
thcipriani moved this task from Backlog to Migration on the Release Pipeline board.