Page MenuHomePhabricator

Determine a standard way of installing MediaWiki lib/extension dependencies within containers
Open, LowPublic

Description

With the near-term goal of deploying minimal MediaWiki containers alongside services in the release pipeline for the purpose of e2e testing services, and the eventual goal of testing and deploying MediaWiki and its extensions via the pipeline to production, we'll need to somehow have Blubber install MediaWiki dependencies (both libraries and extensions) in some sane manner.

We could potentially build support directly into Blubber but this coupled approach seems wrong given Blubber's current design (stateless and preferring delegation to package managers over intimate knowledge of resident applications), and it also would seem like a missed opportunity to simplify the installation of MediaWiki extensions through something standard like composer.

Some ideas so far:

  1. Try to wrangler composer into natively installing extensions in addition to aggregate library dependencies (through the merge plugin).
  2. Write a composer plugin that understands the extension.json schema and can recursively resolve and install extensions. Note that the extension.json schema does not currently allow you to specify test/dev dependencies.
  3. Write some wrapper tooling that can recursively resolve and install MW extensions and leave library dependencies to composer.

In any case, Blubber would provide configuration to properly invoke the underlying dependency manager(s) much the same way it provides configuration for Node and Python dependency managers (npm and pip respectively).

Event Timeline

dduvall created this task.May 4 2018, 1:12 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 4 2018, 1:12 AM
dduvall triaged this task as Normal priority.
dduvall updated the task description. (Show Details)May 19 2018, 11:11 AM

I don't have a full picture of what the deployment container stuff is going to look like, so please correct me if my assumptions are wrong.

and it also would seem like a missed opportunity to simplify the installation of MediaWiki extensions through something standard like composer.

My main suggestion here would be not to do that. As someone who has spent a decent amount of time trying to simplify the installation of MediaWiki extensions, (and probably spent just as much time trying to make composer work), I don't think composer fits the model of what we need out of a manager for MediaWiki extensions. Whether it be security, privacy, naming, version management, branches, etc. It could probably be made to work, and people have. But that's also sacrificed a lot, notably breaking the installer. And just generally, dependency managers and package managers are hard problems. My current plan is to take the input from the current RfC, turn it into a plan that addresses most needs, and then build something that works, has TechCom approval, and can be endorsed as "official".

And, I don't think composer even gives you what you want here - since composer really just downloads extensions. You still need to manually come up with the dependency map inside of MediaWiki (assuming you want realistic e2e tests and aren't planning to enable every extension (that won't work for other reasons)).

The "standard" way of installing MediaWiki extensions is to get library dependencies from mediawiki/vendor, and then use Git to install the version of the extension you want. You'll have to manage extension dependencies manually, but given how long we've been doing that for, I don't think it's an extremely difficult burden. composer currently has no support for integrity verification, which is why we don't use composer-merge-plugin in production. Additionally, we've started to post process the composer output before deploying it (T194646).

Sidenote: We currently don't actually have true dependency data (for what you're trying to do I think). We have hard dependencies in extension.json, and we have test dependencies in CI, but those test dependencies also include dependencies that are just needed for static analysis. But I think what you want is practical dependencies, which is what extensions you actually need to have installed to make the extension useful (closest comparison is Recommends in the Debian world).

dduvall added a comment.EditedMay 30 2018, 10:34 PM

Thanks, @Legoktm. We seem to be working at an intersection of different efforts and use cases—and a glaring absence of sufficient tooling—so there are bound to be conflicting requirements put forth as we continue work on the CD Pipeline. Your experience with composer, particularly that which relates to MediaWiki extension management, is greatly appreciated.

I don't have a full picture of what the deployment container stuff is going to look like, so please correct me if my assumptions are wrong.

Hopefully the tasks related to this one can provide more context for our requirements for extension/skin dependencies for the pipeline, particularly T193777: FY2017/18-Q4: Prove viability of testing staged service containers alongside MediaWiki extension containers.

and it also would seem like a missed opportunity to simplify the installation of MediaWiki extensions through something standard like composer.

My main suggestion here would be not to do that. As someone who has spent a decent amount of time trying to simplify the installation of MediaWiki extensions, (and probably spent just as much time trying to make composer work), I don't think composer fits the model of what we need out of a manager for MediaWiki extensions. Whether it be security, privacy, naming, version management, branches, etc. It could probably be made to work, and people have. But that's also sacrificed a lot, notably breaking the installer. And just generally, dependency managers and package managers are hard problems. My current plan is to take the input from the current RfC, turn it into a plan that addresses most needs, and then build something that works, has TechCom approval, and can be endorsed as "official".

A couple of us from Release Engineering attended the RfC hackathon session and made a suggestion for the kind of thing that might help push our work forward.

Is there a rough timeline for work from that RfC? Regardless of what we implement or adopt for the pipeline in the coming quarters, it's great to know there will be a comprehensive and official solution to extension management that we can eventually switch over to. That said, we'd rather not rely on unplanned work in setting our own goals for the pipeline.

It was probably a mistake to include the "simplify the installation of MediaWiki extensions" bit in the task's description, as it's not a chief goal of Release Engineering or a concern of the pipeline. It was stated as a purported side benefit, but that statement didn't incorporate the realities and history of work on MW extension management that you have rightly pointed out.

And, I don't think composer even gives you what you want here - since composer really just downloads extensions. You still need to manually come up with the dependency map inside of MediaWiki (assuming you want realistic e2e tests and aren't planning to enable every extension (that won't work for other reasons)).

Composer's ability to recursively resolve a complete dependency graph certainly does seem limited. The experiments I was running used its vcs repo support to resolve/install extensions from our git repos, but then I noticed that Composer can't recursively resolve dependencies from repos so that approach seems like a bust.

My next thought was that it might be possible to build in support for recursion into repos, but that brings me to a question for you: Was upstream responsive to feature requests or patches?

The "standard" way of installing MediaWiki extensions is to get library dependencies from mediawiki/vendor, and then use Git to install the version of the extension you want. You'll have to manage extension dependencies manually, but given how long we've been doing that for, I don't think it's an extremely difficult burden.

Right. It doesn't seem like a difficult burden when done manually. However a manual process of any complexity is incompatible with the pipeline as we currently envision it.

To satisfy our requirement for automated integration testing of a change to a MediaWiki extension alongside its complementary or required services, we think we need a system that can take only a patchset to core or to an extension (e.g. VE) as input, produce a working container image (e.g. of core + VE + its lib and hard extension/skin dependencies), deploy that image to an isolated k8s namespace along with services (e.g. Parsoid, RESTBase), and perform limited but broad e2e testing. The current submodule and vendor-repo approach does not fit well with this design because:

  1. extension management via submodules is only a thing for core, and only for manually maintained release branches, not master
  2. the external and tightly coupled mapping in integration/config to manage test-level dependencies is a pattern we'd rather avoid as it prohibits the repo-authoritative model mentioned above
  3. the vendor repo is larger than necessary and works against keeping image sizes small, and it also prohibits the repo-authoritative model mentioned above

Sidenote: We currently don't actually have true dependency data (for what you're trying to do I think). We have hard dependencies in extension.json, and we have test dependencies in CI, but those test dependencies also include dependencies that are just needed for static analysis. But I think what you want is practical dependencies, which is what extensions you actually need to have installed to make the extension useful (closest comparison is Recommends in the Debian world).

During our previous offsite, Release Engineering also (briefly) looked at the extension.json schema (mentioned in the task description) and we noticed that same lack of test-/dev-level dependencies and suggests/recommends functionality. Do you see that schema being extended at any point? Do you have any thoughts/advice on ideas 2 and 3 mentioned in the task, either building something integrated with composer that understands extension.json or a completely separate tool?

Vvjjkkii renamed this task from Determine a standard way of installing MediaWiki lib/extension dependencies within containers to bndaaaaaaa.Jul 1 2018, 1:12 AM
Vvjjkkii raised the priority of this task from Normal to High.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: Aklapper.
thcipriani updated the task description. (Show Details)Jul 1 2018, 6:41 PM
thcipriani renamed this task from bndaaaaaaa to Determine a standard way of installing MediaWiki lib/extension dependencies within containers.
CommunityTechBot lowered the priority of this task from High to Normal.Jul 5 2018, 6:36 PM
thcipriani lowered the priority of this task from Normal to Low.
thcipriani moved this task from Backlog to Migration on the Release Pipeline board.

We have filled this task following Release-Engineering-Team May 2018 offsite. One of the need is for CI to be able to **optionally* install dependent extensions. The current system is centrally managed in integration/config.git and suffers from several issues:

  • requires review/merge/deploy from one of the CI maintainer
  • is not aware of branches

That is expressed on MediaWiki.org Extension_management_2018_feedback#Programatically_install_extension_dependencies:

Currently Wikimedia CI relies on a mapping inside the integration/config repository to determine which extensions depend on other extensions within the context of CI; however, outside of our current testing setup this is not usable. Allowing extensions to define those other extensions on which they depend would be generally useful as well as useful for future development of an extension-testing pipeline. TCipriani (WMF) (talk) 14:27, 19 May 2018 (UTC)

Quoting the 3 points Dan wrote originally:

  1. Try to wrangler composer into natively installing extensions in addition to aggregate library dependencies (through the merge plugin).
    • @Legoktm explained above how we cant/dont want to use composer for that.
  1. Write a composer plugin that understands the extension.json schema and can recursively resolve and install extensions. Note that the extension.json schema does not currently allow you to specify test/dev dependencies.
  2. Write some wrapper tooling that can recursively resolve and install MW extensions and leave library dependencies to composer.

The two other options are about introducing a new wrapper. Either as a composer plugin (2) or some CLI utility (3). I think we can just ship a maintenance utility in mediawiki/core that would take care of that, eventually make it a composer plugin later on if need be.


With the extension registry, developers are able to express the dependency upon another extension (from MediaWiki Manual:Extension_registration) an example would be:

extension.json
{
	"requires": {
		"extensions": {
			"FakeExtension": "*"
		}
	}
}

So when currently CI centrally defines:

zuul/parameter_functions.py
dependencies = {
    '3D': ['MultimediaViewer'],
    'MultimediaViewer': ['BetaFeatures'],
}

When one send a patch to mediawiki/extensions/3D, CI build the list of dependencies and inject to the build something like:

EXT_DEPENDENCIES=mediawiki/extensions/MultimediaViewer\\nmediawiki/extensions/BetaFeatures

Instead we should just clone the 3D extension, then run the new CLI utility which would look at the requirements defined in extension.json:

extension.json
{
	"requires": {
		"extensions": {
			"MultimediaViewer": "*"
		}
	}
}

Clone the required extensions/skins and recursively process. Eg MultimediaViewer would be cloned and would need to have:

extension.json
{
	"requires": {
		"extensions": {
			"BetaFeatures": "*"
		}
	}
}

The devil is that on CI the cloning will have to be done using zuul-cloner in order to checkout the proper branch/patch. And we would probably need to have Quibble to handle it for us since that is the main test runner nowadays.

Hence. My proposal is to have the feature implemented directly into Quibble with a feature flag. In this mode, it will ignore the legacy EXT_DEPENDENCIES environment variable (or even dies if it is present), then clone the repositories that are passed as argument and recursively process each of them:

quibble --process-extension-requirements mediawiki/extensions/3D
INFO: cloning mediawiki/extensions/3D
INFO: checking it out patch/branch from Zuul
INFO: 3D depends on MultimediaViewer

INFO: cloning mediawiki/extensions/MultimediaViewer
INFO: checking it out patch/branch from Zuul
INFO: MultimediaViewer depends on BetaFeatures

INFO: cloning mediawiki/extensions/BetaFeatures
INFO: checking it out patch/branch from Zuul

... run composer merge plugin ...
.. install mediawiki ...

And this way we can phase out the dependencies from zuul/parameter_functions.py. Thoughts?

Change 502286 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/quibble@master] Clone requirements from ext dependencies

https://gerrit.wikimedia.org/r/502286

In general I think that's a good idea. Two things I think we'll need to do:

  1. Add support for require-dev to extension.json, since CI needs to know the real dependencies and the test dependencies.
  2. Provide some script to abstract the reading of extension.json since the format isn't guaranteed long-term.

For #2, here's what I have locally right now:

km@km-pt:/srv/mediawiki/core$ php maintenance/checkDependencies.php --extensions=CodeEditor,Scribunto,ApiFeatureUsage,MassMessage --skins=Timeless --json | json_pp
{
   "skins" : {
      "Timeless" : {
         "status" : "loaded"
      }
   },
   "extensions" : {
      "Elastica" : {
         "status" : "missing",
         "why" : [
            "ApiFeatureUsage"
         ]
      },
      "ApiFeatureUsage" : {
         "why" : [
            "ApiFeatureUsage"
         ],
         "status" : "present"
      },
      "WikiEditor" : {
         "why" : [
            "CodeEditor"
         ],
         "status" : "present"
      },
      "CodeEditor" : {
         "why" : [
            "CodeEditor"
         ],
         "status" : "present"
      },
      "Scribunto" : {
         "status" : "loaded"
      },
      "MassMessage" : {
         "why" : [
            "MassMessage"
         ],
         "message" : "MassMessage is not compatible with the current MediaWiki core (version 1.34.0-alpha), it requires: <= 1.31.0.",
         "status" : "incompatible-core"
      }
   }
}

So the workflow becomes:

  1. Install MediaWiki
  2. Clone $ZUUL_PROJECT
  3. Run php maintenance/checkDependencies.php --extensions=Whatever
  4. If there are any "missing" fields, clone those. Go back to step 3. If there are any "incompatible" fields, give up.
  5. Add wfLoadExtension() to LocalSettings.php
  6. Run whatever tests

Change 503735 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[mediawiki/core@master] [WIP] Add checkDependencies.php

https://gerrit.wikimedia.org/r/503735