Page MenuHomePhabricator

In CI BlueSpice repositories should always have BlueSpiceFoundation injected
Open, NormalPublic

Description

To add a BlueSpice extension, one has to add to Zuul the template extension-quibble-composer but we often forget to also have BlueSpiceFoundation injected via zuul/parameter_functions.py.

We should be able to just automatically inject BlueSpiceFoundation whenever an extension has the BlueSpice prefix.

Event Timeline

hashar created this task.Jun 25 2019, 7:31 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 25 2019, 7:31 PM
hashar triaged this task as Normal priority.Jul 1 2019, 4:47 PM
Osnard awarded a token.Jul 3 2019, 5:50 AM

Is there anything I can help with?

hashar added a comment.Jul 3 2019, 7:04 AM

For sure! If you feel adventurous you can dig into zuul/parameter_function.py and attempt to add the dependency. I haven't even looked at the problem, we would probably need to refactor/enhance the code slightly before though :-]

hashar added a comment.Jul 3 2019, 7:07 AM

Or better, I have added a feature to Quibble which makes it process the dependencies as they are defined in extension.json. So that any BlueSpice extension could just use something like:

extension.json
{
  requires: {
    "extensions": {
      "BlueSpiceFoundation": "*"
    }
  }
}

Which in the end would let us drop the crazy dependency tree from CI and delegate it to the extensions/skins. So I guess I am just going to create a Jenkins job that has that feature enabled and try it out, then we can probably just migrate all of BlueSpice extensions to it.

Osnard added a comment.Jul 3 2019, 7:16 AM

This would indeed be awesome! Actually most (all?) of our extensions are already making use of the requires.extensions field in extension.json. Does this new feature also do a version check of the dependency? So, e.g. if BlueSpiceFoundation requires ExtJSBase@1.31, but the master branch of ExtJSBase has set version 1.34 in extension.json, will the "dependency resolver" check other branches and download the one that has a proper version set in extension.json (in this case REL1_31 of ExtJSBase)?

Can you point me to the code of that new feature?

hashar added a comment.Jul 3 2019, 9:33 AM

The feature merely inspect extension.json and then recursively clone the required repositories, fetch the proper patch/branch. Once all the repositories have been cloned, the MediaWiki installer is run which should catch any dependency issues and bails out.

The code in quibble is https://gerrit.wikimedia.org/r/#/c/integration/quibble/+/502286/ which introduces the command line option --resolve-requires which turns on the feature above. It is very rough and just clone repositories.

I have to write doc, write a script to check requirements from zuul/zuul_parameters.py and extension.json are in sync. But in short yeah there is code around :-]

Osnard added a comment.Jul 4 2019, 7:54 AM

As far as I understand the new feature will fetch all the extension repos listed in extension.json/requires.extensions (even recursively) and then checkout on the same branch as the extension-under-test, right? If that branch is not available on a dependency-repo it will fall back to master on that. If the concrete version number of that dependency-extension then does not match the one required by the extension-under-test the CLI installer will skip the wfLoadExtension call in LocalSettings.php and the tests will fail.

Would it make sense to have a map of version number <=> branch, which the new quibble feature could use to find the appropriate branch? Doing this on the fly might be very expensive, as it would require to cehckout branches and look into extension.json until a proper version number was found.

Such a map could look like

{
    ...,
    "ExtJSBase": {
        "1.31" => "REL1_31"
        "1.34" => "master"
    },
    ...
}

This mapping could be created by some script on a daily basis. I think it could be quite simple to implement (actually it won't have to search _all_ branches. Just the latest ones and the LTS).

Is this too much of an edge case to be useful for WMF CI?