Page MenuHomePhabricator

Phaseout CI mediawiki config / extensions_load.txt to load extensions
Closed, ResolvedPublic

Description

integration/jenkins.git has a MediaWiki configuration snippet to detect and load MediaWiki extensions (mediawiki/config.d/50_mw_ext_loader.php) . The same mechanism exists in MediaWiki core itself and CI should rely on it instead.

For the context, we used to have Jenkins build reusing workspaces, hence differents set of extensions were left behind being build. The config snippet is instructed by Zuul for a list of extensions to include via a file extensions_load.txt, the others present on the disk were not loaded. The then detects whether there is an extension.json or an expected entry point then wfLoadExtensions() or require_once.

Nowadays we start with a clean workspace so that system is no more needed.

We would need:

Note: in case an extension lacks extension.json or the expected PHP entry point, LocalSettingsGenerator::generateExtEnableLine forges a require_once with the extension name nonetheless even if the file does not exist. That will later causes an error when failing to require the file. That covers the abort / exit(1); we have in 50_mw_ext_loader.php

Event Timeline

hashar created this task.Mar 13 2018, 9:35 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 13 2018, 9:35 AM
hashar updated the task description. (Show Details)Mar 13 2018, 9:35 AM

Change 419130 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/core@master] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419130

For the context, we used to have Jenkins build reusing workspaces, hence differents set of extensions were left behind being build

Are you planning to get rid of this?

For the context, we used to have Jenkins build reusing workspaces, hence differents set of extensions were left behind being build

Are you planning to get rid of this?

Yes definitely. That is causing too much maintenance troubles and get build to fail randomly because of the dirty state.

Change 419130 merged by jenkins-bot:
[mediawiki/core@master] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419130

hashar updated the task description. (Show Details)Mar 14 2018, 8:54 AM

Change 419384 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/core@REL1_30] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419384

Change 419391 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/core@REL1_29] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419391

Change 419393 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/core@REL1_27] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419393

hashar updated the task description. (Show Details)Mar 14 2018, 11:21 AM

Change 419394 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/core@wmf/1.31.0-wmf.24] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419394

Change 419396 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/core@wmf/1.31.0-wmf.25] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419396

Change 419394 merged by jenkins-bot:
[mediawiki/core@wmf/1.31.0-wmf.24] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419394

Change 419396 merged by jenkins-bot:
[mediawiki/core@wmf/1.31.0-wmf.25] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419396

hashar updated the task description. (Show Details)Mar 14 2018, 8:48 PM

Change 419526 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/core@fundraising/REL1_27] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419526

Change 419605 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/core@master] Let install.php detect and inject extensions [2]

https://gerrit.wikimedia.org/r/419605

Change 419384 merged by jenkins-bot:
[mediawiki/core@REL1_30] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419384

Change 419391 merged by jenkins-bot:
[mediawiki/core@REL1_29] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419391

Change 419393 merged by jenkins-bot:
[mediawiki/core@REL1_27] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419393

Change 419605 merged by jenkins-bot:
[mediawiki/core@master] Let install.php detect and inject extensions [2]

https://gerrit.wikimedia.org/r/419605

Change 424352 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/quibble@master] Make MediaWiki install.php detects extensions

https://gerrit.wikimedia.org/r/424352

Change 424352 merged by jenkins-bot:
[integration/quibble@master] Make MediaWiki install.php detects extensions

https://gerrit.wikimedia.org/r/424352

I havent worked on this for the last two weeks or so. I have been focusing on polishing up Quibble to run against mediawiki/core. Will come back to this next week.

hashar updated the task description. (Show Details)Apr 29 2018, 7:07 PM

mediawiki/core and mediawiki/vendor are using the Quibble based jobs. There are still some jobs using Nodepool though.

I triggered the Quibble job for extensions and skins over the week-end:

SUCCESSFAILURE
Skins3211
Extensions619277

That is encouraging.

Change 419526 abandoned by Reedy:
Let install.php detect and inject extensions

Reason:
https://gerrit.wikimedia.org/r/#/c/mediawiki/core/ /444628/-1..2 brought it in :)

https://gerrit.wikimedia.org/r/419526

hashar removed hashar as the assignee of this task.Aug 1 2018, 3:02 PM

It is still used here and there but I lack time to audit the remaining use cases. So it is back to the backlog.

I think the last two places this is being used is in the legacy Wikibase jobs (T188717), and some fundraising jobs.

I am back on this and the usage has been extended to the phan jobs unfortunately.

dockerfiles/ci-src-setup/setup-mwext.sh:echo -e "${EXT_DEPENDENCIES:-}" >> /tmp/extensions_load.txt

# Use a docker image to clone for a MediaWiki extension or skin
- builder:
    name: docker-ci-src-setup-mwext
        image: 'docker-registry.wikimedia.org/releng/ci-src-setup:0.2.9'
        args: '/srv/setup-mwext.sh'

And that macro docker-ci-src-setup-mwext is used by:

  • mwext-php70-phan-docker
  • mwext-php70-phan-seccheck-docker
  • mwext-php70-phan-seccheck-docker-non-voting
  • mwskin-php70-phan-docker
  • mwskin-php70-phan-seccheck-docker
  • mwskin-php70-phan-seccheck-docker-non-voting

We should be able to use Quibble to clone the repositories / vendor.git and run composer:

quibble --packages-source=vendor --db=sqlite --git-cache=/home/hashar/projects --skip=all -- mediawiki/extensions/3D

But that doesn't run composer install --dev in the extension/skin repository. Then the mediawiki-phan/run.sh entry point tries to look for extensions/XXX/vendor/mediawiki/mediawiki-phan-config/composer.json to get the phan version that should be used.

I guess we can adjust that entry point to install mediawiki/mediawiki-phan-config first.

Taking mediawiki/extensions/3D as an example:

First prepare the repository using Quibble:

ZUUL_PROJECT=mediawiki/extensions/3D \
  quibble --packages-source=vendor --db=sqlite \
  --git-cache=/home/hashar/projects \
  --skip=all
  -c 'cd extensions/3D && composer update'

Then run the magic container:

docker run \
  -e THING_SUBNAME=extensions/3D \
  --volume=/home/hashar/src:/mediawiki \
  --rm -it \
  docker-registry.wikimedia.org/releng/mediawiki-phan:0.1.11 \
  /mediawiki/extensions/3D -m checkstyle

Result:

+ umask 002
+ cd /mediawiki/extensions/3D
+ '[' '!' -f /mediawiki/extensions/3D/.phan/config.php ']'
+ jq -e '.extra."phan"' composer.json
null
+ CFG_COMPOSER=vendor/mediawiki/mediawiki-phan-config/composer.json
+ jq -e '.extra."phan"' vendor/mediawiki/mediawiki-phan-config/composer.json
"1.2.6"
++ jq -r '.extra."phan"' vendor/mediawiki/mediawiki-phan-config/composer.json
+ PHAN_VERSION=1.2.6
+ '[' -f .phan/config.php ']'
+ export PHP_ARGS=-dextension=ast_101.so
+ PHP_ARGS=-dextension=ast_101.so
+ install_phan
+ cd /srv/phan
+ composer require phan/phan:1.2.6
./composer.json has been created
...
  - Installing phan/phan (1.2.6): Downloading (100%)         
...
+ exec /srv/phan/vendor/bin/phan -d . -p
RecursiveDirectoryIterator::__construct(includes/): failed to open dir: No such file or directory
RecursiveDirectoryIterator::__construct(maintenance/): failed to open dir: No such file or directory
RecursiveDirectoryIterator::__construct(tests/phan/stubs/): failed to open dir: No such file or directory
RecursiveDirectoryIterator::__construct(../../tests/phan/stubs): failed to open dir: No such file or directory
<< progress bar showing up >>
$ echo $?
0
$

Seems easier than expected :]

Note that Quibble still runs MediaWiki install.php as well as npm install for mediawiki/core. That is because --command is intended to have a fully working install.

Change 503362 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] phan: use Quibble to prepare MediaWiki sources

https://gerrit.wikimedia.org/r/503362

Change 503364 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: remove setup-mwext.sh entry point

https://gerrit.wikimedia.org/r/503364

hashar claimed this task.May 3 2019, 12:29 PM

Change 503362 merged by jenkins-bot:
[integration/config@master] phan: use Quibble to prepare MediaWiki sources

https://gerrit.wikimedia.org/r/503362

Change 503364 merged by jenkins-bot:
[integration/config@master] docker: remove setup-mwext.sh entry point

https://gerrit.wikimedia.org/r/503364

hashar closed this task as Resolved.May 6 2019, 8:40 AM

The last use case of EXT_DEPENDENCIES was for Phan which I have removed.

For Quibble, that will be done as part of T220199

Change 508320 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] phan: use composer and mysql

https://gerrit.wikimedia.org/r/508320

Change 508320 merged by jenkins-bot:
[integration/config@master] phan: use composer and mysql

https://gerrit.wikimedia.org/r/508320