Quibble support for repositories cloned with canonical name (no clonemapper)
Closed, DeclinedPublic
Actions

Assigned To

None

Authored By

	hashar
	Oct 25 2023, 5:40 PM

Description

Quibble remap the mediawiki/core mediawiki/vendor repo into a traditional layout by using a mapping as to where they should be cloned in the workspace. Eg:

/workspace  > mediawiki/core clone
/workspace/vendor > mediawiki/vendor

The map definition:

quibble/zuul.py

CLONE_MAP = [
    {'name': 'mediawiki/core', 'dest': '.'},
    {'name': 'mediawiki/vendor', 'dest': './vendor'},
    {'name': 'mediawiki/extensions/(.*)', 'dest': './extensions/\\1'},
    {'name': 'mediawiki/skins/(.*)', 'dest': './skins/\\1'},
    {'name': 'mediawiki/services/(.*)', 'dest': './services/\\1'},
]

MediaWiki has support for indicating where extensions and skins are located so we can use:

$wgExtensionDirectory = '/workspace/mediawiki/extensions';
$wgStyleDirectory = '/workspace/mediawiki/skins';

We should probably always set them in the LocalSetting template. Then a feature flag could be passed to disable the clone mapping.

Details

	Subject	Repo	Branch	Lines +/-
	Support clone with project canonical name	integration/quibble	master	+30 -9

Customize query in gerrit

Event Timeline

hashar created this task.Oct 25 2023, 5:40 PM

hashar updated the task description. (Show Details)Oct 25 2023, 5:45 PM

hashar updated the task description. (Show Details)Oct 30 2023, 9:55 AM

Change 971227 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/quibble@master] Support clone with project canonical name

https://gerrit.wikimedia.org/r/971227

gerritbot added a project: Patch-For-Review.Nov 2 2023, 4:18 PM

It is a bit more complicated than my first analysis last week, I was too ambitious.

Quibble clone the repositories then invokes the MediaWiki installer with maintenance/install.php --with-extensions which detects extensions and skins nested in MediaWiki core installation path ($IP) in $IP/extensions and $IP/skins directories:

includes/installer/Installer.php

    public static function getExistingLocalSettings() {
        $IP = wfDetectInstallPath();
...
        global $wgExtensionDirectory, $wgStyleDirectory;
...
        $wgExtensionDirectory = "$IP/extensions";
        $wgStyleDirectory = "$IP/skins";
        require $lsFile;

And if there is already a LocalSettings.php the script bails out asking to run update.php.... Maybe we can symlink them for the installation, set $wgExtensionDirectory and $wgStyleDirectory then remove the symlinks once the installation has completed.

Additional thought: gotta handle vendor or parsoid not being at the root of the MediaWiki core installation, but that can probably be handled via symbolic links.

Change 971227 abandoned by Hashar:

[integration/quibble@master] Support clone with project canonical name

Reason:

The MediaWiki CLI installer uses --with-extensions to discover extensions and skins and it is hardcoded to use MW_INSTALL_PATH, eg it requires repositories to be cloned nested inside MediaWiki :-(

https://gerrit.wikimedia.org/r/971227

Maintenance_bot removed a project: Patch-For-Review.Nov 3 2023, 12:30 PM

I have abandoned the idea to add the feature in Quibble, that would requires patching the MediaWiki installer to support environment variable to determine where extensions and skins are installed (+ backports to all supported branches).

So I guess instead, get Zuul to clone with the Gerrit project names as canonical paths:

/checkouts/mediawiki/core
/checkouts/mediawiki/vendor
/checkouts/mediawiki/extensions/Foobar
/checkouts/mediawiki/skins/Vector
/checkouts/mediawiki/service/parsoid

Then add a build step that would creates symbolic links under /checkouts/mediawiki/core (the installation path or IP in MediaWiki jargon). Example:

#!/bin/bash

set -eu -o pipefail

echo 'Erasing "checkouts"'
rm -fR checkouts

echo 'Creating dummy git checkouts"'
mkdir -p checkouts/mediawiki/core/{extensions,services,skins,vendor}
mkdir -p checkouts/mediawiki/vendor
mkdir -p checkouts/mediawiki/extensions/Foobar
mkdir -p checkouts/mediawiki/extensions/Hello
mkdir -p checkouts/mediawiki/skins/Vector
mkdir -p checkouts/mediawiki/services/parsoid
find checkouts -type d -empty | sort

echo 'Symlinking vendor'
IP=checkouts/mediawiki/core
# Force erase mediawiki/core vendor directory
# Only needed when the build has mediawiki/vendor.git as a dependency, 
# else it is populated by `composer install`
ln -s --relative --force checkouts/mediawiki/vendor "$IP/vendor"

ln -s --relative -t "$IP/extensions/" checkouts/mediawiki/extensions/*
ln -s --relative -t "$IP/skins/" checkouts/mediawiki/skins/*

# Handle parsoid edge case: mediawiki/core has no `services` directory
mkdir -p "$IP/services"
ln -s --relative -t "$IP/services/" checkouts/mediawiki/services/*

echo "Result in installation path $IP"
tree -C "$IP"

output

Erasing "checkouts"
Creating dummy git checkouts"
checkouts/mediawiki/core/extensions
checkouts/mediawiki/core/services
checkouts/mediawiki/core/skins
checkouts/mediawiki/core/vendor
checkouts/mediawiki/extensions/Foobar
checkouts/mediawiki/extensions/Hello
checkouts/mediawiki/services/parsoid
checkouts/mediawiki/skins/Vector
checkouts/mediawiki/vendor
Symlinking vendor
Result in installation path checkouts/mediawiki/core
checkouts/mediawiki/core
├── extensions
│   ├── Foobar -> ../../extensions/Foobar
│   └── Hello -> ../../extensions/Hello
├── services
│   └── parsoid -> ../../services/parsoid
├── skins
│   └── Vector -> ../../skins/Vector
└── vendor
    └── vendor -> ../../vendor

10 directories, 0 files

Quibble runs out of the current working directory (or /workspace when run under Docker` and then expects MediaWiki installation to be in src directory. So under Docker there would need a /workspace/src → /checkouts/mediawiki/core symlink and run Quibble from that /workspace.

We've worked around the immediate need for this. In case this is needed in future we can reopen this task.

Quibble support for repositories cloned with canonical name (no clonemapper)Closed, DeclinedPublicActions

Description

Details

Event Timeline

Quibble support for repositories cloned with canonical name (no clonemapper)
Closed, DeclinedPublic
Actions