We should just get rid of extension-list...

@Legoktm: I agree, it seems unnecessary? But update-l10n breaks without it, currently... I'll dig deeper into it.

It looks like mergeMessageFileList.php can scan the extensions directory, as an alternative to passing an extension-list file, however, we need to scan extensions/* and skins/*

Should I make a way that we can pass multiple directories to the script or perhaps an option to recursively search for everything starting from the root of the mediawiki/core branch checkout?

Anyone have an opinion?

Add support for passing multiple paths to extension-dir arg in mergeMessageFileList.php

https://gerrit.wikimedia.org/r/#/c/268337/

Change 268337 had a related patch set uploaded (by Legoktm):
Support multiple extension-dir paths to be passed to mergeMessageFileList

https://gerrit.wikimedia.org/r/268337

Final step is generating the extension list when the branch is cut.

https://gerrit.wikimedia.org/r/#/c/287565/

From code review of https://gerrit.wikimedia.org/r/#/c/287565/ @hashar wrote:

Do we still need the extension-list anyway? IIRC it is only used for merging l10n messages and the logic could be added to that maintenance script instead.

For CI purposes, we have moved most if not all extensions to use ExtName/ExtName.php as the entry point, and some are migrated to extension.json / skin.json. So the lookup should be quite trivial and we could get rid of extension-list entirely.

@mmodell replied:
It would be great if we could remove the need for this, but somewhere it has to be either generated or computed on demand. My main goal was to remove the chicken-or-egg problem when adding or removing an extension during branch cutting.

I am pretty sure the only use case is maintenance/mergeMessageFileList.php which apparently already has some kind of detection mechanism built-in:
...

$possibilities = [
    "$extdir/$extname/extension.json",
    "$extdir/$extname/skin.json",
    "$extdir/$extname/$extname.php"
];
$found = false;
foreach ( $possibilities as $extfile ) {
    if ( file_exists( $extfile ) ) {

...
That is enabled by passing it --extensions-dir and it "nicely" errors out whenever an extension lack one. The scap task still rely on the extension-list file ... Worth giving that a try?

D288 would move scap to use --extensions-dir and we would no more need the extension-list file.

@hashar is right, this isn't necessary if mergeMessageFileList.php can handle things without the extension list.

Below is a draft comment I wrote at some point. committing it for the record

@mmodell replied:
It would be great if we could remove the need for this, but somewhere it has to be either generated or computed on demand. My main goal was to remove the chicken-or-egg problem when adding or removing an extension during branch cutting.

I am pretty sure the only use case is maintenance/mergeMessageFileList.php which apparently already has some kind of detection mechanism built-in:
...

$possibilities = [
    "$extdir/$extname/extension.json",
    "$extdir/$extname/skin.json",
    "$extdir/$extname/$extname.php"
];
$found = false;
foreach ( $possibilities as $extfile ) {
    if ( file_exists( $extfile ) ) {

...
That is enabled by passing it --extensions-dir and it "nicely" errors out whenever an extension lack one. The scap task still rely on the extension-list file ... Worth giving that a try?

D288 would move scap to use --extensions-dir and we would no more need the extension-list file.

The TLDR is that I wrote the --extensions-dir feature years ago but it never got used afaik.

I'm not afraid of using it, I'm mostly concerned of how we'll handle beta. We currently have the extension meta repo symlink'd for easy installation. Sadly, this would make l10n regeneration impossibly slow if we only used --extensions-dir

beta uses a clone of mediawiki/extensions but surely we could instead use the list of repos for which we cut a branch?

Possibly. But we also deploy some extensions to beta that we don't branch for prod yet. I'd rather just stop using that meta repo entirely--can we think up another solution?

Could we add a "beta-only" section in make-wmf-branch's config.json?

In T125678#3343022, @Legoktm wrote:

Could we add a "beta-only" section in make-wmf-branch's config.json?

Maybeeeee..... but beta doesn't rely on make-wmf-branch so it'd be a weird dependency. Really all this crud should move to wmf-config as scap plugins.

Some parts of the problem: T125678

The subtask is now closed and all of extension-list is now on extension.json but I want to point out that Wikibase doesn't have extension.json, it has extension-repo.json and extension-client.json

@hashar Is this ticket resolved?

@hashar: Could you please answer the last comment? Thanks in advance!

We still rely on operations/mediawiki-config having the list of all extension.json / skin.json in wmf-config/extension-list. The related code is:

# attempt to read extension-list from the branch instead of wmf-config
ext_list = os.path.join(cfg["stage_dir"], "php-%s" % version, "extension-list")

if not os.path.isfile(ext_list):
    # fall back to the old location in wmf-config
    ext_list = "%s/wmf-config/extension-list" % cfg["stage_dir"]

utils.sudo_check_call(
    "www-data",
    "/usr/local/bin/mwscript mergeMessageFileList.php "
    '--wiki="%s" --list-file="%s" '
    '--output="%s"' % (wikidb, ext_list, new_extension_messages),
)

A patch I wrote ten years ago (ad1609059f1) made mergeMessageFileList.php to auto discovers extensions and skins by passing --extension-dir. The idea was to save us the manual maintenance of wmf-config/extension-list and instead rely on crawling the extensions and skins registered as submodule of the mediawiki/core wmf branches.

Would it work? Well maybe. I am not sure what will happens to Beta-Cluster-Infrastructure which has all the extensions and skins hosted on Gerrit registered as submodules which would potentially cause very large l10n files and cache to be generated.