I took a look at possible approaches here. Since T189567: Phaseout CI mediawiki config / extensions_load.txt to load extensions, our CI runs the CLI installer with the with-extensions option. This means the installer scans the extensions (and skins) directories for extension.json files. Whenever it finds one, it adds it to the installer-generated LocalSettings.php. In other words: the set of extensions installed in CI is configured purely by making CI clone additional directories into an appropriate subdirectory. From there, the installer takes care of the rest. Internally, this happens in Quibble's code for installing MediaWiki on the CI side, and in CommandLineInstaller and friends on the MW side.

It is worth noting the installer recognises the legacy PHP entrypoints, which were used before the introduction of extension registration. If the manifest file doesn't have the expected name (for extensions, extension.json), it attempts to load a PHP file with the same name, as the extension. This is a mechanism employed by the Wikibase repository, which includes two separate extensions (the client and the server), see source code for details.

In our case, the example extension is an integral part of the CommunityConfiguration directory. Relatively to the IP, it is located at extensions/CommunityConfiguration/example. In addition to that, its manifest has a non-standard name – it is called example-extension.json. Not sure if that naming has a reason (it shouldn't conflict with the main one, as it is in a subdirectory), but that shouldn't matter for the rest of my comment. Unfortunately, the extension's locations renders the standard way of enabling extensions within CI (=via the scanner in installer) unusable.

Quibble makes use of the installer-generated config, but it also appends its own configuration to it. LocalSettings.php for CI can be found in Quibble's source code (it is a template; full version is visible in CI artefacts in Jenkins if one is interested).

With the above in mind, there are a couple of phases during which we can (at least theoretically) enable the example extension:

1. MW install: We can (somehow) make the example extension visible to the installer, leaving it up to the installer to generate an appropriate wfLoadExtension call.
2. LocalSettings overrides: We can add the wfLoadExtension call to the Quibble overrides. We can do this by actually adding the wfLoadExtension call to the template, or we can introduce a mechanism which would allow CI config to load "subextensions" like this one.
3. Extension registration: We can hook into the extension registration system itself and let it load another extension at the very last minute.

Ad option (1): There are several possible ways how this can be accomplished:

(a) Promote the example extension into a "full" extension: We can promote the example extension into a full extension, letting it have its own repository, CI setup etc. It would presumably need to be gated as well, so that its tests are executed when merging in CommunityConfiguration as well.
(b) Dynamically move the example extension code up: We can move the extension's code one level up when preparing the CI workspace. Even though the code would be a part of the main extension, it wouldn't look like that in CI – and thus the installer would process it just as if it was a full extension.
(c) Add support for non-standard extension manifests to the installer: We can make the installer accept a list of non-standard extension manifests it should scan (in addition to the automatically discovered one). This list can be fed to the installer through CI configuration. Even though the manifest for the example extension wouldn't be automatically detected, it would still be considered by the installer and enabled in the installer-generated part of local settings.

Ad option (2): The easiest way to do this would be directly adding the line quoted in the description to Quibble's LocalSettings (linked above), possibly conditioned by "does this file exist". Alternatively, we can build something more generic – it would be effectively the same approach as passing non-standard extension manifests to the installer, just happening at a different level.

Ad option (3): Extension registration exposes an onRegistration callback (docs). Unfortunately, this is not suitable for registering new extensions – when the callback is executed, the extension registration queue is already being processed (and immutable). Adding a system of chains to extension registration seems not worth the effort, especially when we'd want it to be only used in tests.

Taking the above into account, the most promising approaches seems to be adding support for non-standard manifests to the CLI installer or alternatively, doing the same within the Quibble-generated part of LocalSettings. Considering all other extensions are getting registered by the installer, the first approach might be a better idea. On the other hand, Parsoid's hacks are located within Quibble's local settings, so it might make sense keeping similar code at the same place (that being said, Parsoid's hacks are more extreme than what we're likely to need in our case).

Hope this helps!

Thank you for your detailed thoughts! I somehow had hopes that a mechanism for that already would exist, but since it doesn't, I agree that (1c) sounds best from the options you listed.

Another thought I had would be to basically move away from having an example-extension.json to having a set of php-settings in a file to do the same thing. Still there would be the question for how to load that in CI, but at least that is something that could, in theory, be loaded later.

Either way, I very much appreciate what you wrote here 🙏. Now we probably need RelEng's perspective on all this.

In T373114#10088735, @Michael wrote:

Thank you for your detailed thoughts! I somehow had hopes that a mechanism for that already would exist, but since it doesn't, I agree that (1c) sounds best from the options you listed.

Happy to help! FWIW,The mechanism sort of exists, although I admit it might not be very clear from what I wrote, as I did not explicitly list it as an option. In theory, we could rename extension.json to something else and then introduce the legacy PHP entrypoint (CommunityConfiguration.php), which would then be loaded by CI automatically. This is how Wikibase is loading its two subextensions in CI as it needs.

This is the simplest thing we could do, but it would make installing the main extension more difficult, as you'd have to explicitly specify the non-standard name (similar to the wfLoadExtension call from the extension). It's also a form of technical debt, as I imagine we eventually want to get rid of legacy PHP endpoints fully in favour of extension registration. So, I'd prefer us to have a more long-term solution here.

Another thought I had would be to basically move away from having an example-extension.json to having a set of php-settings in a file to do the same thing. Still there would be the question for how to load that in CI, but at least that is something that could, in theory, be loaded later.

That'd be basically switching to the pre-extension registration system for that example extension. It might work, but I'm not convinced switching back is a particularly good idea in the long term, especially if we decide to load the example extension later than all other extensions. Even if loading later approach works today, it might break with future changes in either CC or Core. One of the benefits of extension registration is that we have all information about all extensions readily available at a certain point, and I think that's a good benefit to keep.

If possible, I'd rather avoid going the Wikibase route and changing how our extension is being loaded in production. Though it is an option on the table, too.

My idea was less intended as "let's do that", and more as "yet another approach that might work with its own set of trade-offs".

But from this conversation, I'm also adding the Continuous-Integration-Infrastructure tag, because I'm not actually sure if this is something to be fixed in CI config or infrastructure.

(My 2c: having multiple extensions in the same repository causes headaches for other tooling's assumptions, and is nice to avoid if at all possible.)

In T373114#10101990, @kostajh wrote:

(My 2c: having multiple extensions in the same repository causes headaches for other tooling's assumptions, and is nice to avoid if at all possible.)

Thanks for the comment. Just for curiosity, would you mind clarifying which kind of (other) headaches you mean here?

In T373114#10102044, @Urbanecm_WMF wrote:

In T373114#10101990, @kostajh wrote:

(My 2c: having multiple extensions in the same repository causes headaches for other tooling's assumptions, and is nice to avoid if at all possible.)

Thanks for the comment. Just for curiosity, would you mind clarifying which kind of (other) headaches you mean here?

e.g., the code quality bot assumes that each repo matches one to one with an extension (for mapping back to projects on sonarcloud.io) and has exceptions handwritten for Wikibase; if you want to experiment with things like parallelizing PHPUnit or Selenium tests, you generally assume that there's only one tests/phpunit or tests/selenium directory, and repos with multiple extensions will break that assumption. I am trying to remember some other pain points I encountered from a few years ago, but generally it's just that most of CI and its tooling assumes 1:1 mapping of repository and extension, and then Wikibase and a handful of others are carved out as exceptions to the rule.

In T373114#10102048, @kostajh wrote:

if you want to experiment with things like parallelizing PHPUnit or Selenium tests, you generally assume that there's only one tests/phpunit or tests/selenium directory, and repos with multiple extensions will break that assumption.

Mh, depending on the specifics of how we load this example extension, this assumption might still hold, just there would be one extension that does not live in a direct subdirectory of mediawiki/extensions/. Though I see the challenges in general.
The global alternative would be to move this example to its own git-repository (probably repurpose the existing one), and to load that in CI and beta. That comes with its own tradeoffs of having to juggle changes to multiple repositories for tests, related code not living in the same place, new dev onboarding being more complex, and so on. But it would not be impossible either.

Triage things:

• Link to sub extension to save folks on this task some searching :)
• Tagging in @hashar as resident quibble expert to weigh in/offer feedback on the ideas.

Question: Why make a subextension?

Is the primary reason to keep them together the cognitive overhead low for new developers? To speed up clone time in CI? Or something else?

Unsolicited thought: maybe there's some way to handle this with a git subtree/submodule + gerrit magic.

+1 to @kostajh I'm guessing no one considered subextensions when building out maintenance scripts/CI stuff—we'll likely discover lots of small tweaks we need. Plus, after those tweaks, everyone designing new processes/maintenance scripts would need to think about sub-extensions forever after. Given that, I'd like to learn more about why it's needed.

Copying my comment from the Slack thread:

[My advice would be] "Don't". Multiple-extensions in a single repo are a complete disaster for CI (see Wikibase). RelEng should decide, of course, but I'd advise just putting it in its own repo which you depend on (so your CI will always run it).

Thank you for the feedback, that is something we can work with!

The main motivation was to have the code together and have it change in a single commit, so that we don't always have to do those little dance of a bunch of Gerrit changes depending on each other. (for example: 1st change adds the new feature, 2nd change uses it in the example and depends on the first, 3rd change adds a test in the core extension making use of the example and depending on the 2nd change)

But maybe that's a tradeoff worth making for not introducing a permanently increased complexity. I kind of had hoped that there would already be a simple way to do it and we just would have to "turn it on". But if the Wikibase-way is the only way that there is and we would need to build something new from scratch, then that's probably not worth it.

I'll look into what it would mean to create it as a separate extension.

Ok, moving forward with the "fully separate extension"-approach for now, I've created a request for creating one in Gerrit: https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests#mediawiki/extensions/CommunityConfigurationExample

In T373114#10133105, @Michael wrote:

Ok, moving forward with the "fully separate extension"-approach for now, I've created a request for creating one in Gerrit: https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests#mediawiki/extensions/CommunityConfigurationExample

One more option: merge the CommunityConfiguration/example code into https://www.mediawiki.org/wiki/Extension:BoilerPlate.

In T373114#10133349, @kostajh wrote:

In T373114#10133105, @Michael wrote:

Ok, moving forward with the "fully separate extension"-approach for now, I've created a request for creating one in Gerrit: https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests#mediawiki/extensions/CommunityConfigurationExample

One more option: merge the CommunityConfiguration/example code into https://www.mediawiki.org/wiki/Extension:BoilerPlate.

Interesting suggestion, but after talking it through with @Urbanecm_WMF, we think it is probably not the right approach for this particular use-case. Here we want complex schemas that cover all the functionality rather exhaustively, so that 1) we can test it in CI and QA, 2) we can show-case various ways how to use it for developers, and 3) it is a source of inspiration for UX designers.

On the other hand, with Extension:Boilerplate, we would want to only add a rather simple schema to make it easy to customize and get started. A sensible thing to have, and I should create a task for it, but it is not this task. :)
Done: T374460: Add simple CommunityConfiguration to Extension:Boilerplate

Change #1072228 had a related patch set uploaded (by Michael Große; author: Michael Große):

[integration/config@master] Enable CommunityConfigurationExample extension

https://gerrit.wikimedia.org/r/1072228

Following the advice here, we now moved the code to a separate extension that now hopefully is straight forward to enable in CI. I've now created a change for that: Enable CommunityConfigurationExample extension. Feedback welcome :)

Change #1072228 merged by jenkins-bot:

[integration/config@master] Zuul: [mediawiki/extensions/CommunityConfigurationExample] Add CI

https://gerrit.wikimedia.org/r/1072228

Mentioned in SAL (#wikimedia-releng) [2024-09-11T20:19:22Z] <James_F> Zuul: [mediawiki/extensions/CommunityConfigurationExample] Add CI, for T373114

Change #1075575 had a related patch set uploaded (by Michael Große; author: Michael Große):

[integration/config@master] gatedextensions: Add dependency extension for CommunityConfiguration

https://gerrit.wikimedia.org/r/1075575

Change #1075575 merged by jenkins-bot:

[integration/config@master] gatedextensions: Add dependency extension for CommunityConfiguration

https://gerrit.wikimedia.org/r/1075575