Page MenuHomePhabricator

In many extensions, composer.json doesn't validate to spec
Open, LowestPublicBUG REPORT

Description

name and description are required properties for a strictly valid composer.json file. However, many extensions do not include these properties. This makes them unsuitable for managing with Composer when otherwise they would be. (You don't get dependency resolution, version constraint matching, etc. and are forced to declare a 'package' repository for each extension that you want to manage with Composer. ref

List of steps to reproduce

  • Go to any extension directory using your terminal.
  • type composer validate

What happens?:
You get publish errors that describe how "name" and "description" are required properties for strict validation.

/var/www/html/extensions/CodeMirror# composer validate
./composer.json is valid for simple usage with Composer but has
strict errors that make it unable to be published as a package
See https://getcomposer.org/doc/04-schema.md for details on the schema
# Publish errors
- name : The property name is required
- description : The property description is required
# General warnings
- No license specified, it is recommended to do so. For closed-source software you may use "proprietary" as license.

What should have happened instead?:
composer.json is valid message

/var/www/html/extensions/AdminLinks# composer validate
./composer.json is valid, but with a few warnings
See https://getcomposer.org/doc/04-schema.md for details on the schema
# General warnings
- require.composer/installers : unbound version constraints (>=1.0.1) should be avoided

Positive example:
https://github.com/ProfessionalWiki/bootstrap/blob/master/composer.json

{
	"name": "mediawiki/bootstrap",
	"type": "mediawiki-extension",
	"description": "Provides the Bootstrap 4 web front-end framework to MediaWiki skins and extensions",
	"keywords": [
		"wiki",
		"MediaWiki",
		"extension",
		"Twitter",
		"Bootstrap"
	],
	"homepage": "https://www.mediawiki.org/wiki/Extension:Bootstrap",
	"readme": "README.md",
	"license": "GPL-3.0-or-later",
	"authors": [
		{
			"name": "Stephan Gambke",
			"email": "s7eph4n@protonmail.com",
			"role": "Developer"
		},
		{
			"name": "Professional.Wiki",
			"email": "info@professional.wiki",
			"homepage": "https://professional.wiki",
			"role": "Maintainer"
		}
	],
	"support": {
		"issues": "https://github.com/cmln/mw-bootstrap/issues",
		"forum": "https://www.mediawiki.org/wiki/Extension_talk:Bootstrap",
		"wiki": "https://www.mediawiki.org/wiki/Extension:Bootstrap",
		"irc": "irc://libera.chat:6667/mediawiki",
		"source": "https://github.com/cmln/mw-bootstrap",
		"docs": "https://github.com/cmln/mw-bootstrap/tree/latest/docs",
		"rss": "https://github.com/cmln/mw-bootstrap/releases.atom"
	},
	"require": {
		"php": ">=5.6",
		"composer/installers": "^2|^1.0.1",
		"mediawiki/scss": "~2.0"
	},
	"require-dev": {
		"mediawiki/mediawiki-codesniffer": "39.0.0",
		"mediawiki/mediawiki-phan-config": "0.11.1",
		"php": ">=7.2"
	},
	"autoload": {
		"psr-4": {
			"Bootstrap\\": "src/",
			"Bootstrap\\Tests\\" : "tests/phpunit/"
		}
	},
	"scripts": {
		"test": [
			"phpcs -p -s"
		],
		"fix": "phpcbf"
	},
	"extra": {
		"branch-alias": {
			"dev-master": "4.x-dev"
		}
	}
}

other information:
I can find extensions that are valid in MediaWiki code search but I didn't work out the regex to be able to do a negative lookahead (perl-compatible regex is not supported) to generate a list of all the affected extensions.

Additional projects without a 'tag' (there are many more affected extensions, this is just a partial list)

  1. https://github.com/wikimedia/mediawiki-extensions-NoTitle/blob/master/composer.json
  2. https://github.com/wikimedia/mediawiki-extensions-NumberFormat/blob/master/composer.json
  3. https://github.com/wikimedia/mediawiki-extensions-RegexFun/blob/master/composer.json
  4. https://github.com/wikimedia/mediawiki-extensions-UrlGetParameters/blob/master/composer.json

Recommended Fix:
Simply adding "name": and "description": properties to invalid composer.json files will fix the issue. Important: Extension names in MediaWiki are (almost?) always CamelCase, while the name property for composer.json MUST be lowercase. Simply insert a 'hyphen' (-) character before any capitalized letters in your extension name. If your extension name is PdfHandler, the composer name would be`mediawiki/pdf-handler`

Example: copy/paste to add this to the top of your composer.json file, and revise as needed.

	"name": "mediawiki/my-extension",
	"type": "mediawiki-extension",
	"description": "Does something great",
	"keywords": [
		"wiki",
		"MediaWiki",
		"extension"
	],
	"homepage": "https://www.mediawiki.org/wiki/Extension:MyExtension",
	"license": "GPL-3.0-or-later",

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
freephile updated the task description. (Show Details)
MusikAnimal subscribed.

It's worth mentioning that installing MediaWiki extensions via composer isn't well-supported (T250406). This is probably why so many of them aren't published and lack the necessary properties in composer.json.

Publishing is not required. In fact, you can't publish a package with an invalid composer.json file.
There is no reason why MediaWiki extensions should have invalid composer.json files.

"isn't well-supported" is exactly due to this bug.

There are no other requirements. You don't have to publish (even if you could). You don't have to tag your software with version numbers (even if you should). You don't have to adopt SemVer (even if you should).

...

I don't think MA was saying this shouldn't be done, only giving reasons why it hasn't been. There are other issues with Composer that have made it difficult to use, and as a result issues like this have not shown up. Thank you for letting the community know the files are insufficient.

composer.json files are mainly used for linter and build tools on the CI or to install dependency for all installs, but not to support install of the extension itself, thats why the files are not needed to be valid.
All the information are just duplicated with extension.json and would introduce maintenance burden or going out of sync. No need to have strictly valid files, along there are usable for CI purpose.

Maybe overlapping with T284653: Missing composer.json details
See also T249573: Remove support for extensions requiring a MediaWiki version via Composer

Reedy renamed this task from Invalid composer.json in many extensions to In many extensions, composer.json doesn't validate to spec.Jun 25 2022, 10:29 PM
Reedy triaged this task as Lowest priority.
Reedy updated the task description. (Show Details)
Legoktm subscribed.

Please use composer validate --no-check-publish.

I'm a 3rd-party (corporate) user of MediaWiki and we use Docker for local sandbox development while hosting official environments (DEV, QA, PROD) in AWS using their analog called ECS (Elastic Container Service).

But we face a problem when it comes to building and deploying MediaWiki using Docker images in combination with Composer. For package discovery, all you need to do normally (without Packagist nor any official package registry) is add a two line entry in the repositories section of composer.local.json - providing the package name and GitHub url for example.

Except that fails when the composer.json file of any extension is invalid. Many MediaWiki extensions do not have valid composer files (missing only name and description). So we have been resorting to duplicating all the information about each extension as package repositories in the 'repositories' section of our composer.local.json. This is cumbersome, manual, and brittle - it is not a recommended practice. It would be much better for us to create and manage a registry like HalloWelt does or this fuller registry by Pierre Rudloff. But this ticket is not about that. That's the goal of https://phabricator.wikimedia.org/T250406 The establishment of any reliable registry might obviate the need for this ticket. Furthermore, WMF might create an official registry after the move to GitLab since GitLab is building support for composer package registries

Meanwhile, we can not easily change our whole software development process to copy WMF. And actually, most MediaWiki consultants and corporate users use the same approach we do. Our system was built by Wikibase Solutions (Netherlands). In a nutshell, we build a MediaWiki container, and during the build process we run composer update to download all the extensions specified in composer.local.json (thanks to the Composer Merge plugin) while we also load our LocalSettings.php into the finished container with appropriate wfLoadExtension() and configuration settings. Lastly, we we run maintenance/update.php and similar scripts for installation of those extensions into a finished container image.

Of course we also use Composer to run scripts for CI similar to WMF (aka composer test or MediaWiki-Docker Selenium tests).

If composer.json is useful to WMF, I can not see how adding a name and description to the Boilerplate extension would be undesirable or have any negative consequences. Plenty of extensions, such as CLDR do have valid composer.json files, and I assume they work fine with WMF CI. You'll notice that all the BlueSpice extensions, SemanticMediaWiki extensions, and other 3rd-party authors follow this convention.

Can we please update the Boilerplate extension, and ask that other extension authors do the same? Or can someone please help me understand how/why adding a name and description would break any usage at WMF? Making the composer.json files valid seems like an important step in the right direction.

I'm happy to supply patches.

But this ticket is not about that. That's the goal of https://phabricator.wikimedia.org/T250406

I understand that you're trying to approach this from a different perspective to distinguish the two feature requests but really it is all the same. We used to add name/description/license/etc. to composer.json with the intent of leaving them unpublished and then people went ahead and published them to packagist anyways, so we had to take them out. You're trying to use composer to manage dependencies...which is what the other ticket wants to enable.

We used to add name/description/license/etc. to composer.json with the intent of leaving them unpublished and then people went ahead and published them to packagist anyways, so we had to take them out.

Although publishing packages is not my goal here, why was this side-effect detrimental to WMF? Because it introduced install options that are not 'official'?

In Element chat I raised this same issue. (Sorry for forking the thread.)

I asked for clarification on my bug report about invalid composer.json files, but the response was unsatisfactory; saying it was the same thing as the (stalled) RFC for Hybrid Extension Management. They are not the same thing in my mind, and I'm not trying to sneak the RFC request through the "backdoor". I do want to manage dependencies with Composer because that's what Composer is for. And, I don't understand how Composer would be incompatible with dependency resolution that is internal to MediaWiki or WMF Continuous Integration.

We used to add name/description/license/etc. to composer.json with the intent of leaving them unpublished and then people went ahead and published them to packagist anyways, so we had to take them out.

I honestly don't know why "publishing" mediawiki packages on Packagist or other registries is bad for WMF; while seemingly the only way 3rd parties can use Composer is to create their own registries.

The RFC itself is stalled because (in my opinion) legoktm is opposed to using Composer for dependency management - even though there is no other solution in existence. There's a POC for a basic homegrown system, but no good explanation that I can find in all this historical discussion for why that's necessary. So, to me, it boils down to "No Composer" because 'not invented here'. Just look at all the frameworks that are using Composer or are supported by Composer/installers -- especially Drupal and Wordpress which are close analogs for MediaWiki.

Anyway, my thought is that this is a better forum for discussion than a bug ticket - but if I'm wrong, I'm happy to discuss these topics in Phabricator or elsewhere.

and @Legoktm said:

Happy to discuss here or elsewhere. I closed the ticket because it was claiming that the composer.json files were invalid except they weren't (I made the PR to composer years ago to fix this!). At that point then the request is to use composer for managing extension dependencies, which I don't see as fundamentally different as the existing RfC. If you disagree with me, feel free to re-open the task

composer validate --no-check-publish

This just tells you that you have valid JSON. Since it omits the check for 'Name' and 'Description', it makes the extension unsuitable for interaction with Composer from GitHub, Gerrit, or any other git repository.

I'm going to re-open this ticket because I do think it is both valid, and separate from T250406

Regarding T250406, I would summarize it as "Create an official public MediaWiki extension registry" (like Hallo Welt does) so Extension authors don't have to bother publishing, and site builders can use Composer.

I'm going to re-open this ticket because I do think it is both valid, and separate from T250406

Regarding T250406, I would summarize it as "Create an official public MediaWiki extension registry" (like Hallo Welt does) so Extension authors don't have to bother publishing, and site builders can use Composer.

I would recommend you retitle this ticket, as "In many extensions, composer.json doesn't validate to spec" doesn't describe the actual problem you're concerned with. Whether or not it validates is orthoganal, you'd just like these extensions (why these ones?) to be managable with composer.

I think all MediaWiki extensions should have a valid composer.json. I tagged those projects I could identify as "affected". Not all extensions have a phabricator tag.

Can you help me understand why a full composer.json interferes with WMF procedures, requirements, tools or code?

I personally like the idea of installing extensions with Composer (we have to run Composer anyway for some extensions) but I'm not sure this validation issue is the main problem.

There are also a few extensions that do have name but are not actually registered on Packagist. (The ones I've seen I've fixed; they've mostly been people putting name: "mediawiki/foobar" without realising that they can't publish under that namespace. The fact that some of them are registered in custom registries is a separate matter, and one we should probably fix Template:Extension to take account of.)

This just tells you that you have valid JSON.

I'm not sure that's true. It also checks other aspects of the schema, and will fail if things are wrong (e.g. it helps find typos in keys etc.).

I think all MediaWiki extensions should have a valid composer.json. Can you help me understand why this interferes with WMF procedures, requirements, tools or code?

It's not that it'd interfere with WMF processes, as far as I know, but that it'd set more firmly the idea that Composer can be used to install extensions. And "officially" that's still not true.

Change 928164 had a related patch set uploaded (by Yaron Koren; author: Yaron Koren):

[mediawiki/extensions/ReplaceText@master] Add lines to composer.json to make it valid

https://gerrit.wikimedia.org/r/928164

Change 928164 merged by jenkins-bot:

[mediawiki/extensions/ReplaceText@master] Add lines to composer.json to make it valid

https://gerrit.wikimedia.org/r/928164

Change 938469 had a related patch set uploaded (by Cicalese; author: Cicalese):

[mediawiki/extensions/DisplayTitle@master] Add name and description properties to composer.json

https://gerrit.wikimedia.org/r/938469

Change 938469 merged by jenkins-bot:

[mediawiki/extensions/DisplayTitle@master] Add name and description properties to composer.json

https://gerrit.wikimedia.org/r/938469

Change 940444 had a related patch set uploaded (by Cicalese; author: Cicalese):

[mediawiki/extensions/DisplayTitle@REL1_40] Add name and description properties to composer.json

https://gerrit.wikimedia.org/r/940444

Change 940445 had a related patch set uploaded (by Cicalese; author: Cicalese):

[mediawiki/extensions/DisplayTitle@REL1_39] Add name and description properties to composer.json

https://gerrit.wikimedia.org/r/940445

Change 940445 merged by jenkins-bot:

[mediawiki/extensions/DisplayTitle@REL1_39] Add name and description properties to composer.json

https://gerrit.wikimedia.org/r/940445

Change 940444 merged by jenkins-bot:

[mediawiki/extensions/DisplayTitle@REL1_40] Add name and description properties to composer.json

https://gerrit.wikimedia.org/r/940444