Some options that come to mind:

• Don't bother with making libraries optional, just require everything that might be useful for someone even if for the current user it is not. Storage is not a big deal these days, and from a security POV having files on the server that are never going to be called is maybe not that big of a deal (although an attacker with write access to the library could probably take advantage of autoloading and hijack some core class). OTOH Composer tends to be pretty slow with many libraries.
• Force optional libraries to be optional extensions with required libraries instead. Extensions have a well-documented and well-understood upgrade path and all kinds of tooling. Quite a bit of an overhead for the developer though.
• Just put the version into the suggest block (so e.g. instead of "monolog/monolog": "Flexible debug logging system" have "monolog/monolog": "Flexible debug logging system (use 1.24.0)") and leave all the work to the user. Pretty bad user experience, especially when the required version changes.
• Put the optional dependencies in composer.local.json-sample, in some inactive format (would be nice if JSON allowed comments, but we can probably just put them in a _require block, put an empty require block next to it and the user can move things over). Update handling could be handled by specifying everything other than the supported version in a conflicts block, so at least there's an error telling the user to fix things (this applies to the previous bullets too).
• Use something like apache site config: have a subdirectory for all the optional requires as separate JSON files, have a composer-merge-plugin rule with a pattern which does not by default match those files and enable them by creating symlinks (maybe support a custom composer command to make it easier). This has sane update handling, but it seems a bit overcomplicated. Also maybe people run MediaWiki on crappy OSes with no symlink support?

There is also the problem of how to do CI when the optional library is not included in mediawiki/vendor (so far all are, I believe).

In T212274#4833289, @Tgr wrote:

Some options that come to mind:

• Don't bother with making libraries optional, just require everything that might be useful for someone even if for the current user it is not. Storage is not a big deal these days, and from a security POV having files on the server that are never going to be called is maybe not that big of a deal (although an attacker with write access to the library could probably take advantage of autoloading and hijack some core class). OTOH Composer tends to be pretty slow with many libraries.

We definitely get people complaining now and again vendor is too bloated (which is why Lego fixed the bundled vendor in the tarballs). I would agree it shouldn't really be a problem in most cases

Then for dev dependancies, we have past events like this T180231: MW 1.27 and 1.28 require-dev versions of phpunit with known security issues

As a sysadmin I prefer librairies unused in production are not enabled with the --no-dev profile of Composer, for performance and security reasons. Optional librairies (monolog, psysh, …) can be defined in require-dev with the version and in suggest when it is relevant for a use-case in production – in fact it is what is currenlty done with the few suggests (monolog, kafka-php, avro).

And it could be documented to add optional librairies in composer.local.json:

{
        "require": {
                "monolog/monolog": "*"
        },
        "extra": {
                "merge-plugin": {
                        "include": [
                                "extensions/example/composer.json"
                        ]
                }
        }
}

Until now I didn’t use the composer.local.json for optional librairies (directly composer.json although it is not optimal) but I’ve just tested the composer.local.json above with Composer 1.x and 2.x and it works fine. The varsion can be specified with a * since the version is determined by the require-dev in composer.json, but it seems the version must be precised for second-optional libraries, for instance if there is "rollbar/rollbar": "*" in composer.local.json (monolog suggests it with version ^1.3 in its require-dev), the version 2.1.0 is installed, which is not desirable.

For Avro, there is a proposal to remove it: T265967: Proposal: drop avro dependency from mediawiki.