Instead of config files, would it be better to "tag" the Wikidata items representing the template/category page somehow? This would allow the community to optimize configuration, without requiring file/repo changes.

@Magnus We discussed that, but if I remember correctly, we had the problem that wikidata maps the template/category names, but it does not help you with the attributes inside of the template. Or do you see a solution to that? Do you know how thoroughly the templates and categories are mapped on wikidata already?

You are correct that this would not solve the attribute mapping, short of some spectacularly horrible hackery.

Deeper analysis for this and all subtasks:

• The base path for all configuration files is https://meta.wikimedia.org/wiki/CommonsHelper2/Data_ (see https://meta.wikimedia.org/wiki/Special:PrefixIndex/CommonsHelper2/Data). We need a configuration variable for this. The default should be null (or an empty string), which should turn the entire feature off.
• We need a mapping from source URLs to the format "de.wikipedia" the CommonsHelper uses. As far as I can see this is achieved with simple string manipulations:
  • Extract the domain part from the URL.
  • Remove ".org" from the end of the string.
• Mapping from database identifiers like "dewiki" to "de.wikipedia" would also be possible, if needed:
  • Always add a dot before the word "wiki". Warning: Not at the beginning of the string, e.g. in "wikivoyage". Special handling for "mediawiki" is also needed.
  • Replace ".wiki" with ".wikipedia".
• The raw wikitext can be pulled via https://meta.wikimedia.org/w/index.php?title=CommonsHelper2/Data_de.wikipedia&action=raw, or the query API: https://meta.wikimedia.org/w/api.php?action=query&format=json&prop=revisions&titles=CommonsHelper2%2FData%20de.wikipedia&formatversion=2&rvprop=content&rvlimit=1&rvdir=newer
• We should cache these configuration files somehow, at least for 5 minutes. Not needed for an MVP, but we should make sure our architecture allows to insert a caching decorator.

Naming suggestions:

• $wgFileImporterCommonsHelperConfigBasePath
• FileImporter\Remote\MediaWiki\CommonsHelperConfigRetriever (includes the domain to config filename mapping)
• FileImporter\Services\Wikitext\CommonsHelperConfigParser turns the config page into a value object
• FileImporter\Data\WikitextConversions is the value object, intentionally not named "CommonsHelper", because I want it to be reusable for other sources that are not the CommonsHelper
• FileImporter\Services\Wikitext\WikitextContentValidator does all checks that don't change the wikitext
• FileImporter\Services\Wikitext\WikitextContentCleaner applies all replacements

Issues:

• I failed to find CommonsHelper2 running. https://tools.wmflabs.org/commonshelper2/ is "not currently serviced". @jan, is this intentional? https://commons.wikimedia.org/wiki/Commons:CommonsHelper_2 redirects to https://tools.wmflabs.org/commonshelper/, but this looks like it's the old version without the configuration files we want. https://commons.wikimedia.org/wiki/User:CommonsHelper2_Bot is blocked.
• I also failed to find the source code. https://bitbucket.org/magnusmanske/commonshelper/src doesn't look like it is working with the configuration files we want to use. Found at https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/tools/commonshelper2, but this repository is unmaintained since 6 years.

@Lea_WMDE, FYI, @Magnus just confirmed my impression: CommonsHelper2 is not running anywhere since toolserver.org got shut down. This does have some consequences:

• The config files are not up to date.
  • This means the quality of the auto-replacements will not be as good as it could be, until the community starts updating the config files again.
  • This means we need to make it obvious how the community can update the auto-replacements.
  • This includes easy-to-reach documentation for the format (and which parts we support).
• If we are continuing our plan (and I believe we should do so), we are basically hijacking the config files, as we are the only user then.
  • This gives us even more freedom. We could even remove parts we are not supporting, and re-add them later.
  • I want us to follow the principle I already outlined above: let's bind as less code as possible against this specific config file format and the source we fetch it from. (This refers to the two class names that include …CommonsHelper… above.)