Map templates to Commons templates whenever possible
Closed, ResolvedPublic13 Story Points

Description

Motivation
When files are moved from e.g. Wikipedia to Commons, their file info more often than not needs to be tweaked. For example, the information template and all its parameters have different names in different wikis. We want to match as many templates as possible automatically, so that users don't need to spend their time with busy work.

Acceptance Criteria

  • Replace all templates listed under "Transfer" in the CommonsHelper2's config files for the source wiki.
  • When the users enter SpecialPage:ImportFile for the first time, the replaced contents should already be visible in the file info wikitext.
  • If some mapping happened, the edit summary field of the SpecialPage:ImportFile will contain: "Replaced x <sourcewiki> template names with <targetwiki> template names"

Notes

  • It is very important to work test driven here, with some file description pages against which expected results can be checked
  • This will probably be a simple search a replace
  • The config file lists all mappings in Templates/Transfer. The header is always the source wiki's template name, and below comes <Commons template name>|<Commons Param name>=<source wiki's param name>|<another Commons Param name>=<source wiki's other param name>...
  • The @ that sometimes appear before the param names should be ignored for now

Related Objects

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 441556 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/extensions/FileImporter@master] Add tests for "Transfer" section in CommonsHelperConfigParser

https://gerrit.wikimedia.org/r/441556

Change 441557 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/extensions/FileImporter@master] Add basic set of test cases for WikiTextContentCleaner

https://gerrit.wikimedia.org/r/441557

Change 441561 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/extensions/FileImporter@master] Rewrite WikiTextContentCleaner to support nested templates

https://gerrit.wikimedia.org/r/441561

Change 441777 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/extensions/FileImporter@master] Add support for template parameters in WikiTextConversions

https://gerrit.wikimedia.org/r/441777

Change 441807 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/extensions/FileImporter@master] Add full parameter support to CommonsHelperConfigParser

https://gerrit.wikimedia.org/r/441807

Change 441820 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/extensions/FileImporter@master] [WIP] Add parameter parser/tokenizer to WikiTextContentCleaner

https://gerrit.wikimedia.org/r/441820

Change 441874 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/extensions/FileImporter@master] Add support for numeric parameters to WikiTextContentCleaner

https://gerrit.wikimedia.org/r/441874

Change 441227 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Replace known templates with Commons templates

https://gerrit.wikimedia.org/r/441227

Change 441552 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Improve and add tests for WikiTextConversions::swapTemplate

https://gerrit.wikimedia.org/r/441552

Change 441556 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Add tests for "Transfer" section in CommonsHelperConfigParser

https://gerrit.wikimedia.org/r/441556

Change 441557 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Add basic set of test cases for WikiTextContentCleaner

https://gerrit.wikimedia.org/r/441557

There should probably be a onwiki regex replace list in the MediaWiki namespace which can be edited by admins, in order to allow easy maintenance of the replacement stuff.

Hi @Steinsplitter we are using these files for the replacement, and they can be edited for any changes you want to make.
For now, we are not including regex, since this has higher risks of exploitation, and deep diving into this topic is something we keep for future versions :)

Change 441561 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Rewrite WikiTextContentCleaner to support nested templates

https://gerrit.wikimedia.org/r/441561

Change 441777 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Add support for template parameters in WikiTextConversions

https://gerrit.wikimedia.org/r/441777

Change 442282 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/extensions/FileImporter@master] Add missing test case to WikiTextContentCleanerTest

https://gerrit.wikimedia.org/r/442282

Change 441807 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Add full parameter support to CommonsHelperConfigParser

https://gerrit.wikimedia.org/r/441807

Change 441820 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Add parameter parser/tokenizer to WikiTextContentCleaner

https://gerrit.wikimedia.org/r/441820

Steinsplitter added a comment.EditedJun 27 2018, 3:51 PM

Hi @Steinsplitter we are using these files for the replacement, and they can be edited for any changes you want to make.
For now, we are not including regex, since this has higher risks of exploitation, and deep diving into this topic is something we keep for future versions :)

Ah, ok :-) Perfect.

Should the pages be semi-protected to prevent abuse by unregistered users (I can do so via TitleBlacklist if neccesary)?

Change 441874 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Add support for numeric parameters to WikiTextContentCleaner

https://gerrit.wikimedia.org/r/441874

Change 442282 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Add missing test case to WikiTextContentCleanerTest

https://gerrit.wikimedia.org/r/442282

Change 442370 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/extensions/FileImporter@master] Micro optimizations to the template parameter parser

https://gerrit.wikimedia.org/r/442370

@Steinsplitter I think we would propose to go without protection for now, to keep aligned to "trust first". If there are misuses, we will happily take you up on your offer, though!

Change 442370 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Micro optimizations to the template parameter parser

https://gerrit.wikimedia.org/r/442370

Vvjjkkii renamed this task from Map templates to Commons templates whenever possible to e4caaaaaaa.Jul 1 2018, 1:10 AM
Vvjjkkii raised the priority of this task from Normal to High.
Vvjjkkii removed thiemowmde as the assignee of this task.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed the point value for this task.
Vvjjkkii edited subscribers, added: thiemowmde; removed: gerritbot.
Bodhisattwa renamed this task from e4caaaaaaa to Map templates to Commons templates whenever possible.Jul 1 2018, 1:21 PM
Bodhisattwa lowered the priority of this task from High to Normal.
Bodhisattwa assigned this task to thiemowmde.
Bodhisattwa updated the task description. (Show Details)
Bodhisattwa added a subscriber: gerritbot.
Pikne added a subscriber: Pikne.Jul 2 2018, 9:48 AM

Currently prefilled edit summary for this is in importer's interface language. I suppose wiki content language (English for Commons) would be preferrable instead.

Currently prefilled edit summary for this is in importer's interface language. I suppose wiki content language (English for Commons) would be preferrable instead.

I created a separate task for that: T198616

thiemowmde updated the task description. (Show Details)
thiemowmde moved this task from Review to Demo on the WMDE-QWERTY-Sprint-2018-06-19 board.
Pikne added a comment.Jul 2 2018, 9:05 PM

Currently prefilled edit summary for this is in importer's interface language. I suppose wiki content language (English for Commons) would be preferrable instead.

Now I note an odd behaviour related to this edit summary. Here revision differences show that the only automatic template replacement was made in revision that has "Imported with FileImporter..." edit summary, not in the one that has edit summary about replacement.

Tobi_WMDE_SW set the point value for this task to 13.Jul 3 2018, 8:10 AM
Lea_WMDE closed this task as Resolved.Wed, Jul 18, 3:39 PM