Page MenuHomePhabricator
Create Task
Maniphest T192376

Implement external user name handling in the file importer
Closed, ResolvedPublic8 Estimated Story Points
Actions

Description

Original Question:
With the FileImporter extension we are copying the file page's history from another wiki. What should happen with the user names of the people who made the edits? Should they be linked to their Commons account or should they be linked to the wiki where they actually made that edit?

Our decision
We want to link user names to the user page of the wiki where they made the edit, since they executed the edit in this role. Furthermore, it will make it more obvious on which edit which wiki was made, if we prefix them, as shown here:

Screenshot-2018-4-24 Revision history of Module Arguments - Wikimedia Commons.png (465×1 px, 75 KB)

Task

  • Prefix and link users in the version history to the wiki where they made the edit

For technical details see T180466: Investigate possible issues with imported user names for FileImporter

Details

SubjectRepoBranchLines +/-
Use interwiki lookup to get prefixesmediawiki/extensions/FileImportermaster+40 -13
ExternalUserNames: Update partly incomplete documentationmediawiki/coremaster+24 -4
Make side-effect of ExternalUserNames::applyPrefix() more obviousmediawiki/extensions/FileImportermaster+2 -3
Always use plain username for file revisionsmediawiki/extensions/FileImportermaster+8 -5
Use default 'imported' prefix when prefix emptymediawiki/extensions/FileImportermaster+105 -1
Add basic logging to SiteTableSourceInterWikiLookupmediawiki/extensions/FileImportermaster+36 -4
Add external username handlingmediawiki/extensions/FileImportermaster+50 -11
Add interface to allow interwiki referencesmediawiki/extensions/FileImportermaster+189 -3
Customize query in gerrit

Related Objects

Mentioned In
rEFLI613e495f3c52: Final NoteDb migration updates
rEFLI4dade443d498: Make side-effect of ExternalUserNames::applyPrefix() more obvious
rEFLI5e9e78bb5fa3: Always use plain username for file revisions
rEFLIa6a7f9435961: Make side-effect of ExternalUserNames::applyPrefix() more obvious
rEFLIfba031e2d3b8: Use default 'imported' prefix when prefix empty
rEFLIbb50c49fb521: Use default 'imported' prefix when prefix empty
rEFLIdae6812f38b5: Always use plain username for file revisions
rEFLIff27face3120: Always use plain username for file revisions
rEFLIea3ed975f044: Use default 'imported' prefix when prefix empty
rEFLIc870c05b210a: Add basic logging to SiteTableSourceInterWikiLookup
rEFLI133244e45c6a: Add basic logging to SiteTableSourceInterWikiLookup
rEFLIe5fdf2176a79: Always use plain username for file revisions
rEFLIfc032729db89: Use default 'imported' prefix when prefix empty
rEFLI4974027d1ebb: Add external username handling
rEFLId365a5f8a6ad: Add external username handling
rEFLI5d26f05b9c0c: Add external username handling
rEFLI362e8582c318: Add interface to allow interwiki references
rEFLIbea311793f13: Add interface to allow interwiki references
rEFLI906354b88792: Add external username handling
rEFLId124524bf241: Add interface to allow interwiki references
rEFLIed63207cd858: Add external username handling
rEFLI0c08bd1983c9: Add interface to allow interwiki references
rEFLI43b7fd83703e: Add external username handling
rEFLI3becab69bf79: Add external username handling
rEFLIffbaf556034f: Add interface to allow interwiki references
rEFLIf3cf0a3d351e: Add external username handling
rEFLI32ad31b92caa: Add interface to allow interwiki references
rEFLIba0963bb9bc3: Add external username handling
rEFLIc5194a36f3a3: Add interface to allow interwiki references
rEFLI967082c01ce1: Add interface to allow interwiki references
rEFLIcdb7fce6c0a5: Add interface to allow interwiki references
rEFLI6503ae96e558: Add interface to allow interwiki references
Mentioned Here
T180466: Investigate possible issues with imported user names for FileImporter

Event Timeline

Lea_WMDE created this task.Apr 17 2018, 3:31 PM
Restricted Application added a project: TCB-Team (now WMDE-TechWish). · View Herald TranscriptApr 17 2018, 3:31 PM
Lea_WMDE added a comment.Apr 17 2018, 3:32 PM

WMDE-Design we need to consider whether we want to trust Single User Login, and link to the current wiki, or link to the source wiki user pages

Lea_WMDE set the point value for this task to 8.Apr 24 2018, 2:35 PM
WMDE-Fisch subscribed.Apr 24 2018, 4:10 PM

User links with prefix ( reference to the source wiki ) will look like this in the history:

Screenshot-2018-4-24 Revision history of Module Arguments - Wikimedia Commons.png (465×1 px, 75 KB)

Lea_WMDE triaged this task as Medium priority.Apr 25 2018, 1:27 PM
Lea_WMDE updated the task description. (Show Details)
Lea_WMDE moved this task from Tickets in preparation/investigation to Tickets ready for pickup on the Move-Files-To-Commons board.Apr 25 2018, 1:32 PM
thiemowmde moved this task from Incoming to Move files to Commons on the TCB-Team (now WMDE-TechWish) board.May 11 2018, 5:07 PM
gerritbot subscribed.May 12 2018, 3:04 PM

Change 432729 had a related patch set uploaded (by WMDE-Fisch; owner: WMDE-Fisch):
[mediawiki/extensions/FileImporter@master] Add interface to allow interwiki references

https://gerrit.wikimedia.org/r/432729

gerritbot added a project: Patch-For-Review.May 12 2018, 3:04 PM
WMDE-Fisch mentioned this in rEFLI6503ae96e558: Add interface to allow interwiki references.May 12 2018, 3:07 PM
Tobi_WMDE_SW added a project: WMDE-QWERTY-Sprint-2018-05-16.May 16 2018, 10:51 AM
Tobi_WMDE_SW moved this task from Tickets ready for pickup to Tickets in sprint on the Move-Files-To-Commons board.
WMDE-Fisch claimed this task.May 22 2018, 10:44 AM
WMDE-Fisch moved this task from Sprint Backlog to Doing on the WMDE-QWERTY-Sprint-2018-05-16 board.
Lea_WMDE added a project: WMDE-QWERTY-Sprint-2018-05-23.May 23 2018, 1:17 PM
Lea_WMDE moved this task from Sprint Backlog to Doing on the WMDE-QWERTY-Sprint-2018-05-23 board.May 23 2018, 1:19 PM
WMDE-Fisch mentioned this in rEFLIcdb7fce6c0a5: Add interface to allow interwiki references.May 24 2018, 8:49 AM
WMDE-Fisch mentioned this in rEFLI967082c01ce1: Add interface to allow interwiki references.May 25 2018, 1:11 PM
WMDE-Fisch mentioned this in rEFLIc5194a36f3a3: Add interface to allow interwiki references.May 25 2018, 1:23 PM
gerritbot added a comment.May 30 2018, 7:21 AM

Change 436225 had a related patch set uploaded (by WMDE-Fisch; owner: WMDE-Fisch):
[mediawiki/extensions/FileImporter@master] Add external username handling

https://gerrit.wikimedia.org/r/436225

WMDE-Fisch mentioned this in rEFLIba0963bb9bc3: Add external username handling.May 30 2018, 7:24 AM
WMDE-Fisch mentioned this in rEFLI32ad31b92caa: Add interface to allow interwiki references.May 30 2018, 7:32 AM
WMDE-Fisch mentioned this in rEFLIf3cf0a3d351e: Add external username handling.
WMDE-Fisch mentioned this in rEFLIffbaf556034f: Add interface to allow interwiki references.May 31 2018, 2:16 PM
WMDE-Fisch mentioned this in rEFLI3becab69bf79: Add external username handling.
WMDE-Fisch mentioned this in rEFLI43b7fd83703e: Add external username handling.May 31 2018, 2:27 PM
WMDE-Fisch moved this task from Doing to Review on the WMDE-QWERTY-Sprint-2018-05-23 board.May 31 2018, 2:27 PM
WMDE-Fisch mentioned this in rEFLI0c08bd1983c9: Add interface to allow interwiki references.May 31 2018, 2:53 PM
WMDE-Fisch mentioned this in rEFLIed63207cd858: Add external username handling.
WMDE-Fisch mentioned this in rEFLId124524bf241: Add interface to allow interwiki references.May 31 2018, 3:00 PM
WMDE-Fisch mentioned this in rEFLI906354b88792: Add external username handling.May 31 2018, 3:08 PM
WMDE-Fisch mentioned this in rEFLIbea311793f13: Add interface to allow interwiki references.Jun 1 2018, 7:20 AM
WMDE-Fisch mentioned this in rEFLI362e8582c318: Add interface to allow interwiki references.Jun 1 2018, 1:12 PM
WMDE-Fisch mentioned this in rEFLI5d26f05b9c0c: Add external username handling.Jun 1 2018, 1:19 PM
WMDE-Fisch mentioned this in rEFLId365a5f8a6ad: Add external username handling.Jun 1 2018, 1:22 PM
gerritbot added a comment.Jun 1 2018, 2:18 PM

Change 432729 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Add interface to allow interwiki references

https://gerrit.wikimedia.org/r/432729

WMDE-Fisch mentioned this in rEFLI4974027d1ebb: Add external username handling.Jun 1 2018, 2:58 PM
ReleaseTaggerBot added a project: MW-1.32-notes (WMF-deploy-2018-06-05 (1.32.0-wmf.7)).Jun 1 2018, 3:00 PM
gerritbot added a comment.Jun 1 2018, 3:13 PM

Change 436225 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Add external username handling

https://gerrit.wikimedia.org/r/436225

WMDE-Fisch added a comment.Jun 1 2018, 5:34 PM

So the patches are merged and deployed on beta. I could run tests there because the beta configuration kind of accurately mirrors the real production cluster. All seems to work fine. I moved the following images from en.wp.beta / de.wp.beta to commons.beta using the FileExporter link and the generated interwiki prefixes and links to the user profiles are working.

https://commons.wikimedia.beta.wmflabs.org/w/index.php?title=File:MO-WMF.jpg&action=history
https://commons.wikimedia.beta.wmflabs.org/w/index.php?title=File:OOjs_UI_icon_alert.svg&action=history

This is for the text history. The file history username do not support external user name handling with prefixes. So there the plain username is displayed. Still, in these cases we at least make sure, that the user is created via CentralAuth SUL if the account is not present. This is also done via the new external username handling support.

Another hint here:

Due to the 2-level linking (here de:de) this also means that we probably can do general interwiki linking, for example with the source-files location in the comment. In some earlier ticker dealing with that I thought that this is not possible when linking from commons to - for example - fr.wp or de.wikivoyage.

We could revisit that. The foundation for this is implemented now and we just have to make use of the prefixes generated here.

WMDE-Fisch moved this task from Review to Demo on the WMDE-QWERTY-Sprint-2018-05-23 board.Jun 1 2018, 5:34 PM
gerritbot added a comment.Jun 4 2018, 3:27 PM

Change 437272 had a related patch set uploaded (by WMDE-Fisch; owner: WMDE-Fisch):
[mediawiki/extensions/FileImporter@master] Use default 'imported' prefix when prefix empty

https://gerrit.wikimedia.org/r/437272

gerritbot added a comment.Jun 4 2018, 3:28 PM

Change 437273 had a related patch set uploaded (by WMDE-Fisch; owner: WMDE-Fisch):
[mediawiki/extensions/FileImporter@master] Always use plain username for file revisions

https://gerrit.wikimedia.org/r/437273

WMDE-Fisch mentioned this in rEFLIfc032729db89: Use default 'imported' prefix when prefix empty.Jun 4 2018, 3:29 PM
WMDE-Fisch mentioned this in rEFLIe5fdf2176a79: Always use plain username for file revisions.
gerritbot added a comment.Jun 4 2018, 3:57 PM

Change 437280 had a related patch set uploaded (by WMDE-Fisch; owner: WMDE-Fisch):
[mediawiki/extensions/FileImporter@master] Add basic logging to SiteTableSourceInterWikiLookup

https://gerrit.wikimedia.org/r/437280

WMDE-Fisch mentioned this in rEFLI133244e45c6a: Add basic logging to SiteTableSourceInterWikiLookup.Jun 4 2018, 3:59 PM
thiemowmde mentioned this in rEFLIc870c05b210a: Add basic logging to SiteTableSourceInterWikiLookup.Jun 4 2018, 5:00 PM
gerritbot added a comment.Jun 5 2018, 9:18 AM

Change 437280 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Add basic logging to SiteTableSourceInterWikiLookup

https://gerrit.wikimedia.org/r/437280

WMDE-Fisch mentioned this in rEFLIea3ed975f044: Use default 'imported' prefix when prefix empty.Jun 5 2018, 9:53 AM
WMDE-Fisch mentioned this in rEFLIff27face3120: Always use plain username for file revisions.
WMDE-Fisch mentioned this in rEFLIdae6812f38b5: Always use plain username for file revisions.Jun 5 2018, 9:55 AM
thiemowmde mentioned this in rEFLIbb50c49fb521: Use default 'imported' prefix when prefix empty.Jun 5 2018, 10:49 AM
thiemowmde mentioned this in rEFLIfba031e2d3b8: Use default 'imported' prefix when prefix empty.Jun 5 2018, 10:57 AM
gerritbot added a comment.Jun 5 2018, 11:02 AM

Change 437272 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Use default 'imported' prefix when prefix empty

https://gerrit.wikimedia.org/r/437272

gerritbot added a comment.Jun 5 2018, 11:51 AM

Change 437452 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/core@master] ExternalUserNames: Update partly incomplete documentation

https://gerrit.wikimedia.org/r/437452

gerritbot added a comment.Jun 5 2018, 11:56 AM

Change 437453 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/extensions/FileImporter@master] Make side-effect of ExternalUserNames::applyPrefix() more obvious

https://gerrit.wikimedia.org/r/437453

thiemowmde mentioned this in rEFLIa6a7f9435961: Make side-effect of ExternalUserNames::applyPrefix() more obvious.Jun 5 2018, 11:57 AM
thiemowmde mentioned this in rEFLI5e9e78bb5fa3: Always use plain username for file revisions.Jun 5 2018, 12:01 PM
thiemowmde mentioned this in rEFLI4dade443d498: Make side-effect of ExternalUserNames::applyPrefix() more obvious.
gerritbot added a comment.Jun 5 2018, 12:06 PM

Change 437273 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Always use plain username for file revisions

https://gerrit.wikimedia.org/r/437273

gerritbot added a comment.Jun 5 2018, 1:23 PM

Change 437453 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Make side-effect of ExternalUserNames::applyPrefix() more obvious

https://gerrit.wikimedia.org/r/437453

• MichaelSchoenitzer_WMDE subscribed.Jun 5 2018, 1:43 PM

When importing files from commons to test.wikimedia.beta.wmflabs.org this does not work. I don't get any link and the usernames are prefixed with imported>. Is this expected behavior?

https://test.wikimedia.beta.wmflabs.org/w/index.php?title=File:Snap_fastener_female_(outer)_side_components.jpg&action=history

gerritbot added a comment.Jun 5 2018, 1:56 PM

Change 437452 merged by jenkins-bot:
[mediawiki/core@master] ExternalUserNames: Update partly incomplete documentation

https://gerrit.wikimedia.org/r/437452

WMDE-Fisch added a comment.Jun 6 2018, 9:26 AM
In T192376#4257548, @MichaelSchoenitzer_WMDE wrote:

When importing files from commons to test.wikimedia.beta.wmflabs.org this does not work. I don't get any link and the usernames are prefixed with imported>. Is this expected behavior?

https://test.wikimedia.beta.wmflabs.org/w/index.php?title=File:Snap_fastener_female_(outer)_side_components.jpg&action=history

So the test.wikimeidia.beta server is not part of the Interwiki map and so it's not possible to interwiki link to it. That's why no interwiki prefix can be retrieved. The default behavior of the Importer used for Wikitext imports in these cases is using imported as prefix. So that was implemented here. A discus sable alternative would be not prefixing at all. But I think that would go against the "new" policy there.

WMDE-Fisch added a comment.Jun 6 2018, 9:49 AM

So looking at the logs, I can see, that the site object for the test server could be retrieved from the sites table, but no interwikiIds were configured. In that case the prefix will be empty and we set the imported prefix for the usernames in the revision history.

WMDE-Fisch added a comment.Jun 6 2018, 10:07 AM

Also ( sry I realized that just now, looking at the link you provided ): The interwiki prefixed links for test images ( e.g. from de.wp.beta ) only work when importing to commons.wikimedia.beta. Since only the FileImporter there is correctly configured to use the sites table and retrieve the interwikiids from it. The FileImporter on test.wm.beta is configured to accept imports from everywhere and does not respect the sites table.

WMDE-Fisch added a comment.Jun 6 2018, 10:11 AM

Last comment ( for now ) to this:

This mirrors best what would happen in production :-)

thiemowmde mentioned this in rEFLI613e495f3c52: Final NoteDb migration updates.Jun 9 2018, 1:36 AM
gerritbot added a comment.Jun 11 2018, 10:33 AM

Change 439566 had a related patch set uploaded (by WMDE-Fisch; owner: WMDE-Fisch):
[mediawiki/extensions/FileImporter@master] Use interwiki lookup to get prefixes

https://gerrit.wikimedia.org/r/439566

WMDE-Fisch added a comment.Jun 11 2018, 10:34 AM

So I tested this different other pages and looked into the sitestable db on the production server. It seems that in many cases the inter wiki prefixes are not set or incomplete in the sites configuration.

I explored the alternative in the patch above. Some general issues with inter wiki linking remain, since adding the lang code is not valid for all wikis and the lookup should probably be improved for links where the source is not Wikipedia.

Lea_WMDE closed this task as Resolved.Jun 12 2018, 8:19 AM
Lea_WMDE moved this task from Demo to Done on the WMDE-QWERTY-Sprint-2018-05-23 board.
gerritbot added a comment.Jul 2 2018, 9:42 AM

Change 439566 abandoned by WMDE-Fisch:
Use interwiki lookup to get prefixes

https://gerrit.wikimedia.org/r/439566

thiemowmde removed a project: TCB-Team (now WMDE-TechWish).Jan 7 2022, 2:59 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL