Page MenuHomePhabricator
Create Task
Maniphest T225515

Correctly link pages from non-english non-Wikipedia pages
Closed, ResolvedPublic2 Estimated Story Points
Actions

Description

Motivation
As of June 05 2019 interwiki linking for non-en-wikis of non-Wikipedias (e.g. de.wikivoyage, it.wikiversity) is not working yet. What happens instead, is that the link is interpreted as a link to the language's Wikipedia. This affects the file page itself, but not the history.
As an example, on de.Wikivoyage there are 669 files on de.wikivoyage.org, out of which 197 contain links to other pages of the wiki.

Currently, not all sister projects are set up to use FileImporter (see the existing config files here)

Acceptance Criteria

  • Also non-en-wikis of non-Wikipedias get correctly prefixed.

Dev notes
Probably we could get the interwiki links from mediawiki, see also previous work and investigations:

After this is solved we could also get rid of the articifical config that whitelisted how to resolve specific paths

Details

SubjectRepoBranchLines +/-
Fix two-hop interwiki prefix lookup reducing "foo.org" to "org"mediawiki/extensions/FileImportermaster+41 -18
Update outdated FileImporterInterWikiMap docsmediawiki/extensions/FileImportermaster+5 -4
Strip the "$1" before scraping for the API URLmediawiki/extensions/FileImportermaster+1 -1
Make fewer assumptions about subdomain structuremediawiki/extensions/FileImportermaster+42 -29
Handle interwiki edge casemediawiki/extensions/FileImportermaster+40 -18
Correctly prefix links to non-english non-Wikipedia sitesmediawiki/extensions/FileImportermaster+372 -51
Customize query in gerrit

Related Objects
Search...

Event Timeline

Lea_WMDE created this task.Jun 11 2019, 2:22 PM
Lea_WMDE moved this task from Backlog to Tickets in preparation/investigation on the Move-Files-To-Commons board.
Lea_WMDE updated the task description. (Show Details)Jun 18 2019, 1:29 PM
Lea_WMDE set the point value for this task to 8.
awight subscribed.Jun 25 2019, 8:55 AM

One thing to try: parse the file info into HTML using restbase on the source wiki, then transform back into wikitext using commonswiki restbase.

Lea_WMDE moved this task from Tickets in preparation/investigation to Tickets ready for pickup on the Move-Files-To-Commons board.Jun 25 2019, 4:30 PM
awight moved this task from Tickets ready for pickup to Tickets in sprint on the Move-Files-To-Commons board.Jul 1 2019, 8:34 AM
awight added a project: WMDE-QWERTY-Sprint-2019-06-26.
awight added a comment.EditedJul 1 2019, 8:59 AM

Before I burn too much more time looking for this in the wild, does anyone have examples of image file info using multi-prefix interwiki links?

Lea_WMDE added a comment.Jul 1 2019, 9:06 AM

Take a look here. I think on de.wikivoyage, an example was for example an image of Berlin that linked to the wikivoyage article of Berlin

awight added a comment.Jul 1 2019, 10:16 AM

Following up on my comment above, I was mistaken about the task. Now I understand that we need to prefix *any* links, including plain (non-interwiki) wiki links to pages on the same site.

Andrew-WMDE claimed this task.Jul 2 2019, 8:21 AM
Andrew-WMDE moved this task from Sprint Backlog to Doing on the WMDE-QWERTY-Sprint-2019-06-26 board.
gerritbot added a comment.Jul 2 2019, 9:27 AM

Change 520197 had a related patch set uploaded (by Andrew-WMDE; owner: Andrew-WMDE):
[mediawiki/extensions/FileImporter@master] [WIP] Correctly link pages from non-english non-Wikipedia pages

https://gerrit.wikimedia.org/r/520197

gerritbot added a project: Patch-For-Review.Jul 2 2019, 9:27 AM
Andrew-WMDE mentioned this in rEFLI3283de807f81: [WIP] Correctly link pages from non-english non-Wikipedia pages.Jul 2 2019, 9:27 AM
Andrew-WMDE mentioned this in rEFLIb9705a0a1566: [WIP] Correctly link pages from non-english non-Wikipedia pages.Jul 2 2019, 3:55 PM
Andrew-WMDE mentioned this in rEFLI36f2f7139c50: [WIP] Correctly link pages from non-english non-Wikipedia pages.Jul 3 2019, 11:55 AM
awight moved this task from Doing to Review on the WMDE-QWERTY-Sprint-2019-06-26 board.Jul 5 2019, 9:30 AM
awight added a comment.Jul 5 2019, 9:42 AM

When we get to the demo, here's a nice edge case which leads to a functional stack of four interwiki prefixes: https://fr.wikisource.org/wiki/Fichier:Alfred_Adler.png . For example, https://test.wikipedia.beta.wmflabs.org/wiki/Special:ImportFile?clientUrl=https://fr.wikisource.org/wiki/Fichier:Alfred_Adler.png .

I'll have to see whether test.beta can do all the interwiki things, we might need to adjust.

gerritbot added a comment.Jul 5 2019, 2:43 PM

Change 520197 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Correctly prefix links to non-english non-Wikipedia sites

https://gerrit.wikimedia.org/r/520197

awight mentioned this in rEFLI4511cfa64713: Correctly prefix links to non-english non-Wikipedia sites.Jul 5 2019, 2:44 PM
ReleaseTaggerBot added a project: MW-1.34-notes (1.34.0-wmf.13; 2019-07-09).Jul 5 2019, 3:00 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 5 2019, 3:10 PM
gerritbot added a comment.Jul 8 2019, 1:31 PM

Change 521270 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/FileImporter@master] Handle interwiki edge case

https://gerrit.wikimedia.org/r/521270

gerritbot added a project: Patch-For-Review.Jul 8 2019, 1:31 PM
gerritbot added a comment.Jul 8 2019, 6:23 PM

Change 521270 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Handle interwiki edge case

https://gerrit.wikimedia.org/r/521270

awight mentioned this in rEFLIe2c46d40007d: Handle interwiki edge case.Jul 8 2019, 6:23 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 8 2019, 7:11 PM
gerritbot added a comment.Jul 9 2019, 9:38 AM

Change 521448 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/FileImporter@master] Make fewer assumptions about subdomain structure

https://gerrit.wikimedia.org/r/521448

gerritbot added a project: Patch-For-Review.Jul 9 2019, 9:38 AM
gerritbot added a comment.Jul 9 2019, 10:47 AM

Change 521448 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Make fewer assumptions about subdomain structure

https://gerrit.wikimedia.org/r/521448

WMDE-Fisch mentioned this in rEFLI468102fee148: Make fewer assumptions about subdomain structure.Jul 9 2019, 10:48 AM
Maintenance_bot removed a project: Patch-For-Review.Jul 9 2019, 11:10 AM
gerritbot added a comment.Jul 9 2019, 11:17 AM

Change 521472 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/FileImporter@master] Strip the "$1" before scraping for the API URL

https://gerrit.wikimedia.org/r/521472

gerritbot added a project: Patch-For-Review.Jul 9 2019, 11:17 AM
WMDE-Fisch added a project: WMDE-QWERTY-Spike-2019-07-09.Jul 9 2019, 11:39 AM
WMDE-Fisch changed the point value for this task from 8 to 2.
Tobi_WMDE_SW moved this task from Sprint Backlog to Review on the WMDE-QWERTY-Spike-2019-07-09 board.Jul 9 2019, 11:39 AM
gerritbot added a comment.Jul 9 2019, 2:21 PM

Change 521472 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Strip the "$1" before scraping for the API URL

https://gerrit.wikimedia.org/r/521472

WMDE-Fisch mentioned this in rEFLIc5e5c6da66e6: Strip the "$1" before scraping for the API URL.Jul 9 2019, 2:22 PM
WMDE-Fisch moved this task from Review to Demo on the WMDE-QWERTY-Sprint-2019-06-26 board.Jul 9 2019, 2:37 PM

Can be tested with https://commons.wikimedia.beta.wmflabs.org/w/index.php?title=Special:ImportFile&clientUrl=https%3A%2F%2Fde.wiktionary.beta.wmflabs.org%2Fwiki%2FDatei%3ABlue_Test.png

See the link at the bottom of the rendered preview wikt:de:Sandbox and the original here: https://de.wiktionary.beta.wmflabs.org/wiki/Datei:Blue_Test.png

Maintenance_bot removed a project: Patch-For-Review.Jul 9 2019, 3:10 PM
awight moved this task from Review to Demo on the WMDE-QWERTY-Spike-2019-07-09 board.Jul 11 2019, 9:03 AM
Lea_WMDE closed this task as Resolved.Jul 25 2019, 1:51 PM
Lea_WMDE moved this task from Demo to Done on the WMDE-QWERTY-Sprint-2019-06-26 board.
Lea_WMDE mentioned this in T196976: Allow two- and one-path interwiki linking for user names.Jul 29 2019, 1:27 PM
gerritbot added a comment.Sep 16 2020, 8:46 AM

Change 627752 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/extensions/FileImporter@master] Update outdated FileImporterInterWikiMap docs

https://gerrit.wikimedia.org/r/627752

gerritbot added a project: Patch-For-Review.Sep 16 2020, 8:46 AM
gerritbot added a comment.Sep 16 2020, 9:30 AM

Change 627771 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/extensions/FileImporter@master] Fix two-hop interwiki prefix lookup reducing "foo.org" to "org"

https://gerrit.wikimedia.org/r/627771

gerritbot added a comment.Sep 17 2020, 8:38 AM

Change 627752 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Update outdated FileImporterInterWikiMap docs

https://gerrit.wikimedia.org/r/627752

thiemowmde mentioned this in rEFLI5ff05bd2b051: Update outdated FileImporterInterWikiMap docs.Sep 17 2020, 8:41 AM
gerritbot added a comment.Sep 17 2020, 9:00 AM

Change 627771 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Fix two-hop interwiki prefix lookup reducing "foo.org" to "org"

https://gerrit.wikimedia.org/r/627771

ReleaseTaggerBot added a project: MW-1.36-notes (1.36.0-wmf.10; 2020-09-22).Sep 17 2020, 9:00 AM
thiemowmde mentioned this in rEFLI8ed944eb1822: Fix two-hop interwiki prefix lookup reducing "foo.org" to "org".Sep 17 2020, 9:01 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL