Page MenuHomePhabricator

Local file pages are getting indexed by Google (duplicate content?)
Open, Needs TriagePublic

Description

Problem

I have a Google alert for my name and for the past 30 days I've slowly receiving hundreds of alerts for essentially the same Commons file page. A few examples:

This despite the fact that:

  • These do not exist as pages on the local wiki.
  • Are not in the sitemap (and can't be, because they don't exist as pages).
  • Have <link rel="canonical" href="https://commons.wikimedia.org/wiki/File:Timo_Tijhof_at_Wikimania_2015_Reception_cropped.jpg">

Background

You can take any filepage on Commons and view it through a local domain. This will essentially proxy the File description from Commons, although translated to the local language.

Unlike most pages on our wikis, File pages are actually nearly completely translated in the content. These local pages of course have the skin localised, but the content area is fully translated as well:

  • Wikitext templates for File information (source, date, author) and are template-generated into the right language.
  • Wikitext for File description, caption, and license select the appropiate translation internally and show that one.
  • File metadata about the camera (EXIF) and File upload history are generated by MediaWiki core, but appended to the content area as part of the File page content.

As such, there is very little overlap between the translations and thus unlikely to be detected or recognised by Google as duplicate content.

Possible cause

I'm guessing that these are discovered by Google through a different mechanism that also shows central content in local context, namely GlobalUserPage.

https://meta.wikimedia.org/wiki/User:Timo_Tijhof_(WMF)

And these in turn link to the local proxy of the File page.

Although for files that are not my profile picture, the same can happen organically. Once any photo from Commons is used in a local article, that article will link to the local File page.

Possible solution

  1. Make the Parser link to Commons instead of the local proxy page, when the file is stored on Commons. (See: T421188)

Upside:

  • These duplicates fall out of Google, because they don't really exist other than when you link to them. They are generated/displayed on-demand, much akin to a search query or Special page. There is no record of them.

Neutral:

  • MediaViewer is enabled on WMF wikis by default, and in practice when you click on an image, it opens MediaViewer, instead of this link to local File page. Changing the link thus should have minimal impact on where anyone goes when they click it.
  • Inside MediaViewer, there is a "More information" link that goes the File page. That link already points to the Commons version.
  • There is a popular gadget, Direct_imagelinks_to_Commons, which changes these links with JavaScript from local links to Commons links. It mainly exists so that editors can get to Commons with one click instead of two clicks. Today it requires two clicks, both with MediaViewer enabled (1: click image to open MMV, 2: click More infomration), and with MediaViewer disabled (

(1: click image to open local File page, 2: click View on Commons). With this gadget, those with MMV disabled simply click the image to go to Commons, and even those with MMV enabled can get there direclty by opening the link in a new tab (thus bypassing the MMV JS click handler).

Downside:

  • For most readers, the local file page is going to be in their language (local wiki content language, their default user interface language), whereas Commons will be in English.
    • On the other hand, Commons has my gadget AnonymousI18N.js enabled by default, which looks at your referrer (i.e. nl.wikipedia.org) and asks you in a localised manner to reload the page "in <X> language". For example, in private browsing mode, open https://nl.wikipedia.org/wiki/Bestand:Example.jpg and click "Bekijken op Wikimedia Commons" and notice the blue banner on top asking " Wikimedia Commons in het Nederlands".
  • The local File page is the only way to quickly see where this image is used on their wiki. On Commons, they would need to query GlobalUsage and then paginate a few hundred times to their wiki.
  • For third-party wikis, the local File description page is cached and available within their network/intranet.
  • For third-party wikis, the local File description page is the only way to find local file usage through normal navigation (i.e. not manually querying Special:WhatLinksHere). This won't be available on Commons, not even if we build a better UI or allow for a hundred paginations, because Commons/GlobalUsage only stores usage information about WMF's own wikis.

Event Timeline

Why is Google not listening to the canonical tag though ?
Possibly because it's cross domain ?

We could simply noindex all such pages… might hurt overall indexing however, when the thumbs link to all those noindexed pages.

Yeah, noindexing virtual pages is appealing.

  • virtual File pages (i.e. no real page exists locally, it inherits on-demand from a foreign FileRepo like Commons)
  • virtual User page (GlobalUserPage)
  • virtual MediaWiki-namespace pages (i.e. default interface message content).

None of these exist in the page database and are not discoverable via sitemaps, site search, dumps, Special:AllPages, etc. because they don't exist. They just materialize on-demand from a user interaction or link if you ask for them, through some inheritance chain.

The concern I have is indeed with ranking.

What we want:

  • Index file pages only once (the Commons version).
  • Wikipedia articles add weight to those directly through the weight and relevant of these articles.

What we have:

  • Index Commons files pages
  • Index local file pages.
  • Wikipedia articles give some scattered weight to their local File pages, which in turn all give some of their weight to the Commons version.

It is not clear what will happen if Wikipedia articles keep pointing to local File pages which are no-indexed (weight is lost?). In theory follow,noindex is a thing, but I don't know how heavy that counts. It might be as bad as today, or it might be worse.

I'll ask Google in the next partnership meeting whether cross-origin canonical can/should work. That might let the current situation work as-is.

Note that historically we did succesfully use rel=canonical from en.m.wikipedia to en.wikipedia, which is also cross domain. But, it might be that it is limited within a registered wildcard domain.

Also note that, if the pattern observed in this task is new, then that would imply this hostirically also worked from Wikipedia to Commons, and perhaps only needs bugfix rather than a new feature.