I was looking through the new index T400023, and I noticed while we did some work with regard to T250317 in the past for JSON+LD metadata, that the license data we add does not seem to be recognized by Google. We should probably see if that can be fixed, because that just makes it a waste of bytes.

[edit] I was mistaken, this was due to some licenses that Google doesn't recognize.

I updated the task description to add the most recent statistics from Google.

Turnilo: pageviews_daily (type: user, referer_name: Google, project: commons) from April to October. - https://w.wiki/FoFG

Notable dates:

• April 1 - July 4: Stable pattern of 300-400K Google-referred page views per day.
• June 23: We disable mobile redirect for Googlebot on Commons.
• July 5: Rapid increase in Commons pageviews referred from Google, continues to increase for 11 weeks in a row until it finds a new high on Sep 20 at ~800K/day.
• August 6: We submit sitemap for Commons.

Same, but from 2023 to 2025 (this dataset starts in April 2023):

On our end, we see an +100% increase from 2.5M to 5M weekly pageviews. This matches the Google Console data in shape, but not in absolute values, as it reports a similar doubling but from 1.0M to 1.8M weekly "clicks".

Zooming out even further via the projectview_hourly table in Hadoop. (This is more crude because this dataset has referer_class "search engine", and not referer_name "Google". But that should be close enough for this purpose given it makes up 90% of that slice.)

SELECT CONCAT(year,'-',LPAD(month,2,'0')) _date, SUM(view_count) _count FROM wmf.projectview_hourly WHERE year > 0 AND project='commons.wikimedia' AND agent_type='user' AND referer_class='external (search engine)' GROUP BY year, month ORDER BY _date ASC;

Turnilo: pageviews_daily (type: spider, browser_family: Googlebot, project: commons) from 2020 to 2025. - https://w.wiki/FoGo

Beware of caveats with this.

• This is an aggregate based on Googlebot UA, not Google IP/ASN (we don't keep beyond 90 days), so this is subject to impersonations. Afaik we block such impersonations before they are counted, but that logic may very well have changed since 2020.
• It isn't a given that the crawl raw should be constant, because activity on Commons isn't constant.
• It isn't a given that Google Search (only) crawls pages from pageview URLs with a Googlebot UA. It also uses the REST API.

I expected a drop somewhere in 2023 or 2024, and then a recovery after our intervention. But, that's not what we see. I guess that makes sense, because the issue Googlebot certainly kept trying, but it didn't like what it found (mobile redirect with canonical pointing in the opposite direction).

Notable dates:

• June 23: Disabled mobile redirect for Googlebot on Commons.
• July 14: Raised rate limit for Googlebot.
• August 6: Submitted sitemap for Commons.

So in short.... visits to Commons originating from Google have almost doubled due to these interventions ? That is amazing news !

For posterity, capturing the equivalent "after" shots for Commons here.

In T54647#10902206, @tstarling wrote on 10 Jun 2025:

In T54647#10897742, @tstarling wrote:

46% of Googlebot requests use the smartphone User-Agent, and these mostly get a 302 redirect to commons.m.wikimedia.org. But the mobile site gives a link rel=canonical pointing back to commons.wikimedia.org, i.e. the redirect it just crawled.

Now that I have search console access to the mobile site, I can confirm that Google refuses to index the mobile site for this reason.

In T400022 task description, wrote on 20 July 2025: