Change #1161727 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[operations/puppet@production] Suppress mobile redirect for Googlebot Smartphone on Commons

https://gerrit.wikimedia.org/r/1161727

Change #1161727 merged by Tim Starling:

[operations/puppet@production] Suppress mobile redirect for Googlebot Smartphone on Commons

https://gerrit.wikimedia.org/r/1161727

Our webrequest metrics confirm that we're not sending redirects to Googlebot anymore. There was a spike in Googlebot's request rate to Commons a suspiciously short time after the deployment, but there have been spikes like that in the past so it could be coincidental.

Adding to the confounding factors, over the last two weeks, there was a prolonged increase in requests to Commons, and a corresponding recovery in the total indexed page count from 50.5M to 52.4M.

In a discussion with a representative from Google today, we had confirmation that this is probably a good idea.

We still don't have confirmation from the search console indexing report, but here's a longer term view of the inbound request rate from Turnilo. When the yellow line (redirects) went to zero as a result of this intervention, within minutes there was a spike in overall request rate. A lot of requests received a 304 (not modified) response, in pink, which was basically absent from the prior data. Google Search Console reports that 93% of requests are for "refresh" and 7% are for discovery. Previously we were responding to many of the refreshes with a 302, but now we're giving a lot of 304s. Hopefully the indexing report will confirm that this is preventing pages from being deindexed.

Google Search Console gives time series data for the refresh/discovery breakdown. This data attributes the recent spike entirely to refreshes, discovery requests are basically flat.

Change #1187876 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/puppet@production] varnish: Switch Commons/Googlebot pilot from desktop to unified mobile

https://gerrit.wikimedia.org/r/1187876

Change #1187875 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/puppet@production] varnish: Assert whether Commons/Googlebot gets desktop or mobile HTML

https://gerrit.wikimedia.org/r/1187875

Change #1187875 merged by BCornwall:

[operations/puppet@production] varnish: Assert whether Commons/Googlebot gets desktop or mobile HTML

https://gerrit.wikimedia.org/r/1187875

Change #1187876 merged by BCornwall:

[operations/puppet@production] varnish: Switch Commons/Googlebot pilot from desktop to unified mobile

https://gerrit.wikimedia.org/r/1187876

Googlebot is now (back) to indexing the mobile version of Wikimedia Commons.

During the experiment, we disabled the redirect and simply let commons.wikimedia.org respond in-place when Googlebot hits that domain, which meant the Googlebot Mobile crawler received the desktop site. That's harmless for indexing I suppose, but may bite us in the future and it'd be inconsistent long-term with other sites.

$ curl -si 'https://commons.wikimedia.org/wiki/Main_Page' -H 'User-Agent: Krinkle (Googlebot)' | grep -iE 'HTTP/|location|vary|mobileaction'
HTTP/2 200 
vary: Accept-Encoding,X-Subdomain,Cookie,Authorization
<div class="mw-collapsible-toggle" style="font-weight:bold; ; text-align:start;"><span class="tpl-hidden-arrowtoggle skin-invert"></span><span class="tpl-hidden-headertext">By location</span></div>
	<li id="footer-places-mobileview"><a href="//commons.m.wikimedia.org/w/index.php?title=Main_Page&amp;mobileaction=toggle_view_mobile" class="noprint stopMobileRedirectToggle">Mobile view</a></li>

$ curl -si 'https://commons.wikimedia.org/wiki/Main_Page' -H 'User-Agent: Krinkle Mobile (Googlebot)' | grep -iE 'HTTP/|location|vary|mobileaction'
HTTP/2 200 
vary: Accept-Encoding,X-Subdomain,Cookie,Authorization
<div class="mw-collapsible-toggle" style="font-weight:bold; ; text-align:start;"><span class="tpl-hidden-arrowtoggle skin-invert"></span><span class="tpl-hidden-headertext">By location</span></div>
	<li id="footer-places-mobileview"><a href="//commons.m.wikimedia.org/w/index.php?title=Main_Page&amp;mobileaction=toggle_view_mobile" class="noprint stopMobileRedirectToggle">Mobile view</a></li>

$ curl -si 'https://commons.wikimedia.org/wiki/Main_Page' -H 'User-Agent: Krinkle (Googlebot)' | grep -iE 'HTTP/|location|vary|mobileaction'
HTTP/2 200 
vary: Accept-Encoding,X-Subdomain,Cookie,Authorization
<div class="mw-collapsible-toggle" style="font-weight:bold; ; text-align:start;"><span class="tpl-hidden-arrowtoggle skin-invert"></span><span class="tpl-hidden-headertext">By location</span></div>
	<li id="footer-places-mobileview"><a href="//commons.m.wikimedia.org/w/index.php?title=Main_Page&amp;mobileaction=toggle_view_mobile" class="noprint stopMobileRedirectToggle">Mobile view</a></li>

$ curl -si 'https://commons.wikimedia.org/wiki/Main_Page' -H 'User-Agent: Krinkle Mobile (Googlebot)' | grep -iE 'HTTP/|location|vary|mobileaction'
HTTP/2 200 
vary: Accept-Encoding,X-Subdomain,Cookie,Authorization
<div class="mw-collapsible-toggle" style="font-weight:bold; ; text-align:start;"><span class="tpl-hidden-arrowtoggle skin-invert"></span><span class="tpl-hidden-headertext">By location</span></div>
	<li id="footer-places-desktop-toggle"><a id="mw-mf-display-toggle" href="//commons.wikimedia.org/w/index.php?title=Main_Page&amp;mobileaction=toggle_view_desktop" data-event-name="switch_to_desktop">Desktop</a></li>