Looking more closely, it happened started between 18:00 and 20:00:

https://wikitech.wikimedia.org/w/index.php?title=Server_Admin_Log&oldid=172775#2015-07-30

18:02 logmsgbot: twentyafterfour rebuilt wikiversions.cdb and synchronized wikiversions files: all wikis to 1.26wmf16
18:33 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I6665bf31: Use relative URLs to construct load.php requests (duration: 00m 12s)
18:49 logmsgbot: ori Synchronized wmf-config/CommonSettings.php: I6db1771bf4: Use absolute URLs to construct load.php requests (duration: 00m 12s)
...
19:48 logmsgbot: ori Synchronized wmf-config: I0990ac5b: Update URL configuration for mobile when entering mobile mode (duration: 00m 12s)

The 1.26wmf16 switch is a red-herring. The actual cause is the change to mobile load.php urls. Confirmed by the fact that the graph went up, then down again, and finally kept going up. This follows @ori's commits that switched back and forth between mobile and desktop urls, eventually sticking with mobile (for mobile context). These commits address T106966. Mobile was wrongly using the desktop domain for load.php up until 2015-07-30. (See also T112646; which fixed another instance of this.)

This change should've been an improvement (in terms of DNS, SPDY, and more). But made things worse.

With help of @Catrope, I narrowed it down to request headers being malformed for mobile responses.

Desktop:

https://en.wikipedia.org/w/load.php?debug=false&lang=en&modules=startup&only=scripts&skin=vector
cache-control:public, max-age=300, s-maxage=300
https://en.wikipedia.org/w/load.php?debug=false&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=vector&version=c54hDTto
cache-control:public, max-age=2592000, s-maxage=2592000

Mobile:

https://en.m.wikipedia.org/w/load.php?debug=false&lang=en&modules=startup&only=scripts&skin=minerva&target=mobile
s-maxage=300, must-revalidate, max-age=0
https://en.m.wikipedia.org/w/load.php?debug=false&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=minerva&version=c54hDTto
s-maxage=300, must-revalidate, max-age=0

Caused by varnish/mobile-frontend.inc.vcl.erb#L166-L169:

if (resp.http.Cache-Control ~ "s-maxage=[1-9]") {
	set resp.http.Cache-Control = "s-maxage=300, must-revalidate, max-age=0";
} else {
	set resp.http.Cache-Control = "private, s-maxage=0, max-age=0, must-revalidate";
}

Introduced in 2011 with rOPUPe1e13ed3d696: Merge all existing Varnish3 VCL files into one new master file, with include… by @mark.

So mobile performance got worse because up until now mobile users were enjoying load.php requests from other domains that don't have bad Varnish configuration overrides (first bits.wikimedia.org, and later desktop domains; both are fine; only mobile has this override).

This is probably just the equivalent of T102898: ResourceLoader Cache-Control headers are being eaten in production for the mobile VCLs.

I tried to see if I could simply apply https://gerrit.wikimedia.org/r/#/c/219101/7/templates/varnish/text-frontend.inc.vcl.erb to mobile-frontend-inc.vcl.erb , but the logic for mobile is different:

if (resp.http.Cache-Control ~ "s-maxage=[1-9]") {
        set resp.http.Cache-Control = "s-maxage=300, must-revalidate, max-age=0";
} else {
        set resp.http.Cache-Control = "private, s-maxage=0, max-age=0, must-revalidate";
}

This (or any other CC mangling) should not be applied to load.php requests, but I'm not fluent enough in VCL to make that happen.

Change 239826 had a related patch set uploaded (by Krinkle):
varnish: Don't disable Cache-Control for all mobile traffic

https://gerrit.wikimedia.org/r/239826

Change 239826 merged by Ori.livneh:
varnish: Don't disable Cache-Control for all mobile traffic

https://gerrit.wikimedia.org/r/239826

This appears to be resolved now? The percentage of long cache headers in 304 responses went back down to 1-2% yesterday.