MediaWiki is hitting redis for session data multiple times in a single page request.
Here's how I'm seeing this. I picked a codfw application server, since they are not getting any traffic, except PyBal health-checks, which hit Special:BlankPage on enwiki. I then run tcpdump looking for traffic with a destination port of 6379:
curl 'http://localhost/wiki/S%C3%A3o_Paulo' -s \ -H 'host: pt.wikipedia.org' \ -H 'X-Forwarded-Proto: https' \ -H 'WMF-Last-Access=12-Feb-2016; optin=beta; centralauth_LoggedOut=XXX; GeoIP=US:CA:San_Francisco:37.79:-122.39:v4; forceHTTPS=true; centralauth_User=Ori+Livneh; centralauth_Token=XXX; centralauth_Session=XXX'
(where XXX are valid values copied from my browser, which I have redacted here.)
I see the following:
$ sudo tcpdump -i eth0 -s 65535 -A -ttt port 6379 | grep -B2 'ptwiki:' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes GET $49 ptwiki:MWSession:fg6...(redacted)...k44 -- GET $49 ptwiki:MWSession:fg6...(redacted)...k44 -- GET $49 ptwiki:MWSession:fg6...(redacted)...k44 -- GET $49 ptwiki:MWSession:fg6...(redacted)...k44 -- GET $49 ptwiki:MWSession:fg6...(redacted)...k44 -- SETEX $49 ptwiki:MWSession:fg6...(redacted)...k44 -- SETEX $49 ptwiki:MWSession:fg6...(redacted)...k44 -- SETEX $49 ptwiki:MWSession:fg6...(redacted)...k44 -- SETEX $49 ptwiki:MWSession:fg6...(redacted)...k44
If I issue the request again, I see another 9 redis ops, but the key is different:
-- GET $49 ptwiki:MWSession:jvf...(redacted)...1hj -- GET $49 ptwiki:MWSession:jvf...(redacted)...1hj -- GET $49 ptwiki:MWSession:jvf...(redacted)...1hj -- GET $49 ptwiki:MWSession:jvf...(redacted)...1hj -- GET $49 ptwiki:MWSession:jvf...(redacted)...1hj -- SETEX $49 ptwiki:MWSession:jvf...(redacted)...1hj -- SETEX $49 ptwiki:MWSession:jvf...(redacted)...1hj -- SETEX $49 ptwiki:MWSession:jvf...(redacted)...1hj -- SETEX $49 ptwiki:MWSession:jvf...(redacted)...1hj
The key changes on every request.
You can see that this is related to the recent auth work by looking at the ops/sec graphs for the session redis hosts on Ganglia: