⚓ T95448 Move bits traffic to text/mobile clusters

Subject	Repo	Branch	Lines +/-
bits-legacy: remove special https://bits redirects for secure wikis	operations/puppet	production	+0 -9
bits-legacy: remove beacon/statsv support	operations/puppet	production	+1 -1
Decom bits cluster varnish/lvs configuration	operations/puppet	production	+0 -350
Remove cache::bits role from bits-cluster hosts	operations/puppet	production	+8 -5
decom bits service IPs	operations/dns	master	+0 -24
bits.wm.o -> text-cluster	operations/dns	master	+1 -1
switch bits to meta refs in dataset/snapshot html	operations/puppet	production	+7 -7
Add legacy bits.wm.o support to text-lb VCL	operations/puppet	production	+63 -1
static URLs now seem to all be /static/, update regex	operations/puppet	production	+2 -2
Move media beacon off of bits	operations/mediawiki-config	master	+1 -1
Prepare for death of bits.wikimedia.org	apps/android/wikipedia	master	+4 -4

Status	Assigned	Task
Resolved	Dzahn	T94896 Optimize prod's resource domains for SPDY/HTTP2
Duplicate	None	T101339 Expand misc cluster into cache PoPs
Resolved	BBlack	T86718 Upgrade eqiad-misc varnish cluster from 2 to 4 systems.
Resolved	BBlack	T107430 Decom bits.wikimedia.org hostname
Resolved	BBlack	T95448 Move bits traffic to text/mobile clusters
Resolved	ori	T97750 wmgUseBits = false still loads favicon from bits.wikimedia.org
Resolved	ori	T106966 Mobile web wrongly uses desktop domain for load.php

BBlack created this task.Apr 8 2015, 5:58 PM

BBlack raised the priority of this task from to Medium.

BBlack updated the task description. (Show Details)

BBlack added projects: acl*sre-team, Varnish.

BBlack subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 8 2015, 5:58 PM

BBlack added a parent task: T94896: Optimize prod's resource domains for SPDY/HTTP2.Apr 8 2015, 5:58 PM

BBlack added a project: Traffic.Apr 22 2015, 2:44 PM

BBlack set Security to None.

MediaWiki's default expectation is that static assets and text are served from the same host. It actually takes quite a bit of hackery to convince it to use bits. So (happily) this change is straightforward to implement -- it requires not much more than the removal of code.

See https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/CommonSettings.php#L227-244

Note also ori's couple of related varnish patches here for a potential path towards various solutions:
https://gerrit.wikimedia.org/r/#/c/206351/
https://gerrit.wikimedia.org/r/#/c/206348/

faidon noted that there's some risk if we just naively rid ourselves of the bits hostname, it could have a detrimental impact on non-SPDY clients, who currently get to use some $domain-vs-bits.wm.o connection parallelism. We need to think that through, and if it's an issue, maybe there are ways we can still get the best of both worlds with e.g. same-IP + certs, etc....

BBlack mentioned this in T86718: Upgrade eqiad-misc varnish cluster from 2 to 4 systems..Apr 24 2015, 1:17 PM

BBlack removed a subtask: T95449: Purchase SSDs for legacy bits cache machines for re-use in other clusters.Apr 24 2015, 3:13 PM

BBlack added a parent task: T86718: Upgrade eqiad-misc varnish cluster from 2 to 4 systems..Apr 24 2015, 3:16 PM

To elaborate: I'm worried that of the repercussions this would have for non-SPDY clients, as they wouldn't be able to use a different set of connections to fetch those resources (i.e. traditional domain sharding). "Non-SPDY" clients right now are the vast majority of anons, as we're not HTTPS by default, so this isn't something to take lightly.

I don't think we reach the connection limits for anything but upload, but still, this probably needs some further investigation (if it hasn't happened already?) and measuring its effect to page load time post-deployment.

This also means that the crazy humongous cookies that we hand out users would be sent for bits resources as well, but considering the low amount of requests due to RL bundling this may not have a huge effect overall. Hopefully :)

I think @ori 's contention last we discussed this on IRC was that the current count of $domain + bits connections in a typical page load (combined) is under the threshold for common client-side connection parallelism limits to a single domainname. Thus we'd see the same opportunities for parallelism in the non-SPDY case regardless of whether the bits domain was separate or not.

BBlack moved this task from Backlog to Upcoming on the Traffic board.Apr 27 2015, 3:18 PM

https://gerrit.wikimedia.org/r/206878 <- vcl_hash support for this (optimization)

BBlack mentioned this in rOPUPf39a6912799a: de-dupe /static hashing for text/mobile.Apr 28 2015, 11:15 PM

BBlack moved this task from Upcoming to Traffic team actively servicing on the Traffic board.Apr 29 2015, 9:00 PM

BBlack updated the task description. (Show Details)Apr 29 2015, 9:04 PM

BBlack updated the task description. (Show Details)Apr 29 2015, 11:09 PM

BBlack updated the task description. (Show Details)

BBlack mentioned this in rOPUP7dda1feb0f5f: wikimedia.vcl: issue an HTTP 204 for all beacon/* requests.Apr 29 2015, 11:09 PM

looking at mediawiki-config repo, it's not immediately obvious to me how we'd test this on one wiki or anything like that. We could obviously gut the $wmfHostname['bits'] references globally in beta and roll out by groups, of course.

ori mentioned this in rOMWCf8d18fd6f959: Add 'wmgUseBits' config option; true by default..Apr 30 2015, 6:28 PM

In T95448#1249875, @BBlack wrote:

looking at mediawiki-config repo, it's not immediately obvious to me how we'd test this on one wiki or anything like that. We could obviously gut the $wmfHostname['bits'] references globally in beta and roll out by groups, of course.

Nah, it's easier than that. Done in I188bbb829 and applied to sewikibooks (a closed wiki) as a test-case.

This is live on mediawikiwiki now as well ( https://www.mediawiki.org/ )!

BBlack updated the task description. (Show Details)Apr 30 2015, 9:26 PM

Krinkle subscribed.May 2 2015, 12:25 AM

Krinkle added a subtask: T97750: wmgUseBits = false still loads favicon from bits.wikimedia.org.May 2 2015, 12:34 AM

ori closed subtask T97750: wmgUseBits = false still loads favicon from bits.wikimedia.org as Resolved.May 2 2015, 12:44 AM

Krenair subscribed.May 2 2015, 12:15 PM

• MZMcBride subscribed.May 5 2015, 1:53 AM

ori mentioned this in T98404: Update EventLogging URL on iOS and Android apps.May 6 2015, 8:29 PM

Change 208315 had a related patch set uploaded (by Yuvipanda):
Prepare for death of bits.wikimedia.org

https://gerrit.wikimedia.org/r/208315

gerritbot added a project: Patch-For-Review.May 6 2015, 8:37 PM

Change 208315 merged by jenkins-bot:
Prepare for death of bits.wikimedia.org

https://gerrit.wikimedia.org/r/208315

• bearND mentioned this in rAPAWb2d46b5316f1: Prepare for death of bits.wikimedia.org.May 12 2015, 11:22 PM

• Spage mentioned this in T99886: No /static/1.26wmf7 tree -- mediawiki.org missing "Powered by MediaWiki" logo and ResourceLoader ?debug=1 mode broken.May 21 2015, 11:16 AM

Change 214822 had a related patch set uploaded (by Ori.livneh):
Move media beacon off of bits

https://gerrit.wikimedia.org/r/214822

Change 214822 merged by Ori.livneh:
Move media beacon off of bits

https://gerrit.wikimedia.org/r/214822

ori mentioned this in rOMWC6eead1d954c5: Move media beacon off of bits.May 30 2015, 4:53 PM

Change 215624 had a related patch set uploaded (by BBlack):
Add legacy bits.wm.o support to text-lb VCL

https://gerrit.wikimedia.org/r/215624

BBlack added a parent task: T101339: Expand misc cluster into cache PoPs.Jun 4 2015, 12:05 AM

Change 219107 had a related patch set uploaded (by BBlack):
static URLs now seem to all be /static/, update regex

https://gerrit.wikimedia.org/r/219107

Change 219107 merged by BBlack:
static URLs now seem to all be /static/, update regex

https://gerrit.wikimedia.org/r/219107

BBlack mentioned this in rOPUPc1fd51aa3963: static URLs now seem to all be /static/, update regex.Jun 18 2015, 1:06 AM

• Elitre subscribed.Jun 23 2015, 2:15 PM

Should this be closed?

/w/static and wikiname/load.php URLs still work on bits, e.g. https://bits.wikimedia.org/static/1.26wmf12/resources/assets/poweredby_mediawiki_88x31.png , but I don't see anything generating them.

Don't forget to update https://wikitech.wikimedia.org/wiki/Bits.wikimedia.org.

Restricted Application added a subscriber: Matanya. · View Herald TranscriptJul 3 2015, 1:41 AM

Yeah, to the degree possible, we've left everything we can still working on bits during the transition. The next major step we're coming up on is physically folding bits into the text cluster (moving the DNS resolution and VCL code there), while still leaving it as logically-functional as it is today. At that point, we can begin planning for its eventual long-term decom/demise.

Krinkle mentioned this in T94896: Optimize prod's resource domains for SPDY/HTTP2.Jul 10 2015, 11:10 AM

fgiunchedi assigned this task to BBlack.Jul 20 2015, 10:50 AM

fgiunchedi subscribed.

Are we basically done with all of the bits.wm.o traffic removals we can accomplish quickly and easily? I'd like to merge the cluster over into text-lb at this point ( https://gerrit.wikimedia.org/r/#/c/215624/ + a followup DNS patch afterwards to move the hostname), so we can unblock the bits cluster decom/cleanup and misc-cluster plans.

BBlack added a parent task: T107430: Decom bits.wikimedia.org hostname.Jul 30 2015, 3:23 PM

BBlack updated the task description. (Show Details)

Change 215624 merged by BBlack:
Add legacy bits.wm.o support to text-lb VCL

https://gerrit.wikimedia.org/r/215624

BBlack updated the task description. (Show Details)Jul 30 2015, 3:34 PM

Change 228021 had a related patch set uploaded (by BBlack):
bits.wm.o -> text-cluster

https://gerrit.wikimedia.org/r/228021

Change 228029 had a related patch set uploaded (by BBlack):
decom bits service IPs

https://gerrit.wikimedia.org/r/228029

Change 228032 had a related patch set uploaded (by BBlack):
switch bits to meta refs in dataset/snapshot html

https://gerrit.wikimedia.org/r/228032

Change 228033 had a related patch set uploaded (by BBlack):
Remove cache::bits roles from bits-cluster hosts

https://gerrit.wikimedia.org/r/228033

Change 228034 had a related patch set uploaded (by BBlack):
Decom bits cluster varnish/lvs configuration

https://gerrit.wikimedia.org/r/228034

Change 228032 merged by BBlack:
switch bits to meta refs in dataset/snapshot html

https://gerrit.wikimedia.org/r/228032

We probably shouldn't move forward on the bits->text switch at the DNS level (and patches beyond that) until T106966 is resolved, so that we don't have zero-whitelisters seeing mobile loading load.php traffic from non-mobile.

ori closed subtask T106966: Mobile web wrongly uses desktop domain for load.php as Resolved.Jul 30 2015, 7:50 PM

BBlack mentioned this in rOPUP035a31826148: Add legacy bits.wm.o support to text-lb VCL.Jul 30 2015, 10:16 PM

BBlack mentioned this in rOPUP1023c858cf32: switch bits to meta refs in dataset/snapshot html.

Change 228021 merged by BBlack:
bits.wm.o -> text-cluster

https://gerrit.wikimedia.org/r/228021

BBlack mentioned this in rODNS6d72a82b98c8: bits.wm.o -> text-cluster.Aug 3 2015, 9:10 PM

Change 228029 merged by BBlack:
decom bits service IPs

https://gerrit.wikimedia.org/r/228029

BBlack mentioned this in rODNS36fd73f7f03a: decom bits service IPs.Aug 5 2015, 4:32 PM

Change 228033 merged by BBlack:
Remove cache::bits role from bits-cluster hosts

https://gerrit.wikimedia.org/r/228033

BBlack mentioned this in rOPUPc832641dcd46: Remove cache::bits role from bits-cluster hosts.Aug 6 2015, 1:32 PM