Page MenuHomePhabricator
Create Task
Maniphest T109286

Merge mobile cache into text cache
Closed, ResolvedPublic
Actions

Description

[See also T89177 and T102524 - may eventually merge those in as duplicates]

The driving factors here are:

  1. We want to align the two on various analytics and functional header info related to X-Forwarded-For, X-Forwarded-By, X-CS, X-CS2, X-Analytics, etc. That and X-Subdomain are really the key functional differences in the VCL code of the two today (the other is Cookie handling).
  2. One of the key reasons to leave mobile segregated in the past was its high potential fragmentation due to Vary-ing on X-CS for many different carriers in support of Zero. However, that was long ago addressed with the "X-CS: ON" -related work. Only a handful of special URLs now fully vary on a real carrier-id.
  3. Mobile is currently under-utilizing a 4-node cache cluster at each site, when its traffic could easily now be merged into the text cluster in terms of load and hot dataset size, etc. Reducing the count of distinct cache clusters with independent complex configurations is a big complexity/maintenance win at various levels, and helps to further reduce the required minimum machine counts at cache datacenters to support all traffic reliably.

Some of the key steps and/or issues to address here (many can be done as slow refactor work leading up to the bigger switches):

  • Align the current text and mobile VCL code better where they differ for trivial or non-existent reasons, or one (usually text!) is simply better-configured than the other.
  • get netmapper and analytics synced up between the clusters
  • Fix mobile cookie handling and merge code-wise with text cookie handling
  • Turn on the key mobile code in the text-caches with X-Subdomain regexes protecting exclusive parts in both directions (e.g. current text's mobile redirect code, and mobile's X-Subdomain code).
  • Ensure no mobile-vs-desktop cache pollution on the text cluster (fixed via Vary changes or vcl_hash)
  • Coordinate with Analytics on the shift of mobile request logs to the text data sources, make any necessary changes there to support
  • Move the mobile IPs to the text-cluster. The two would still differ on IP address, but would both come through the same nginx proxy and varnish cluster.

[... at this stage, we can decom the actual mobile cache cluster and reuse it for other purposes ...]

Details

Related Gerrit Patches:
operations/puppet : productiontorrus: remove cache_mobile stuff
operations/puppet : productioncache_mobile decom: 2/2 Remove most cache config
operations/puppet : productioncache_mobile LVS decom: 3/3 remove conftool service data
operations/puppet : productioncache_mobile LVS decom: 2/3 remove conftool node data
operations/puppet : productioncache_mobile decom: 1/2 remove realserver IPs
operations/puppet : productioncache_mobile LVS decom: 1/2 remove LVS service
operations/puppet : productioneqiad: remove last cache_mobile frontend
operations/puppet : productioneqiad: remove most mobile frontends from cache_mobile
operations/puppet : productioneqiad: add text nodes to mobile cluster
operations/puppet : productioneqiad: add text nodes to cache_mobile frontends
operations/puppet : productionesams: remove varnish-fe,nginx services from mobile cluster
operations/puppet : productionesams: add text nodes to mobile cluster
operations/puppet : productionulsfo: remove varnish-fe,nginx services from mobile cluster
operations/puppet : productionulsfo: add text nodes to mobile cluster
operations/puppet : productionmobile-lb: use text caches as LVS backends
operations/puppet : productioncodfw: remove varnish-fe,nginx services from mobile cluster
operations/puppet : productioncodfw: add text nodes to mobile cluster
operations/puppet : productioncache_text: add mobile IPs to loopback
operations/puppet : productiontext VCL: protect mobile cache from text pollution
operations/puppet : productionText VCL: same no-article-cache for mobile as desktop
operations/puppet : productiontext VCL: remove hiera mobile/text conditionals
operations/puppet : productionvarnish: use same VCL files for text+mobile
operations/puppet : productionMove all X-Analytics code to analytics.inc, include in common VCL
operations/puppet : productionX-Client-IP: get rid of temp var, update commentary
operations/puppet : productionX-Client-IP 12/12 - switch zero analytics to use XC/XCM
operations/puppet : productionX-Client-IP 11/12 - remove outdated 404-01b zero case
operations/puppet : productionX-Client-IP 10/12 - switch zero.inc to using XC + XCM
operations/puppet : productionX-Client-IP 9/12 - Set X-C + X-C-M
operations/puppet : productionX-Client-IP 8/12 - Set X-CIP
operations/puppet : productionX-Client-IP 7/12 - Set X-T-P
operations/puppet : productionX-Client-IP 6/12 - unset the 4x new headers
operations/puppet : productionX-Client-IP 5/12 - recv_fe_ip_proc frontend-only
operations/puppet : productionX-Client-IP 4/12 - move XFF-setter out of recv_fe_ip_processing
operations/puppet : productionX-Client-IP 3/12 - remove fe default on be guard
operations/puppet : productionX-Client-IP 2/12 - rename ip_proc sub, move req.restarts guard
operations/puppet : productionX-Client-IP 1/12 - just move netmapper import + init
operations/puppet : productionmove netmapper processing to common VCL
operations/puppet : productionMove all X-Analytics code to analytics.inc, include in common VCL
operations/puppet : productionmove netmapper processing to common VCL
operations/puppet : productionmove zerofetcher to r::c::base
operations/puppet : productionmobile: remove "Temp test" CC header block for icons/gadget
operations/puppet : productioncache_mobile: raise max conns to 1k like text
operations/puppet : productioncache_mobile: limit 4xx to 1m like text
operations/puppet : productionadd zero updater to text.pp, align position
operations/puppet : productiongenericize the cluster_nodes-like variables to reduce diffs
operations/puppet : productiontrivial mobile/text.pp diff reductions
operations/puppet : productionclean up text/mobile whitespace-only diffs
operations/puppet : productionmobile: add bits compat code
operations/puppet : productionstandardize hiera-overridable class/config params
operations/puppet : productionvcl: merge cluster_options into vcl_config, refactor
operations/puppet : productionAlign mobile VCL much closer to text VCL
operations/puppet : productionadd various text backend defs to mobile
operations/puppet : productionmobile vcl: tighten up disableImages cookie regex
operations/puppet : productionvcl_cookies: reduce text-vs-mobile cookie variance
operations/puppet : productionvcl_cookies: use common pass_auth in mobile
operations/puppet : productionvcl_cookies: re-arrange mobile recv order a bit to match text
operations/puppet : productionvcl_cookies: switch mobile to text-common code, mostly
operations/puppet : productionvcl_cookies: standardize on X-Orig-Cookie
operations/puppet : productionalign text+mobile on filter_(headers|noise) in shared code
operations/puppet : productionre-order range / purge to align text+mobile

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Oct 7 2015, 8:41 PM

Change 244204 merged by BBlack:
X-Client-IP 4/12 - move XFF-setter out of recv_fe_ip_processing

https://gerrit.wikimedia.org/r/244204

gerritbot added a comment.Oct 7 2015, 10:24 PM

Change 244205 merged by BBlack:
X-Client-IP 5/12 - recv_fe_ip_proc frontend-only

https://gerrit.wikimedia.org/r/244205

gerritbot added a comment.Oct 7 2015, 10:52 PM

Change 244206 merged by BBlack:
X-Client-IP 6/12 - unset the 4x new headers

https://gerrit.wikimedia.org/r/244206

gerritbot added a comment.Oct 7 2015, 11:11 PM

Change 244207 merged by BBlack:
X-Client-IP 7/12 - Set X-T-P

https://gerrit.wikimedia.org/r/244207

gerritbot added a comment.Oct 7 2015, 11:32 PM

Change 244208 merged by BBlack:
X-Client-IP 8/12 - Set X-CIP

https://gerrit.wikimedia.org/r/244208

gerritbot added a comment.Oct 8 2015, 2:02 AM

Change 244209 merged by BBlack:
X-Client-IP 9/12 - Set X-C X-C-M

https://gerrit.wikimedia.org/r/244209

gerritbot added a comment.Oct 8 2015, 2:53 AM

Change 244210 merged by BBlack:
X-Client-IP 10/12 - switch zero.inc to using XC XCM

https://gerrit.wikimedia.org/r/244210

gerritbot added a comment.Oct 8 2015, 3:02 AM

Change 244211 merged by BBlack:
X-Client-IP 11/12 - remove outdated 404-01b zero case

https://gerrit.wikimedia.org/r/244211

gerritbot added a comment.Oct 8 2015, 3:13 AM

Change 244212 merged by BBlack:
X-Client-IP 12/12 - switch zero analytics to use XC/XCM

https://gerrit.wikimedia.org/r/244212

gerritbot added a comment.Oct 8 2015, 1:26 PM

Change 244442 had a related patch set uploaded (by BBlack):
X-Client-IP: get rid of temp var, update commentary

https://gerrit.wikimedia.org/r/244442

gerritbot added a comment.Oct 8 2015, 1:29 PM

Change 244442 merged by BBlack:
X-Client-IP: get rid of temp var, update commentary

https://gerrit.wikimedia.org/r/244442

gerritbot added a comment.Oct 8 2015, 2:10 PM

Change 243977 merged by BBlack:
Move all X-Analytics code to analytics.inc, include in common VCL

https://gerrit.wikimedia.org/r/243977

BBlack updated the task description. (Show Details)Dec 3 2015, 12:34 AM
gerritbot added a comment.Dec 8 2015, 9:15 PM

Change 257699 had a related patch set uploaded (by BBlack):
varnish: use same VCL files for text mobile

https://gerrit.wikimedia.org/r/257699

gerritbot added a comment.Dec 8 2015, 11:27 PM

Change 257774 had a related patch set uploaded (by BBlack):
text VCL: remove hiera mobile/text conditionals

https://gerrit.wikimedia.org/r/257774

gerritbot added a comment.Dec 9 2015, 4:31 PM

Change 257699 merged by BBlack:
varnish: use same VCL files for text mobile

https://gerrit.wikimedia.org/r/257699

gerritbot added a comment.Dec 9 2015, 6:41 PM

Change 257774 merged by BBlack:
text VCL: remove hiera mobile/text conditionals

https://gerrit.wikimedia.org/r/257774

gerritbot added a comment.Dec 10 2015, 8:12 PM

Change 258208 had a related patch set uploaded (by BBlack):
Text VCL: same no-article-cache for mobile as desktop

https://gerrit.wikimedia.org/r/258208

gerritbot added a comment.Dec 10 2015, 8:19 PM

Change 258208 merged by BBlack:
Text VCL: same no-article-cache for mobile as desktop

https://gerrit.wikimedia.org/r/258208

BBlack added a subscriber: Ottomata.Dec 11 2015, 2:41 PM

I've successfully tested loading mobile content through the text cache (local DNS hack of m-dot hostname to text cluster), and used the same URLs through both and seen that they don't pollute each other. So in functional user-facing terms, I think we're ready to make the switch here. However, I think we'll need to coordinate with Analytics about how this affects their stats streams (e.g. what used to be requests in webrequest_mobile will now be mixed in with desktop in webrequest_text, differentiated by the m-dot request hostnames).

@Ottomata?

Ottomata added subscribers: JAllemandou, Nuria.Dec 11 2015, 3:06 PM

@JAllemandou, @Nuria ^ :)

(We'll talk about this at standup today)

gerritbot added a comment.Dec 11 2015, 3:16 PM

Change 258458 had a related patch set uploaded (by BBlack):
cache_text: add mobile IPs to loopback

https://gerrit.wikimedia.org/r/258458

gerritbot added a comment.Dec 11 2015, 3:16 PM

Change 258459 had a related patch set uploaded (by BBlack):
mobile-lb: use text caches as LVS backends

https://gerrit.wikimedia.org/r/258459

BBlack updated the task description. (Show Details)Dec 11 2015, 4:06 PM
gerritbot added a comment.Dec 12 2015, 12:07 AM

Change 258648 had a related patch set uploaded (by BBlack):
text VCL: protect mobile cache from text pollution

https://gerrit.wikimedia.org/r/258648

gerritbot added a comment.Dec 12 2015, 12:08 AM

Change 258648 merged by BBlack:
text VCL: protect mobile cache from text pollution

https://gerrit.wikimedia.org/r/258648

Ottomata closed subtask T122650: Disable legacy tsv mobile, zero and 5xx-mobile jobs as Resolved.Dec 30 2015, 3:42 PM

Ok, I think we are ready on the Analytics side. We'll need to do some things right after this change is made, so some planning is in order over in https://phabricator.wikimedia.org/T122651. Let's set a date for making this change, and we'll make sure we are ready to do our part.

BBlack added a comment.Jan 13 2016, 3:36 PM

The switch of traffic off of the mobile cluster is tentatively scheduled to begin on Tuesday, Jan 19th and take at least several hours. Will update here when it's complete.

gerritbot added a comment.Jan 19 2016, 6:23 PM

Change 258458 merged by Ema:
cache_text: add mobile IPs to loopback

https://gerrit.wikimedia.org/r/258458

BBlack added a comment.Jan 20 2016, 8:27 PM

The traffic move from mobile->text is now on hold (we did convert codfw, then we rolled back) due to purge-related issues that need to be addressed first, in blocking task T124165.

gerritbot added a comment.Jan 22 2016, 11:42 AM

Change 265710 had a related patch set uploaded (by Ema):
codfw: add text nodes to mobile cluster

https://gerrit.wikimedia.org/r/265710

ema moved this task from Up Next to In Progress on the Traffic board.Jan 22 2016, 11:47 AM
gerritbot added a comment.Jan 22 2016, 11:59 AM

Change 265710 merged by Ema:
codfw: add text nodes to mobile cluster

https://gerrit.wikimedia.org/r/265710

gerritbot added a comment.Jan 22 2016, 3:49 PM

Change 265742 had a related patch set uploaded (by Ema):
codfw: remove varnish-fe,nginx services from mobile cluster

https://gerrit.wikimedia.org/r/265742

ema added a subscriber: ema.Jan 22 2016, 3:58 PM
gerritbot added a comment.Jan 22 2016, 4:03 PM

Change 265742 merged by Ema:
codfw: remove varnish-fe,nginx services from mobile cluster

https://gerrit.wikimedia.org/r/265742

gerritbot added a comment.Jan 22 2016, 6:23 PM

Change 258459 abandoned by BBlack:
mobile-lb: use text caches as LVS backends

https://gerrit.wikimedia.org/r/258459

BBlack updated the task description. (Show Details)Jan 22 2016, 10:33 PM
gerritbot added a comment.Jan 25 2016, 3:11 PM

Change 266230 had a related patch set uploaded (by Ema):
ulsfo: add text nodes to mobile cluster

https://gerrit.wikimedia.org/r/266230

gerritbot added a comment.Jan 25 2016, 3:17 PM

Change 266230 merged by Ema:
ulsfo: add text nodes to mobile cluster

https://gerrit.wikimedia.org/r/266230

gerritbot added a comment.Jan 25 2016, 4:47 PM

Change 266253 had a related patch set uploaded (by Ema):
ulsfo: remove varnish-fe,nginx services from mobile cluster

https://gerrit.wikimedia.org/r/266253

gerritbot added a comment.Jan 25 2016, 5:02 PM

Change 266253 merged by Ema:
ulsfo: remove varnish-fe,nginx services from mobile cluster

https://gerrit.wikimedia.org/r/266253

gerritbot added a comment.Jan 26 2016, 9:01 AM

Change 266475 had a related patch set uploaded (by Ema):
esams: add text nodes to mobile cluster

https://gerrit.wikimedia.org/r/266475

gerritbot added a comment.Jan 26 2016, 10:46 AM

Change 266475 merged by Ema:
esams: add text nodes to mobile cluster

https://gerrit.wikimedia.org/r/266475

gerritbot added a comment.Jan 26 2016, 2:54 PM

Change 266499 had a related patch set uploaded (by Ema):
esams: remove varnish-fe,nginx services from mobile cluster

https://gerrit.wikimedia.org/r/266499

gerritbot added a comment.Jan 26 2016, 3:05 PM

Change 266499 merged by Ema:
esams: remove varnish-fe,nginx services from mobile cluster

https://gerrit.wikimedia.org/r/266499

gerritbot added a comment.Jan 26 2016, 3:40 PM

Change 266503 had a related patch set uploaded (by Ema):
eqiad: add text nodes to mobile cluster

https://gerrit.wikimedia.org/r/266503

gerritbot added a comment.Jan 28 2016, 10:18 PM

Change 267159 had a related patch set uploaded (by BBlack):
eqiad: add text nodes to cache_mobile frontends

https://gerrit.wikimedia.org/r/267159

gerritbot added a comment.Jan 28 2016, 10:19 PM

Change 267160 had a related patch set uploaded (by BBlack):
eqiad: remove mobile frontends from cache_mobile

https://gerrit.wikimedia.org/r/267160

gerritbot added a comment.Jan 28 2016, 10:20 PM

Change 267159 merged by BBlack:
eqiad: add text nodes to cache_mobile frontends

https://gerrit.wikimedia.org/r/267159

gerritbot added a comment.Jan 29 2016, 9:20 AM

Change 266503 abandoned by Ema:
eqiad: add text nodes to mobile cluster

Reason:
Already done in https://gerrit.wikimedia.org/r/#/c/267159/.

https://gerrit.wikimedia.org/r/266503

gerritbot added a comment.Jan 29 2016, 10:38 AM

Change 267230 had a related patch set uploaded (by BBlack):
eqiad: remove last cache_mobile frontend

https://gerrit.wikimedia.org/r/267230

gerritbot added a comment.Jan 29 2016, 10:41 AM

Change 267160 merged by BBlack:
eqiad: remove most mobile frontends from cache_mobile

https://gerrit.wikimedia.org/r/267160

BBlack added a comment.Jan 29 2016, 2:24 PM

Status update: We're pretty much done with the cache traffic migration, but there's still 1x eqiad mobile cache (cp1060) pooled with low weight to keep mobile webrequest analytics data flowing, until they're ready on their end to deal with it, which will probably be on Monday. After that we'll probably hold on any further related work for a week just in case latent issues or complaints crop up, so that it's easier to revert. Assuming no issues and comfort levels are high, then we can proceed with removing the puppet definitions for the cache_mobile cluster in LVS/varnish terms and moving the mobile IPs into cache_text's list of IPs.

Ottomata added a comment.Jan 29 2016, 2:35 PM

Ok great! We’re having some issues with jobs right now due to some Kafka
problems, and we’ll want to make sure everything is fine before we try to
move on this. Hopefully everything will be fine by Monday and we can
proceed.

Ottomata added a comment.Feb 1 2016, 3:37 PM

I just talked to @BBlack, and also looked at requests in the webrequest_mobile topic in Kafka. There are still real user requests from cp1060, but most of the requests in that topic are internal monitoring requests. Analytics jobs will not block as long as at least some data continues to flow into webrequest_mobile. I told @BBlack that they can go ahead and drain traffic from cp1060, and that we will proceed with Analytics changes after that is done, but before monitoring traffic is also turned off.

BBlack added a comment.Feb 1 2016, 4:55 PM

cp1060 is depooled for users now. Once Analytics is done with their oozie thing, we can proceed on the next steps for actually stopping the cache_mobile cluster itself (which is devoid of real users now).

gerritbot added a comment.Feb 3 2016, 4:47 PM

Change 267230 merged by BBlack:
eqiad: remove last cache_mobile frontend

https://gerrit.wikimedia.org/r/267230

gerritbot added a comment.Feb 3 2016, 9:35 PM

Change 268226 had a related patch set uploaded (by BBlack):
cache_mobile LVS decom: 1/2 remove LVS service

https://gerrit.wikimedia.org/r/268226

gerritbot added a comment.Feb 3 2016, 9:35 PM

Change 268227 had a related patch set uploaded (by BBlack):
cache_mobile LVS decom: 2/2 remove conftool data

https://gerrit.wikimedia.org/r/268227

gerritbot added a comment.Feb 3 2016, 9:35 PM

Change 268228 had a related patch set uploaded (by BBlack):
cache_mobile decom: 1/2 remove realserver IPs

https://gerrit.wikimedia.org/r/268228

gerritbot added a comment.Feb 3 2016, 9:35 PM

Change 268229 had a related patch set uploaded (by BBlack):
cache_mobile decom: 2/2 Remove most cache config

https://gerrit.wikimedia.org/r/268229

gerritbot added a comment.Feb 8 2016, 2:03 PM

Change 268226 merged by BBlack:
cache_mobile LVS decom: 1/2 remove LVS service

https://gerrit.wikimedia.org/r/268226

gerritbot added a comment.Feb 8 2016, 2:17 PM

Change 268228 merged by BBlack:
cache_mobile decom: 1/2 remove realserver IPs

https://gerrit.wikimedia.org/r/268228

gerritbot added a comment.Feb 8 2016, 2:39 PM

Change 269127 had a related patch set uploaded (by BBlack):
cache_mobile LVS decom: 3/3 remove conftool service data

https://gerrit.wikimedia.org/r/269127

gerritbot added a comment.Feb 8 2016, 2:41 PM

Change 268227 merged by BBlack:
cache_mobile LVS decom: 2/3 remove conftool node data

https://gerrit.wikimedia.org/r/268227

gerritbot added a comment.Feb 8 2016, 2:46 PM

Change 269127 merged by BBlack:
cache_mobile LVS decom: 3/3 remove conftool service data

https://gerrit.wikimedia.org/r/269127

gerritbot added a comment.Feb 8 2016, 3:10 PM

Change 268229 merged by BBlack:
cache_mobile decom: 2/2 Remove most cache config

https://gerrit.wikimedia.org/r/268229

BBlack closed this task as Resolved.Feb 8 2016, 3:27 PM
BBlack updated the task description. (Show Details)
gerritbot added a comment.Feb 8 2016, 3:31 PM

Change 269141 had a related patch set uploaded (by BBlack):
torrus: remove cache_mobile stuff

https://gerrit.wikimedia.org/r/269141

gerritbot added a comment.Feb 8 2016, 3:32 PM

Change 269141 merged by BBlack:
torrus: remove cache_mobile stuff

https://gerrit.wikimedia.org/r/269141

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL