Page MenuHomePhabricator

Refactor varnish puppet config
Closed, ResolvedPublic

Description

This is already a work-in-progress, with the related commits showing up here:

https://gerrit.wikimedia.org/r/#/q/project:operations/puppet+branch:production+topic:cache-cleanup,n,z

Since "refactor" is sort of endless and abstract as a task goal, here's some of the objectives we're going after here (may evolve as the factoring does, to some degree):

  • Adhere to our puppet style conventions as much as reaosonably possible
  • Eliminate $::realm and $::site conditional code through hieraization
  • De-duplicate role::cache::$foo code into base/1layer/2layer as appropriate
  • Create new module to sit between role::cache::$foo and the various software-specific modules like varnish, nginx, etc
  • Move WMF-specific VCL from modules/varnish + templates/varnish to new module
  • De-duplicate shared VCL between clusters (especially text + mobile), align current slight variations where appropriate

Details

Related Gerrit Patches:
operations/puppet : productionRevert "Revert "Revert "cache_text/mobile: send randomized pass traffic directly to t1 backends"""
operations/puppet : productionvarnish: appropriate t2-fe -> t1-be backend_options
operations/puppet : productionRevert "Revert "cache_text/mobile: send randomized pass traffic directly to t1 backends""
operations/puppet : productionRevert "cache_text/mobile: send randomized pass traffic directly to t1 backends"
operations/puppet : productioncache_text/mobile: send randomized pass traffic directly to t1 backends
operations/puppet : productionVCL: no special handling for CentralAutoLogin
operations/puppet : productioncache_upload: send randomized pass traffic directly to t1 backends
operations/puppet : productioncache_misc: send randomized pass traffic directly to t1 backends
operations/puppet : productionvarnish: return (pass) for CAL URLs
operations/puppet : productionvarnish: always use backend_random for pass/hfp
operations/puppet : productionadd backend_random to maps and upload clusters config
operations/puppet : productionadd backend_random to maps and upload clusters in conftool-data
operations/puppet : productioncache_upload: remove unused "rendering" backend
operations/puppet : productionvarnish: security_audit backend explicitly tier-one-only
operations/puppet : productionvarnish: cache /api/rest_v1/ in backends
operations/puppet : productionMove all X-Analytics code to analytics.inc, include in common VCL
operations/puppet : productionX-Client-IP: get rid of temp var, update commentary
operations/puppet : productionX-Client-IP 12/12 - switch zero analytics to use XC/XCM
operations/puppet : productionX-Client-IP 11/12 - remove outdated 404-01b zero case
operations/puppet : productionX-Client-IP 10/12 - switch zero.inc to using XC + XCM
operations/puppet : productionX-Client-IP 9/12 - Set X-C + X-C-M
operations/puppet : productionX-Client-IP 8/12 - Set X-CIP
operations/puppet : productionX-Client-IP 7/12 - Set X-T-P
operations/puppet : productionX-Client-IP 6/12 - unset the 4x new headers
operations/puppet : productionX-Client-IP 5/12 - recv_fe_ip_proc frontend-only
operations/puppet : productionX-Client-IP 4/12 - move XFF-setter out of recv_fe_ip_processing
operations/puppet : productionX-Client-IP 3/12 - remove fe default on be guard
operations/puppet : productionX-Client-IP 2/12 - rename ip_proc sub, move req.restarts guard
operations/puppet : productionX-Client-IP 1/12 - just move netmapper import + init
operations/puppet : productionmove netmapper processing to common VCL
operations/puppet : productionMove all X-Analytics code to analytics.inc, include in common VCL
operations/puppet : productionmove netmapper processing to common VCL
operations/puppet : productionvarnish: standardize/de-duplicate do_gzip
operations/puppet : productiongenericize the cluster_nodes-like variables to reduce diffs
operations/puppet : productionstandardize hiera-overridable class/config params
operations/puppet : productionvcl: merge cluster_options into vcl_config, refactor

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

This is highly-interrelated with T109286 as well, but I'm not sure that either really blocks the other. They'll probably happen in concert slowly over time.

Change 235145 had a related patch set uploaded (by BBlack):
vcl: merge cluster_options into vcl_config, refactor

https://gerrit.wikimedia.org/r/235145

Change 235146 had a related patch set uploaded (by BBlack):
standardize hiera-overridable class/config params

https://gerrit.wikimedia.org/r/235146

Change 235145 merged by BBlack:
vcl: merge cluster_options into vcl_config, refactor

https://gerrit.wikimedia.org/r/235145

Change 235146 merged by BBlack:
standardize hiera-overridable class/config params

https://gerrit.wikimedia.org/r/235146

Change 235189 had a related patch set uploaded (by BBlack):
genericize the cluster_nodes-like variables to reduce diffs

https://gerrit.wikimedia.org/r/235189

Change 235189 merged by BBlack:
genericize the cluster_nodes-like variables to reduce diffs

https://gerrit.wikimedia.org/r/235189

jcrespo set Security to None.

Change 237366 had a related patch set uploaded (by BBlack):
varnish: standardize/de-duplicate do_gzip

https://gerrit.wikimedia.org/r/237366

Change 237366 merged by BBlack:
varnish: standardize/de-duplicate do_gzip

https://gerrit.wikimedia.org/r/237366

Change 243395 had a related patch set uploaded (by BBlack):
move netmapper processing to common VCL

https://gerrit.wikimedia.org/r/243395

Change 243406 had a related patch set uploaded (by BBlack):
Move all X-Analytics code to common VCL

https://gerrit.wikimedia.org/r/243406

Change 243395 merged by BBlack:
move netmapper processing to common VCL

https://gerrit.wikimedia.org/r/243395

Change 243406 merged by BBlack:
Move all X-Analytics code to analytics.inc, include in common VCL

https://gerrit.wikimedia.org/r/243406

Change 243976 had a related patch set uploaded (by BBlack):
move netmapper processing to common VCL

https://gerrit.wikimedia.org/r/243976

Change 243977 had a related patch set uploaded (by BBlack):
Move all X-Analytics code to analytics.inc, include in common VCL

https://gerrit.wikimedia.org/r/243977

Change 243976 merged by BBlack:
move netmapper processing to common VCL

https://gerrit.wikimedia.org/r/243976

Change 244201 had a related patch set uploaded (by BBlack):
X-Client-IP 1/12 - just move netmapper import init

https://gerrit.wikimedia.org/r/244201

Change 244202 had a related patch set uploaded (by BBlack):
X-Client-IP 2/12 - rename ip_proc sub, move req.restarts guard

https://gerrit.wikimedia.org/r/244202

Change 244203 had a related patch set uploaded (by BBlack):
X-Client-IP 3/12 - remove fe default on be guard

https://gerrit.wikimedia.org/r/244203

Change 244204 had a related patch set uploaded (by BBlack):
X-Client-IP 4/12 - move XFF-setter out of recv_fe_ip_processing

https://gerrit.wikimedia.org/r/244204

Change 244205 had a related patch set uploaded (by BBlack):
X-Client-IP 5/12 - recv_fe_ip_proc frontend-only

https://gerrit.wikimedia.org/r/244205

Change 244206 had a related patch set uploaded (by BBlack):
X-Client-IP 6/12 - unset the 4x new headers

https://gerrit.wikimedia.org/r/244206

Change 244207 had a related patch set uploaded (by BBlack):
X-Client-IP 7/12 - Set X-T-P

https://gerrit.wikimedia.org/r/244207

Change 244208 had a related patch set uploaded (by BBlack):
X-Client-IP 8/12 - Set X-CIP

https://gerrit.wikimedia.org/r/244208

Change 244209 had a related patch set uploaded (by BBlack):
X-Client-IP 9/12 - Set X-C X-C-M

https://gerrit.wikimedia.org/r/244209

Change 244210 had a related patch set uploaded (by BBlack):
X-Client-IP 10/12 - switch zero.inc to using XC XCM

https://gerrit.wikimedia.org/r/244210

Change 244211 had a related patch set uploaded (by BBlack):
X-Client-IP 11/12 - remove outdated 404-01b zero case

https://gerrit.wikimedia.org/r/244211

Change 244212 had a related patch set uploaded (by BBlack):
X-Client-IP 12/12 - switch zero analytics to use XC/XCM

https://gerrit.wikimedia.org/r/244212

Change 244201 merged by BBlack:
X-Client-IP 1/12 - just move netmapper import init

https://gerrit.wikimedia.org/r/244201

Change 244202 merged by BBlack:
X-Client-IP 2/12 - rename ip_proc sub, move req.restarts guard

https://gerrit.wikimedia.org/r/244202

Change 244203 merged by BBlack:
X-Client-IP 3/12 - remove fe default on be guard

https://gerrit.wikimedia.org/r/244203

Change 244204 merged by BBlack:
X-Client-IP 4/12 - move XFF-setter out of recv_fe_ip_processing

https://gerrit.wikimedia.org/r/244204

Change 244205 merged by BBlack:
X-Client-IP 5/12 - recv_fe_ip_proc frontend-only

https://gerrit.wikimedia.org/r/244205

Change 244206 merged by BBlack:
X-Client-IP 6/12 - unset the 4x new headers

https://gerrit.wikimedia.org/r/244206

Change 244207 merged by BBlack:
X-Client-IP 7/12 - Set X-T-P

https://gerrit.wikimedia.org/r/244207

Change 244208 merged by BBlack:
X-Client-IP 8/12 - Set X-CIP

https://gerrit.wikimedia.org/r/244208

Change 244209 merged by BBlack:
X-Client-IP 9/12 - Set X-C X-C-M

https://gerrit.wikimedia.org/r/244209

Change 244210 merged by BBlack:
X-Client-IP 10/12 - switch zero.inc to using XC XCM

https://gerrit.wikimedia.org/r/244210

Change 244211 merged by BBlack:
X-Client-IP 11/12 - remove outdated 404-01b zero case

https://gerrit.wikimedia.org/r/244211

Change 244212 merged by BBlack:
X-Client-IP 12/12 - switch zero analytics to use XC/XCM

https://gerrit.wikimedia.org/r/244212

Change 244442 had a related patch set uploaded (by BBlack):
X-Client-IP: get rid of temp var, update commentary

https://gerrit.wikimedia.org/r/244442

Change 244442 merged by BBlack:
X-Client-IP: get rid of temp var, update commentary

https://gerrit.wikimedia.org/r/244442

Change 243977 merged by BBlack:
Move all X-Analytics code to analytics.inc, include in common VCL

https://gerrit.wikimedia.org/r/243977

Change 257630 had a related patch set uploaded (by BBlack):
varnish: cache /api/rest_v1/ in backends

https://gerrit.wikimedia.org/r/257630

Change 257631 had a related patch set uploaded (by BBlack):
varnish: security_audit backend explicitly tier-one-only

https://gerrit.wikimedia.org/r/257631

Change 257632 had a related patch set uploaded (by BBlack):
varnish: return (pass) for CAL URLs

https://gerrit.wikimedia.org/r/257632

Change 257633 had a related patch set uploaded (by BBlack):
cache_upload: remove unused "rendering" backend

https://gerrit.wikimedia.org/r/257633

Change 257634 had a related patch set uploaded (by BBlack):
add backend_random to maps and upload clusters in conftool-data

https://gerrit.wikimedia.org/r/257634

Change 257635 had a related patch set uploaded (by BBlack):
add backend_random to maps and upload clusters config

https://gerrit.wikimedia.org/r/257635

Change 257636 had a related patch set uploaded (by BBlack):
varnish: always use backend_random for pass/hfp

https://gerrit.wikimedia.org/r/257636

Change 257630 merged by BBlack:
varnish: cache /api/rest_v1/ in backends

https://gerrit.wikimedia.org/r/257630

Change 257631 merged by BBlack:
varnish: security_audit backend explicitly tier-one-only

https://gerrit.wikimedia.org/r/257631

Change 257633 merged by BBlack:
cache_upload: remove unused "rendering" backend

https://gerrit.wikimedia.org/r/257633

Change 257634 merged by BBlack:
add backend_random to maps and upload clusters in conftool-data

https://gerrit.wikimedia.org/r/257634

Change 257635 merged by BBlack:
add backend_random to maps and upload clusters config

https://gerrit.wikimedia.org/r/257635

Change 257636 merged by BBlack:
varnish: always use backend_random for pass/hfp

https://gerrit.wikimedia.org/r/257636

Change 258207 had a related patch set uploaded (by BBlack):
VCL: no special handling for CentralAutoLogin

https://gerrit.wikimedia.org/r/258207

Change 257632 abandoned by BBlack:
varnish: return (pass) for CAL URLs

https://gerrit.wikimedia.org/r/257632

Change 258463 had a related patch set uploaded (by BBlack):
cache_misc: send randomized pass traffic directly to t1 backends

https://gerrit.wikimedia.org/r/258463

Change 258464 had a related patch set uploaded (by BBlack):
cache_upload: send randomized pass traffic directly to t1 backends

https://gerrit.wikimedia.org/r/258464

Change 258465 had a related patch set uploaded (by BBlack):
cache_text/mobile: send randomized pass traffic directly to t1 backends

https://gerrit.wikimedia.org/r/258465

Change 258463 merged by BBlack:
cache_misc: send randomized pass traffic directly to t1 backends

https://gerrit.wikimedia.org/r/258463

Change 258464 merged by BBlack:
cache_upload: send randomized pass traffic directly to t1 backends

https://gerrit.wikimedia.org/r/258464

Change 258207 merged by BBlack:
VCL: no special handling for CentralAutoLogin

https://gerrit.wikimedia.org/r/258207

Change 258465 merged by BBlack:
cache_text/mobile: send randomized pass traffic directly to t1 backends

https://gerrit.wikimedia.org/r/258465

Change 259518 had a related patch set uploaded (by BBlack):
Revert "cache_text/mobile: send randomized pass traffic directly to t1 backends"

https://gerrit.wikimedia.org/r/259518

Change 259518 merged by BBlack:
Revert "cache_text/mobile: send randomized pass traffic directly to t1 backends"

https://gerrit.wikimedia.org/r/259518

Change 259800 had a related patch set uploaded (by BBlack):
Revert "Revert "cache_text/mobile: send randomized pass traffic directly to t1 backends""

https://gerrit.wikimedia.org/r/259800

Change 259800 merged by BBlack:
Revert "Revert "cache_text/mobile: send randomized pass traffic directly to t1 backends""

https://gerrit.wikimedia.org/r/259800

Change 260048 had a related patch set uploaded (by BBlack):
varnish: appropriate t2-fe -> t1-be backend_options

https://gerrit.wikimedia.org/r/260048

Change 260048 merged by BBlack:
varnish: appropriate t2-fe -> t1-be backend_options

https://gerrit.wikimedia.org/r/260048

Change 260204 had a related patch set uploaded (by BBlack):
Revert "Revert "Revert "cache_text/mobile: send randomized pass traffic directly to t1 backends"""

https://gerrit.wikimedia.org/r/260204

Change 260204 merged by BBlack:
Revert "Revert "Revert "cache_text/mobile: send randomized pass traffic directly to t1 backends"""

https://gerrit.wikimedia.org/r/260204

BBlack closed this task as Resolved.Feb 19 2016, 4:50 PM
BBlack claimed this task.

In spite of good intentions, of course any "refactor complicated thing X" ticket just drags on forever as a pointless moving target. Closing this because it's not really useful as a task.