Page MenuHomePhabricator

Refactor varnish puppet config
Closed, ResolvedPublic

Description

This is already a work-in-progress, with the related commits showing up here:

https://gerrit.wikimedia.org/r/#/q/project:operations/puppet+branch:production+topic:cache-cleanup,n,z

Since "refactor" is sort of endless and abstract as a task goal, here's some of the objectives we're going after here (may evolve as the factoring does, to some degree):

  • Adhere to our puppet style conventions as much as reaosonably possible
  • Eliminate $::realm and $::site conditional code through hieraization
  • De-duplicate role::cache::$foo code into base/1layer/2layer as appropriate
  • Create new module to sit between role::cache::$foo and the various software-specific modules like varnish, nginx, etc
  • Move WMF-specific VCL from modules/varnish + templates/varnish to new module
  • De-duplicate shared VCL between clusters (especially text + mobile), align current slight variations where appropriate

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+2 -4
operations/puppetproduction+136 -51
operations/puppetproduction+4 -2
operations/puppetproduction+2 -4
operations/puppetproduction+4 -2
operations/puppetproduction+0 -12
operations/puppetproduction+2 -1
operations/puppetproduction+1 -1
operations/puppetproduction+24 -22
operations/puppetproduction+43 -34
operations/puppetproduction+18 -0
operations/puppetproduction+62 -45
operations/puppetproduction+0 -5
operations/puppetproduction+2 -4
operations/puppetproduction+4 -22
operations/puppetproduction+218 -246
operations/puppetproduction+11 -12
operations/puppetproduction+16 -39
operations/puppetproduction+0 -5
operations/puppetproduction+16 -54
operations/puppetproduction+29 -0
operations/puppetproduction+26 -2
operations/puppetproduction+35 -8
operations/puppetproduction+7 -0
operations/puppetproduction+5 -0
operations/puppetproduction+7 -7
operations/puppetproduction+1 -1
operations/puppetproduction+29 -28
operations/puppetproduction+14 -8
operations/puppetproduction+167 -145
operations/puppetproduction+211 -243
operations/puppetproduction+167 -145
operations/puppetproduction+7 -20
operations/puppetproduction+22 -22
operations/puppetproduction+49 -50
operations/puppetproduction+91 -91
Show related patches Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

This is highly-interrelated with T109286 as well, but I'm not sure that either really blocks the other. They'll probably happen in concert slowly over time.

Change 235145 had a related patch set uploaded (by BBlack):
vcl: merge cluster_options into vcl_config, refactor

https://gerrit.wikimedia.org/r/235145

Change 235146 had a related patch set uploaded (by BBlack):
standardize hiera-overridable class/config params

https://gerrit.wikimedia.org/r/235146

Change 235145 merged by BBlack:
vcl: merge cluster_options into vcl_config, refactor

https://gerrit.wikimedia.org/r/235145

Change 235146 merged by BBlack:
standardize hiera-overridable class/config params

https://gerrit.wikimedia.org/r/235146

Change 235189 had a related patch set uploaded (by BBlack):
genericize the cluster_nodes-like variables to reduce diffs

https://gerrit.wikimedia.org/r/235189

Change 235189 merged by BBlack:
genericize the cluster_nodes-like variables to reduce diffs

https://gerrit.wikimedia.org/r/235189

jcrespo set Security to None.

Change 237366 had a related patch set uploaded (by BBlack):
varnish: standardize/de-duplicate do_gzip

https://gerrit.wikimedia.org/r/237366

Change 237366 merged by BBlack:
varnish: standardize/de-duplicate do_gzip

https://gerrit.wikimedia.org/r/237366

Change 243395 had a related patch set uploaded (by BBlack):
move netmapper processing to common VCL

https://gerrit.wikimedia.org/r/243395

Change 243406 had a related patch set uploaded (by BBlack):
Move all X-Analytics code to common VCL

https://gerrit.wikimedia.org/r/243406

Change 243395 merged by BBlack:
move netmapper processing to common VCL

https://gerrit.wikimedia.org/r/243395

Change 243406 merged by BBlack:
Move all X-Analytics code to analytics.inc, include in common VCL

https://gerrit.wikimedia.org/r/243406

Change 243976 had a related patch set uploaded (by BBlack):
move netmapper processing to common VCL

https://gerrit.wikimedia.org/r/243976

Change 243977 had a related patch set uploaded (by BBlack):
Move all X-Analytics code to analytics.inc, include in common VCL

https://gerrit.wikimedia.org/r/243977

Change 243976 merged by BBlack:
move netmapper processing to common VCL

https://gerrit.wikimedia.org/r/243976

Change 244201 had a related patch set uploaded (by BBlack):
X-Client-IP 1/12 - just move netmapper import init

https://gerrit.wikimedia.org/r/244201

Change 244202 had a related patch set uploaded (by BBlack):
X-Client-IP 2/12 - rename ip_proc sub, move req.restarts guard

https://gerrit.wikimedia.org/r/244202

Change 244203 had a related patch set uploaded (by BBlack):
X-Client-IP 3/12 - remove fe default on be guard

https://gerrit.wikimedia.org/r/244203

Change 244204 had a related patch set uploaded (by BBlack):
X-Client-IP 4/12 - move XFF-setter out of recv_fe_ip_processing

https://gerrit.wikimedia.org/r/244204

Change 244205 had a related patch set uploaded (by BBlack):
X-Client-IP 5/12 - recv_fe_ip_proc frontend-only

https://gerrit.wikimedia.org/r/244205

Change 244206 had a related patch set uploaded (by BBlack):
X-Client-IP 6/12 - unset the 4x new headers

https://gerrit.wikimedia.org/r/244206

Change 244207 had a related patch set uploaded (by BBlack):
X-Client-IP 7/12 - Set X-T-P

https://gerrit.wikimedia.org/r/244207

Change 244208 had a related patch set uploaded (by BBlack):
X-Client-IP 8/12 - Set X-CIP

https://gerrit.wikimedia.org/r/244208

Change 244209 had a related patch set uploaded (by BBlack):
X-Client-IP 9/12 - Set X-C X-C-M

https://gerrit.wikimedia.org/r/244209

Change 244210 had a related patch set uploaded (by BBlack):
X-Client-IP 10/12 - switch zero.inc to using XC XCM

https://gerrit.wikimedia.org/r/244210

Change 244211 had a related patch set uploaded (by BBlack):
X-Client-IP 11/12 - remove outdated 404-01b zero case

https://gerrit.wikimedia.org/r/244211

Change 244212 had a related patch set uploaded (by BBlack):
X-Client-IP 12/12 - switch zero analytics to use XC/XCM

https://gerrit.wikimedia.org/r/244212

Change 244201 merged by BBlack:
X-Client-IP 1/12 - just move netmapper import init

https://gerrit.wikimedia.org/r/244201

Change 244202 merged by BBlack:
X-Client-IP 2/12 - rename ip_proc sub, move req.restarts guard

https://gerrit.wikimedia.org/r/244202

Change 244203 merged by BBlack:
X-Client-IP 3/12 - remove fe default on be guard

https://gerrit.wikimedia.org/r/244203

Change 244204 merged by BBlack:
X-Client-IP 4/12 - move XFF-setter out of recv_fe_ip_processing

https://gerrit.wikimedia.org/r/244204

Change 244205 merged by BBlack:
X-Client-IP 5/12 - recv_fe_ip_proc frontend-only

https://gerrit.wikimedia.org/r/244205

Change 244206 merged by BBlack:
X-Client-IP 6/12 - unset the 4x new headers

https://gerrit.wikimedia.org/r/244206

Change 244207 merged by BBlack:
X-Client-IP 7/12 - Set X-T-P

https://gerrit.wikimedia.org/r/244207

Change 244208 merged by BBlack:
X-Client-IP 8/12 - Set X-CIP

https://gerrit.wikimedia.org/r/244208

Change 244209 merged by BBlack:
X-Client-IP 9/12 - Set X-C X-C-M

https://gerrit.wikimedia.org/r/244209

Change 244210 merged by BBlack:
X-Client-IP 10/12 - switch zero.inc to using XC XCM

https://gerrit.wikimedia.org/r/244210

Change 244211 merged by BBlack:
X-Client-IP 11/12 - remove outdated 404-01b zero case

https://gerrit.wikimedia.org/r/244211

Change 244212 merged by BBlack:
X-Client-IP 12/12 - switch zero analytics to use XC/XCM

https://gerrit.wikimedia.org/r/244212

Change 244442 had a related patch set uploaded (by BBlack):
X-Client-IP: get rid of temp var, update commentary

https://gerrit.wikimedia.org/r/244442

Change 244442 merged by BBlack:
X-Client-IP: get rid of temp var, update commentary

https://gerrit.wikimedia.org/r/244442

Change 243977 merged by BBlack:
Move all X-Analytics code to analytics.inc, include in common VCL

https://gerrit.wikimedia.org/r/243977

Change 257630 had a related patch set uploaded (by BBlack):
varnish: cache /api/rest_v1/ in backends

https://gerrit.wikimedia.org/r/257630

Change 257631 had a related patch set uploaded (by BBlack):
varnish: security_audit backend explicitly tier-one-only

https://gerrit.wikimedia.org/r/257631

Change 257632 had a related patch set uploaded (by BBlack):
varnish: return (pass) for CAL URLs

https://gerrit.wikimedia.org/r/257632

Change 257633 had a related patch set uploaded (by BBlack):
cache_upload: remove unused "rendering" backend

https://gerrit.wikimedia.org/r/257633

Change 257634 had a related patch set uploaded (by BBlack):
add backend_random to maps and upload clusters in conftool-data

https://gerrit.wikimedia.org/r/257634

Change 257635 had a related patch set uploaded (by BBlack):
add backend_random to maps and upload clusters config

https://gerrit.wikimedia.org/r/257635

Change 257636 had a related patch set uploaded (by BBlack):
varnish: always use backend_random for pass/hfp

https://gerrit.wikimedia.org/r/257636

Change 257630 merged by BBlack:
varnish: cache /api/rest_v1/ in backends

https://gerrit.wikimedia.org/r/257630

Change 257631 merged by BBlack:
varnish: security_audit backend explicitly tier-one-only

https://gerrit.wikimedia.org/r/257631

Change 257633 merged by BBlack:
cache_upload: remove unused "rendering" backend

https://gerrit.wikimedia.org/r/257633

Change 257634 merged by BBlack:
add backend_random to maps and upload clusters in conftool-data

https://gerrit.wikimedia.org/r/257634

Change 257635 merged by BBlack:
add backend_random to maps and upload clusters config

https://gerrit.wikimedia.org/r/257635

Change 257636 merged by BBlack:
varnish: always use backend_random for pass/hfp

https://gerrit.wikimedia.org/r/257636

Change 258207 had a related patch set uploaded (by BBlack):
VCL: no special handling for CentralAutoLogin

https://gerrit.wikimedia.org/r/258207

Change 257632 abandoned by BBlack:
varnish: return (pass) for CAL URLs

https://gerrit.wikimedia.org/r/257632

Change 258463 had a related patch set uploaded (by BBlack):
cache_misc: send randomized pass traffic directly to t1 backends

https://gerrit.wikimedia.org/r/258463

Change 258464 had a related patch set uploaded (by BBlack):
cache_upload: send randomized pass traffic directly to t1 backends

https://gerrit.wikimedia.org/r/258464

Change 258465 had a related patch set uploaded (by BBlack):
cache_text/mobile: send randomized pass traffic directly to t1 backends

https://gerrit.wikimedia.org/r/258465

Change 258463 merged by BBlack:
cache_misc: send randomized pass traffic directly to t1 backends

https://gerrit.wikimedia.org/r/258463

Change 258464 merged by BBlack:
cache_upload: send randomized pass traffic directly to t1 backends

https://gerrit.wikimedia.org/r/258464

Change 258207 merged by BBlack:
VCL: no special handling for CentralAutoLogin

https://gerrit.wikimedia.org/r/258207

Change 258465 merged by BBlack:
cache_text/mobile: send randomized pass traffic directly to t1 backends

https://gerrit.wikimedia.org/r/258465

Change 259518 had a related patch set uploaded (by BBlack):
Revert "cache_text/mobile: send randomized pass traffic directly to t1 backends"

https://gerrit.wikimedia.org/r/259518

Change 259518 merged by BBlack:
Revert "cache_text/mobile: send randomized pass traffic directly to t1 backends"

https://gerrit.wikimedia.org/r/259518

Change 259800 had a related patch set uploaded (by BBlack):
Revert "Revert "cache_text/mobile: send randomized pass traffic directly to t1 backends""

https://gerrit.wikimedia.org/r/259800

Change 259800 merged by BBlack:
Revert "Revert "cache_text/mobile: send randomized pass traffic directly to t1 backends""

https://gerrit.wikimedia.org/r/259800

Change 260048 had a related patch set uploaded (by BBlack):
varnish: appropriate t2-fe -> t1-be backend_options

https://gerrit.wikimedia.org/r/260048

Change 260048 merged by BBlack:
varnish: appropriate t2-fe -> t1-be backend_options

https://gerrit.wikimedia.org/r/260048

Change 260204 had a related patch set uploaded (by BBlack):
Revert "Revert "Revert "cache_text/mobile: send randomized pass traffic directly to t1 backends"""

https://gerrit.wikimedia.org/r/260204

Change 260204 merged by BBlack:
Revert "Revert "Revert "cache_text/mobile: send randomized pass traffic directly to t1 backends"""

https://gerrit.wikimedia.org/r/260204

BBlack closed this task as Resolved.Feb 19 2016, 4:50 PM
BBlack claimed this task.

In spite of good intentions, of course any "refactor complicated thing X" ticket just drags on forever as a pointless moving target. Closing this because it's not really useful as a task.