Page MenuHomePhabricator
Create Task
Maniphest T96847

Refactor varnish puppet config
Closed, ResolvedPublic
Actions

Description

This is already a work-in-progress, with the related commits showing up here:

https://gerrit.wikimedia.org/r/#/q/project:operations/puppet+branch:production+topic:cache-cleanup,n,z

Since "refactor" is sort of endless and abstract as a task goal, here's some of the objectives we're going after here (may evolve as the factoring does, to some degree):

  • Adhere to our puppet style conventions as much as reaosonably possible
  • Eliminate $::realm and $::site conditional code through hieraization
  • De-duplicate role::cache::$foo code into base/1layer/2layer as appropriate
  • Create new module to sit between role::cache::$foo and the various software-specific modules like varnish, nginx, etc
  • Move WMF-specific VCL from modules/varnish + templates/varnish to new module
  • De-duplicate shared VCL between clusters (especially text + mobile), align current slight variations where appropriate

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Revert "Revert "Revert "cache_text/mobile: send randomized pass traffic directly to t1 backends"""operations/puppetproduction+2 -4
varnish: appropriate t2-fe -> t1-be backend_optionsoperations/puppetproduction+136 -51
Revert "Revert "cache_text/mobile: send randomized pass traffic directly to t1 backends""operations/puppetproduction+4 -2
Revert "cache_text/mobile: send randomized pass traffic directly to t1 backends"operations/puppetproduction+2 -4
cache_text/mobile: send randomized pass traffic directly to t1 backendsoperations/puppetproduction+4 -2
VCL: no special handling for CentralAutoLoginoperations/puppetproduction+0 -12
cache_upload: send randomized pass traffic directly to t1 backendsoperations/puppetproduction+2 -1
cache_misc: send randomized pass traffic directly to t1 backendsoperations/puppetproduction+1 -1
varnish: return (pass) for CAL URLsoperations/puppetproduction+24 -22
varnish: always use backend_random for pass/hfpoperations/puppetproduction+43 -34
add backend_random to maps and upload clusters configoperations/puppetproduction+18 -0
add backend_random to maps and upload clusters in conftool-dataoperations/puppetproduction+62 -45
cache_upload: remove unused "rendering" backendoperations/puppetproduction+0 -5
varnish: security_audit backend explicitly tier-one-onlyoperations/puppetproduction+2 -4
varnish: cache /api/rest_v1/ in backendsoperations/puppetproduction+4 -22
Move all X-Analytics code to analytics.inc, include in common VCLoperations/puppetproduction+218 -246
X-Client-IP: get rid of temp var, update commentaryoperations/puppetproduction+11 -12
X-Client-IP 12/12 - switch zero analytics to use XC/XCMoperations/puppetproduction+16 -39
X-Client-IP 11/12 - remove outdated 404-01b zero caseoperations/puppetproduction+0 -5
X-Client-IP 10/12 - switch zero.inc to using XC + XCMoperations/puppetproduction+16 -54
X-Client-IP 9/12 - Set X-C + X-C-Moperations/puppetproduction+29 -0
X-Client-IP 8/12 - Set X-CIPoperations/puppetproduction+26 -2
X-Client-IP 7/12 - Set X-T-Poperations/puppetproduction+35 -8
X-Client-IP 6/12 - unset the 4x new headersoperations/puppetproduction+7 -0
X-Client-IP 5/12 - recv_fe_ip_proc frontend-onlyoperations/puppetproduction+5 -0
X-Client-IP 4/12 - move XFF-setter out of recv_fe_ip_processingoperations/puppetproduction+7 -7
X-Client-IP 3/12 - remove fe default on be guardoperations/puppetproduction+1 -1
X-Client-IP 2/12 - rename ip_proc sub, move req.restarts guardoperations/puppetproduction+29 -28
X-Client-IP 1/12 - just move netmapper import + initoperations/puppetproduction+14 -8
move netmapper processing to common VCLoperations/puppetproduction+167 -145
Move all X-Analytics code to analytics.inc, include in common VCLoperations/puppetproduction+211 -243
move netmapper processing to common VCLoperations/puppetproduction+167 -145
varnish: standardize/de-duplicate do_gzipoperations/puppetproduction+7 -20
genericize the cluster_nodes-like variables to reduce diffsoperations/puppetproduction+22 -22
standardize hiera-overridable class/config paramsoperations/puppetproduction+49 -50
vcl: merge cluster_options into vcl_config, refactoroperations/puppetproduction+91 -91
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
BBlack added a comment.Aug 17 2015, 1:46 PM

This is highly-interrelated with T109286 as well, but I'm not sure that either really blocks the other. They'll probably happen in concert slowly over time.

gerritbot subscribed.Aug 31 2015, 11:16 PM

Change 235145 had a related patch set uploaded (by BBlack):
vcl: merge cluster_options into vcl_config, refactor

https://gerrit.wikimedia.org/r/235145

gerritbot added a project: Patch-For-Review.Aug 31 2015, 11:16 PM

Change 235146 had a related patch set uploaded (by BBlack):
standardize hiera-overridable class/config params

https://gerrit.wikimedia.org/r/235146

gerritbot added a comment.Sep 1 2015, 4:32 AM

Change 235145 merged by BBlack:
vcl: merge cluster_options into vcl_config, refactor

https://gerrit.wikimedia.org/r/235145

BBlack mentioned this in rOPUP4d4a71745417: vcl: merge cluster_options into vcl_config, refactor.Sep 1 2015, 4:32 AM
gerritbot added a comment.Sep 1 2015, 4:39 AM

Change 235146 merged by BBlack:
standardize hiera-overridable class/config params

https://gerrit.wikimedia.org/r/235146

BBlack mentioned this in rOPUPc6806db07036: standardize hiera-overridable class/config params.Sep 1 2015, 4:40 AM
gerritbot added a comment.Sep 1 2015, 6:14 AM

Change 235189 had a related patch set uploaded (by BBlack):
genericize the cluster_nodes-like variables to reduce diffs

https://gerrit.wikimedia.org/r/235189

gerritbot added a comment.Sep 1 2015, 6:43 AM

Change 235189 merged by BBlack:
genericize the cluster_nodes-like variables to reduce diffs

https://gerrit.wikimedia.org/r/235189

BBlack mentioned this in rOPUP30d44564164e: genericize the cluster_nodes-like variables to reduce diffs.Sep 1 2015, 6:44 AM
jcrespo removed a project: Patch-For-Review.Sep 8 2015, 10:19 AM
jcrespo set Security to None.
gerritbot added a comment.Sep 10 2015, 1:16 PM

Change 237366 had a related patch set uploaded (by BBlack):
varnish: standardize/de-duplicate do_gzip

https://gerrit.wikimedia.org/r/237366

gerritbot added a project: Patch-For-Review.Sep 10 2015, 1:16 PM
gerritbot added a comment.Sep 10 2015, 7:47 PM

Change 237366 merged by BBlack:
varnish: standardize/de-duplicate do_gzip

https://gerrit.wikimedia.org/r/237366

BBlack mentioned this in rOPUPfeb122488b7c: varnish: standardize/de-duplicate do_gzip.Sep 10 2015, 7:47 PM
gerritbot added a comment.Oct 3 2015, 1:56 PM

Change 243395 had a related patch set uploaded (by BBlack):
move netmapper processing to common VCL

https://gerrit.wikimedia.org/r/243395

gerritbot added a comment.Oct 3 2015, 4:10 PM

Change 243406 had a related patch set uploaded (by BBlack):
Move all X-Analytics code to common VCL

https://gerrit.wikimedia.org/r/243406

gerritbot added a comment.Oct 6 2015, 5:45 AM

Change 243395 merged by BBlack:
move netmapper processing to common VCL

https://gerrit.wikimedia.org/r/243395

BBlack mentioned this in rOPUPf092ee39eb89: move netmapper processing to common VCL.Oct 6 2015, 5:45 AM
gerritbot added a comment.Oct 6 2015, 6:12 AM

Change 243406 merged by BBlack:
Move all X-Analytics code to analytics.inc, include in common VCL

https://gerrit.wikimedia.org/r/243406

BBlack mentioned this in rOPUPd200332d2a8d: Move all X-Analytics code to analytics.inc, include in common VCL.Oct 6 2015, 6:13 AM
gerritbot added a comment.Oct 6 2015, 6:13 PM

Change 243976 had a related patch set uploaded (by BBlack):
move netmapper processing to common VCL

https://gerrit.wikimedia.org/r/243976

gerritbot added a comment.Oct 6 2015, 6:13 PM

Change 243977 had a related patch set uploaded (by BBlack):
Move all X-Analytics code to analytics.inc, include in common VCL

https://gerrit.wikimedia.org/r/243977

gerritbot added a comment.Oct 6 2015, 8:58 PM

Change 243976 merged by BBlack:
move netmapper processing to common VCL

https://gerrit.wikimedia.org/r/243976

BBlack mentioned this in rOPUP20a1de72bc96: move netmapper processing to common VCL.Oct 6 2015, 8:59 PM
gerritbot added a comment.Oct 7 2015, 5:31 PM

Change 244201 had a related patch set uploaded (by BBlack):
X-Client-IP 1/12 - just move netmapper import init

https://gerrit.wikimedia.org/r/244201

gerritbot added a comment.Oct 7 2015, 5:31 PM

Change 244202 had a related patch set uploaded (by BBlack):
X-Client-IP 2/12 - rename ip_proc sub, move req.restarts guard

https://gerrit.wikimedia.org/r/244202

gerritbot added a comment.Oct 7 2015, 5:31 PM

Change 244203 had a related patch set uploaded (by BBlack):
X-Client-IP 3/12 - remove fe default on be guard

https://gerrit.wikimedia.org/r/244203

gerritbot added a comment.Oct 7 2015, 5:31 PM

Change 244204 had a related patch set uploaded (by BBlack):
X-Client-IP 4/12 - move XFF-setter out of recv_fe_ip_processing

https://gerrit.wikimedia.org/r/244204

gerritbot added a comment.Oct 7 2015, 5:31 PM

Change 244205 had a related patch set uploaded (by BBlack):
X-Client-IP 5/12 - recv_fe_ip_proc frontend-only

https://gerrit.wikimedia.org/r/244205

gerritbot added a comment.Oct 7 2015, 5:31 PM

Change 244206 had a related patch set uploaded (by BBlack):
X-Client-IP 6/12 - unset the 4x new headers

https://gerrit.wikimedia.org/r/244206

gerritbot added a comment.Oct 7 2015, 5:31 PM

Change 244207 had a related patch set uploaded (by BBlack):
X-Client-IP 7/12 - Set X-T-P

https://gerrit.wikimedia.org/r/244207

gerritbot added a comment.Oct 7 2015, 5:31 PM

Change 244208 had a related patch set uploaded (by BBlack):
X-Client-IP 8/12 - Set X-CIP

https://gerrit.wikimedia.org/r/244208

gerritbot added a comment.Oct 7 2015, 5:31 PM

Change 244209 had a related patch set uploaded (by BBlack):
X-Client-IP 9/12 - Set X-C X-C-M

https://gerrit.wikimedia.org/r/244209

gerritbot added a comment.Oct 7 2015, 5:31 PM

Change 244210 had a related patch set uploaded (by BBlack):
X-Client-IP 10/12 - switch zero.inc to using XC XCM

https://gerrit.wikimedia.org/r/244210

gerritbot added a comment.Oct 7 2015, 5:31 PM

Change 244211 had a related patch set uploaded (by BBlack):
X-Client-IP 11/12 - remove outdated 404-01b zero case

https://gerrit.wikimedia.org/r/244211

gerritbot added a comment.Oct 7 2015, 5:31 PM

Change 244212 had a related patch set uploaded (by BBlack):
X-Client-IP 12/12 - switch zero analytics to use XC/XCM

https://gerrit.wikimedia.org/r/244212

gerritbot added a comment.Oct 7 2015, 7:43 PM

Change 244201 merged by BBlack:
X-Client-IP 1/12 - just move netmapper import init

https://gerrit.wikimedia.org/r/244201

BBlack mentioned this in rOPUP633dcb401388: X-Client-IP 1/12 - just move netmapper import + init.Oct 7 2015, 7:43 PM
gerritbot added a comment.Oct 7 2015, 8:24 PM

Change 244202 merged by BBlack:
X-Client-IP 2/12 - rename ip_proc sub, move req.restarts guard

https://gerrit.wikimedia.org/r/244202

BBlack mentioned this in rOPUP61855e421164: X-Client-IP 2/12 - rename ip_proc sub, move req.restarts guard.Oct 7 2015, 8:25 PM
gerritbot added a comment.Oct 7 2015, 8:40 PM

Change 244203 merged by BBlack:
X-Client-IP 3/12 - remove fe default on be guard

https://gerrit.wikimedia.org/r/244203

BBlack mentioned this in rOPUPfcfce07a2aca: X-Client-IP 3/12 - remove fe default on be guard.Oct 7 2015, 8:40 PM
gerritbot added a comment.Oct 7 2015, 8:41 PM

Change 244204 merged by BBlack:
X-Client-IP 4/12 - move XFF-setter out of recv_fe_ip_processing

https://gerrit.wikimedia.org/r/244204

BBlack mentioned this in rOPUP2982dcd22caf: X-Client-IP 4/12 - move XFF-setter out of recv_fe_ip_processing.Oct 7 2015, 8:41 PM
gerritbot added a comment.Oct 7 2015, 10:24 PM

Change 244205 merged by BBlack:
X-Client-IP 5/12 - recv_fe_ip_proc frontend-only

https://gerrit.wikimedia.org/r/244205

BBlack mentioned this in rOPUP238a5cf4bf1b: X-Client-IP 5/12 - recv_fe_ip_proc frontend-only.Oct 7 2015, 10:25 PM
gerritbot added a comment.Oct 7 2015, 10:52 PM

Change 244206 merged by BBlack:
X-Client-IP 6/12 - unset the 4x new headers

https://gerrit.wikimedia.org/r/244206

BBlack mentioned this in rOPUPfe47ba5e1089: X-Client-IP 6/12 - unset the 4x new headers.Oct 7 2015, 10:52 PM
gerritbot added a comment.Oct 7 2015, 11:11 PM

Change 244207 merged by BBlack:
X-Client-IP 7/12 - Set X-T-P

https://gerrit.wikimedia.org/r/244207

BBlack mentioned this in rOPUPc29ebea7b6ee: X-Client-IP 7/12 - Set X-T-P.Oct 7 2015, 11:12 PM
gerritbot added a comment.Oct 7 2015, 11:32 PM

Change 244208 merged by BBlack:
X-Client-IP 8/12 - Set X-CIP

https://gerrit.wikimedia.org/r/244208

BBlack mentioned this in rOPUP1198ceee51a2: X-Client-IP 8/12 - Set X-CIP.Oct 7 2015, 11:33 PM
gerritbot added a comment.Oct 8 2015, 2:02 AM

Change 244209 merged by BBlack:
X-Client-IP 9/12 - Set X-C X-C-M

https://gerrit.wikimedia.org/r/244209

gerritbot added a comment.Oct 8 2015, 2:53 AM

Change 244210 merged by BBlack:
X-Client-IP 10/12 - switch zero.inc to using XC XCM

https://gerrit.wikimedia.org/r/244210

gerritbot added a comment.Oct 8 2015, 3:02 AM

Change 244211 merged by BBlack:
X-Client-IP 11/12 - remove outdated 404-01b zero case

https://gerrit.wikimedia.org/r/244211

gerritbot added a comment.Oct 8 2015, 3:13 AM

Change 244212 merged by BBlack:
X-Client-IP 12/12 - switch zero analytics to use XC/XCM

https://gerrit.wikimedia.org/r/244212

gerritbot added a comment.Oct 8 2015, 1:26 PM

Change 244442 had a related patch set uploaded (by BBlack):
X-Client-IP: get rid of temp var, update commentary

https://gerrit.wikimedia.org/r/244442

gerritbot added a comment.Oct 8 2015, 1:29 PM

Change 244442 merged by BBlack:
X-Client-IP: get rid of temp var, update commentary

https://gerrit.wikimedia.org/r/244442

gerritbot added a comment.Oct 8 2015, 2:10 PM

Change 243977 merged by BBlack:
Move all X-Analytics code to analytics.inc, include in common VCL

https://gerrit.wikimedia.org/r/243977

BBlack added a subtask: T119394: Convert misc cluster to 2-layer.Nov 23 2015, 3:25 PM
BBlack added a subtask: T119396: Create globally-unique varnish cache cluster port/instancename mappings.Nov 23 2015, 3:32 PM
BBlack closed subtask T119394: Convert misc cluster to 2-layer as Resolved.Nov 25 2015, 6:33 PM
gerritbot added a comment.Dec 8 2015, 4:33 PM

Change 257630 had a related patch set uploaded (by BBlack):
varnish: cache /api/rest_v1/ in backends

https://gerrit.wikimedia.org/r/257630

gerritbot added a comment.Dec 8 2015, 4:33 PM

Change 257631 had a related patch set uploaded (by BBlack):
varnish: security_audit backend explicitly tier-one-only

https://gerrit.wikimedia.org/r/257631

gerritbot added a comment.Dec 8 2015, 4:33 PM

Change 257632 had a related patch set uploaded (by BBlack):
varnish: return (pass) for CAL URLs

https://gerrit.wikimedia.org/r/257632

gerritbot added a comment.Dec 8 2015, 4:33 PM

Change 257633 had a related patch set uploaded (by BBlack):
cache_upload: remove unused "rendering" backend

https://gerrit.wikimedia.org/r/257633

gerritbot added a comment.Dec 8 2015, 4:33 PM

Change 257634 had a related patch set uploaded (by BBlack):
add backend_random to maps and upload clusters in conftool-data

https://gerrit.wikimedia.org/r/257634

gerritbot added a comment.Dec 8 2015, 4:33 PM

Change 257635 had a related patch set uploaded (by BBlack):
add backend_random to maps and upload clusters config

https://gerrit.wikimedia.org/r/257635

gerritbot added a comment.Dec 8 2015, 4:33 PM

Change 257636 had a related patch set uploaded (by BBlack):
varnish: always use backend_random for pass/hfp

https://gerrit.wikimedia.org/r/257636

gerritbot added a comment.Dec 9 2015, 2:42 AM

Change 257630 merged by BBlack:
varnish: cache /api/rest_v1/ in backends

https://gerrit.wikimedia.org/r/257630

gerritbot added a comment.Dec 9 2015, 2:44 AM

Change 257631 merged by BBlack:
varnish: security_audit backend explicitly tier-one-only

https://gerrit.wikimedia.org/r/257631

gerritbot added a comment.Dec 9 2015, 2:49 AM

Change 257633 merged by BBlack:
cache_upload: remove unused "rendering" backend

https://gerrit.wikimedia.org/r/257633

gerritbot added a comment.Dec 9 2015, 2:51 AM

Change 257634 merged by BBlack:
add backend_random to maps and upload clusters in conftool-data

https://gerrit.wikimedia.org/r/257634

gerritbot added a comment.Dec 9 2015, 3:37 AM

Change 257635 merged by BBlack:
add backend_random to maps and upload clusters config

https://gerrit.wikimedia.org/r/257635

gerritbot added a comment.Dec 9 2015, 3:13 PM

Change 257636 merged by BBlack:
varnish: always use backend_random for pass/hfp

https://gerrit.wikimedia.org/r/257636

gerritbot added a comment.Dec 10 2015, 8:12 PM

Change 258207 had a related patch set uploaded (by BBlack):
VCL: no special handling for CentralAutoLogin

https://gerrit.wikimedia.org/r/258207

gerritbot added a comment.Dec 10 2015, 8:15 PM

Change 257632 abandoned by BBlack:
varnish: return (pass) for CAL URLs

https://gerrit.wikimedia.org/r/257632

gerritbot added a comment.Dec 11 2015, 3:44 PM

Change 258463 had a related patch set uploaded (by BBlack):
cache_misc: send randomized pass traffic directly to t1 backends

https://gerrit.wikimedia.org/r/258463

gerritbot added a comment.Dec 11 2015, 3:44 PM

Change 258464 had a related patch set uploaded (by BBlack):
cache_upload: send randomized pass traffic directly to t1 backends

https://gerrit.wikimedia.org/r/258464

gerritbot added a comment.Dec 11 2015, 3:44 PM

Change 258465 had a related patch set uploaded (by BBlack):
cache_text/mobile: send randomized pass traffic directly to t1 backends

https://gerrit.wikimedia.org/r/258465

gerritbot added a comment.Dec 14 2015, 11:53 AM

Change 258463 merged by BBlack:
cache_misc: send randomized pass traffic directly to t1 backends

https://gerrit.wikimedia.org/r/258463

gerritbot added a comment.Dec 14 2015, 12:52 PM

Change 258464 merged by BBlack:
cache_upload: send randomized pass traffic directly to t1 backends

https://gerrit.wikimedia.org/r/258464

gerritbot added a comment.Dec 14 2015, 3:26 PM

Change 258207 merged by BBlack:
VCL: no special handling for CentralAutoLogin

https://gerrit.wikimedia.org/r/258207

gerritbot added a comment.Dec 14 2015, 3:51 PM

Change 258465 merged by BBlack:
cache_text/mobile: send randomized pass traffic directly to t1 backends

https://gerrit.wikimedia.org/r/258465

gerritbot added a comment.Dec 16 2015, 3:47 PM

Change 259518 had a related patch set uploaded (by BBlack):
Revert "cache_text/mobile: send randomized pass traffic directly to t1 backends"

https://gerrit.wikimedia.org/r/259518

gerritbot added a comment.Dec 16 2015, 3:48 PM

Change 259518 merged by BBlack:
Revert "cache_text/mobile: send randomized pass traffic directly to t1 backends"

https://gerrit.wikimedia.org/r/259518

gerritbot added a comment.Dec 17 2015, 9:34 PM

Change 259800 had a related patch set uploaded (by BBlack):
Revert "Revert "cache_text/mobile: send randomized pass traffic directly to t1 backends""

https://gerrit.wikimedia.org/r/259800

gerritbot added a comment.Dec 17 2015, 9:41 PM

Change 259800 merged by BBlack:
Revert "Revert "cache_text/mobile: send randomized pass traffic directly to t1 backends""

https://gerrit.wikimedia.org/r/259800

gerritbot added a comment.Dec 18 2015, 5:18 PM

Change 260048 had a related patch set uploaded (by BBlack):
varnish: appropriate t2-fe -> t1-be backend_options

https://gerrit.wikimedia.org/r/260048

gerritbot added a comment.Dec 19 2015, 3:29 PM

Change 260048 merged by BBlack:
varnish: appropriate t2-fe -> t1-be backend_options

https://gerrit.wikimedia.org/r/260048

gerritbot added a comment.Dec 19 2015, 4:35 PM

Change 260204 had a related patch set uploaded (by BBlack):
Revert "Revert "Revert "cache_text/mobile: send randomized pass traffic directly to t1 backends"""

https://gerrit.wikimedia.org/r/260204

gerritbot added a comment.Dec 19 2015, 4:35 PM

Change 260204 merged by BBlack:
Revert "Revert "Revert "cache_text/mobile: send randomized pass traffic directly to t1 backends"""

https://gerrit.wikimedia.org/r/260204

BBlack closed this task as Resolved.Feb 19 2016, 4:50 PM
BBlack claimed this task.

In spite of good intentions, of course any "refactor complicated thing X" ticket just drags on forever as a pointless moving target. Closing this because it's not really useful as a task.

BBlack closed subtask T119396: Create globally-unique varnish cache cluster port/instancename mappings as Declined.Feb 13 2018, 3:09 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits