Related to: T110472 (parsoidcache eventual decom) and T93428 (streamline service deploy)
For services, I'd like to move away from having to define custom VCL code stanzas and puppet backend-related fragments, and get to a place where we can define the service's varnish-layer needs in an abstract data structure recording some standard options for how the service behaves and its basic attributes, from which standardized VCL fragments are templated.
Hypothetical example (there are lots of holes in this if you stare at it long enough, it's just meant to approximate the direction of things):
text_services => {
cxserver => {
recv => {
host_eq => 'cxserver.wikimedia.org',
# or path_re => '/api/cx_v1/....'
},
cache_policy => 'pass' # no caching at all
# cache_policy => 'default' # normal, obeys CC-headers
# cache_policy => { ... future configurable declarative complexities ... }
backend => {
method => 'chash' # or round-robin, etc...
hosts => [ 'cxserver.svc.eqiad.wmnet' ],
port => 8000,
probe_options => {
timeout => 30s,
max_connections => 1000,
},
},
},
}The abstraction here is meant to treat the entire varnish cache layer of our infrastructure (ignoring internal details like layers and tiers) as a black box, and simply define the rules for how the cache layer as a whole handles requests for a certain distinct service. We can start out only implementing a limited set of options for the services that are simplest in terms of Varnish config needs, and migrate more services into this scheme (and away from current manual VCL code + puppet frags) as this mechanism gains the ability to handle more-complex cases. As things stand today, most of the services being handled by the parsoidcache (other than parsoid itself) currently, as well as most of the services flowing through the misc cluster, could be handled by a very basic version of the above without a lot of complexity-handling.
Perhaps ideally, this can be factored to remove some of the duplication between the backend data above and the inbound-side LVS definitions for the same internal service to really get the configuration minimized, but starting with just the abstraction of the Varnish bits is a good initial step in that direction (and not all backends are actually LVS-based anyways).