Currently the L4 LBs sends healtchecks for Varnish through HAProxy "transparently", meaning that HAProxy routes these requests along with the other (non-hc) ones to Varnish.
This could be problematic during high traffic (legitimate or not) as an eventual limit on maximum number of connections towards Varnish will impact the healthcheck requests too, with obvious consequences.
A solution is to use another, dedicated backend on HAProxy just for healthchecks, with a custom ACL to differentiate the traffic.
Implementation notes:
Currently healthchecks sent by PyBal have the following characteristics that should be matched by the HAProxy ACL:
- Host: healthcheck.wikimedia.org
- Url: /varnish-fe
- Source IPv4/IPv6 addr: [depends on the DC]
The ACL and dedicated backend in the HAProxy configuration should be surrounded by a hiera switch to allow a smoother and safer deployment across all cp hosts.
The list of the source IP addresses (L4 LBs) should go into a separate list file for ACL readability and to easily differentiate them per DC.
The change has been currently deployed to:
- ulsfo
- eqsin
- codfw
- drms
- eqiad
- esams