to reduce load on LVS hosts

My recollection is that it wasn't really about raw load or PPS at the LVSes. It was that our Linux kernel settings have some tunable icmp ratelimiting built in to avoid certain DDoS or reflection attacks, and that the volume of random echo pings was high enough that it was causing some legitimate ICMPs (like PTB for MTU probing) to be dropped by the shared ratelimiter. So this was to offload the echoes and make room within the ratelimit for the important cases.

All of this (both the behavior of 3rd parties and what those icmp tunables look like in modern kernels and/or our sysctl tuning) may have changed since then, of course.

The current puppetized tuneables are at: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/8ed59718c7a7603b61d7d42e05726fd11dae5eaa/modules/lvs/manifests/kernel_config.pp#49

Reading into the code above and the history more and self-correcting: the ratelimiter doesn't apply to PTB packets, just some other informational packets. Apparently we bumped the ratelimiter first as a short-term mitigation (for all the sites), I guess primarily to avoid what looks like ping loss to our monitoring and/or users, then deployed the ping offloader in some places as well as a better way to deal with it (and I guess at thousands per second, the pps reduction probably is useful, although I don't know to what degree).

Do we have any way to measure it's impact? I had a quick look at available promethues metrics and didn't see much corresponding to icmp (but may have missed them).

My thinking was we could possibly disable the offload in in eqiad or codfw and measure the impact? Certainly DDOS patterns are constantly changing and perhaps there is less ICMP flooding than before?

Right now we seem to be getting away with it at the POPs anyway. Presumably we're not hitting those rate limits or we'd be getting the spurious "host down" alerts we want to avoid.

Lastly we could potentially have some sort of rate-limiting configured on the switch-side for ICMP echo, which was IP-aware and didn't count packets from our own internal systems. A bit of work in that however, the QoS stuff we've been testing doesn't have complex network-side classification, and is only designed to kick in when the overall rate exceeds link speed.

some sort of rate-limiting configured on the switch-side for ICMP echo, which was IP-aware and didn't count packets from our own internal systems

Personally I'd rather we be reliable on external pings (within reason, anyways), otherwise users will assume it's evidence we're down or having problems when we're not, based on their unreliable pings.

https://grafana.wikimedia.org/d/000000513/ping-offload might be a good starting point (might need some updates/tweaking to get the exact data you want, though)

In T345809#9168116, @BBlack wrote:

some sort of rate-limiting configured on the switch-side for ICMP echo, which was IP-aware and didn't count packets from our own internal systems

Personally I'd rather we be reliable on external pings (within reason, anyways), otherwise users will assume it's evidence we're down or having problems when we're not, based on their unreliable pings.

Thanks @BBlack, that does make sense.

In T345809#9168091, @cmooney wrote:

Do we have any way to measure it's impact? I had a quick look at available promethues metrics and didn't see much corresponding to icmp (but may have missed them).

My thinking was we could possibly disable the offload in in eqiad or codfw and measure the impact? Certainly DDOS patterns are constantly changing and perhaps there is less ICMP flooding than before?

If I remember correctly, the initial drive for the ping* hosts wasn't trouble caused by the day-to-day, but some Google cloud image or service started to use "ping wikipedia" as an ongoing health or connectivy check. That caused notable impact to us, but eventually we managed to reach someone at Google to fix that. But such a situation might reappear any time eben though we don't currently see it in the metrics.

Change 964521 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/alerts@master] netops: remove pingoffload alert from esams

https://gerrit.wikimedia.org/r/964521

Change 964521 merged by Filippo Giunchedi:

[operations/alerts@master] netops: remove pingoffload alert from esams

https://gerrit.wikimedia.org/r/964521

Thanks for the task and feedback. If the issue is abuse from a limited number of providers (like in T163312: lvs2001: intermittent packet loss from Icinga checks it seems better to filter out that kind of traffic in an incident response kind of way than maintaining permanent infrastructure for it.
esams however have a constant "background noise" of ICMP an order of magnitude higher (and kind of always had), that's why the other POPs didn't have any ping offload VMs. That said, it doesn't seem to be causing any issue probably thanks to the Kernel tuning done previously.

Based on that and the fact that we will later on have a new LB system, I'd be more inclined to decommission them everywhere than provision new ones.