FWIW, I think this pre-dated IPSec and probably isn't related to it. In earlier investigations it looked like a monitoring failure of some kind and not real. Later theory was that it was real, and related to the issues with short RA TTLs for the IPv6 default gateways, but it has persisted since fixing that as well...

This has always hit text and upload lb's (again, IPv6, in eqiad) as well, but usually they're less-likely than mobile to reach 3/3 and actually send an alert. I've downtimed all 3 temporarily just to avoid the excess paging, and we can look at this more in detail Monday.

So, this evening the same flaps hit (as they do most evenings), but they hit codfw ipv6 service IPs rather than eqiad. Both DCs have been active and monitored all along, so this means the alerts follow the large-scale traffic more than anything else, and they don't seem to be eqiad-specific. They've never hit esams or ulsfo before, though.

Digging around a bit and thinking, I stumbled on a new theory: this might be because of the small fixed default size of /proc/sys/net/ipv6/route/max_size. It's dynamically adjustable with normal sysctl, but it's always a fixed 4096 by default, whereas the ipv4 equivalent ends up being 2147483647 (2^31-1), and supposedly is based on system memory constraints and such. The big cache clusters (upload, mobile, text) in codfw were all showing values just under 4K for wc -l /proc/net/ipv6_route in codfw this evening, and presumably the same in eqiad when it has full traffic. For whatever reason, the same tables in ulsfo and esams are much smaller and nowhere near the 4K value (ulsfo closer to 2K-ish, esams only a few hundreds), which would explain why this doesn't hit those DCs.

I've manually adjusted the sysctl on all of the cp* machines to 131072 via salt for now, and I've been tailing the icinga log since then (nearly an hour now) and haven't even seen any 1/3 soft-fails for v6 service endpoints or the usual v6 ipsec flaps, either. Will leave the tail running overnight. Once the max_size was lifted, the codfw wc -l /proc/net/ipv6_route values jumped from their just-under-4K values to values in 5-7K range on those machines.

Will continue to observe, and tomorrow will dig deeper on exactly how that table works and what a sane way to size it would be, and also why esams seems so much less affected when it has at least as much total traffic (is v6 adoption really that much less in the EU than the US? there are other possible factors too, such as more cache boxes per cluster).

Update: log tail caught a few 1/3 soft fails on ipsec ipv6 still, but those could be due to legitimate timing and/or packet loss. The rate of them is much lower than before (more than an order of magnitude), and no soft-fails on ipv6 service IPs yet. Will keep watching through the next expected flap window.

See also https://bugzilla.redhat.com/show_bug.cgi?id=1221915 and the similar reports linked within, etc...

In T113154#1683862, @BBlack wrote:

Digging around a bit and thinking, I stumbled on a new theory: this might be because of the small fixed default size of /proc/sys/net/ipv6/route/max_size. It's dynamically adjustable with normal sysctl, but it's always a fixed 4096 by default, whereas the ipv4 equivalent ends up being 2147483647 (2^31-1), and supposedly is based on system memory constraints and such. The big cache clusters (upload, mobile, text) in codfw were all showing values just under 4K for wc -l /proc/net/ipv6_route in codfw this evening, and presumably the same in eqiad when it has full traffic. For whatever reason, the same tables in ulsfo and esams are much smaller and nowhere near the 4K value (ulsfo closer to 2K-ish, esams only a few hundreds), which would explain why this doesn't hit those DCs.

I've manually adjusted the sysctl on all of the cp* machines to 131072 via salt for now, and I've been tailing the icinga log since then (nearly an hour now) and haven't even seen any 1/3 soft-fails for v6 service endpoints or the usual v6 ipsec flaps, either. Will leave the tail running overnight. Once the max_size was lifted, the codfw wc -l /proc/net/ipv6_route values jumped from their just-under-4K values to values in 5-7K range on those machines.

Will continue to observe, and tomorrow will dig deeper on exactly how that table works and what a sane way to size it would be, and also why esams seems so much less affected when it has at least as much total traffic (is v6 adoption really that much less in the EU than the US? there are other possible factors too, such as more cache boxes per cluster).

That's a very good find and catch. A few years ago we were having frequent issues with the IPv4 route cache filling up, in particular in cases of traffic surges. We had to workaround these until upstream Linux ditched the IPv4 route cache entirely.

This is the knob for the IPv6 routing table but IIRC is also the same for the route cache table. 4096 would definitely not be enough and cause this effect. Ironically, us switching to HTTPS for which we use sh probably delayed the occurence of this issue considerably!

IPv6 connectivity is on the rise (exponentially, even) which could explain the sudden occurence of this issue. It's currently at ~8.5% globally and ~21.5% in the US, according to Google's IPv6 statistics, the most complete analysis on the subject. The iOS 9 release probably also gave a boost to the number of IPv6 users globally because of a changed behavior in the OS.

To answer your other question on US vs. EU IPv6: as you can see from the Google stats above, Europe's IPv6 connectivity varies greatly from country to country due to different operators, but it's clear that on average cannot top the US' numbers.

Moreover, I ran a quick log analysis over our own (sampled 1:1000) logs for two dates, yesterday and Aug 8th. The findings are:

• IPv6 globally is at 8.45%, was 7.64% back in August.
• 46.49% of our (IPv4/IPv6) traffic hits esams (42.85% in August); ulsfo is at 20.56% (14.11%), eqiad+codfw is at 32.95% (43.04% in August). Wwe've reshuffled half of the US' traffic during this time period, so the difference between Aug/Sep makes sense.
• 11.47% (11.87%) of all of requests ending up at eqiad+codfw are IPv6. For ulsfo it's 10.6% (3.79%) and for esams it's 5.35% (4.67%).
• IPv6 requests hitting eqiad+codfw+ulsfo are 70.55% of all IPv6 requests (73.79%).
• IPv6 requests hitting just eqiad+codfw are now 44.70% (66.78%) of all IPv6 requests, i.e. ulsfo handles 25.85% (7.01%) of all IPv6 requests.

All of the above seem to support that a) eqiad/codfw get a lot more IPv6 traffic both in comparison and in absolute numbers than esams/ulsfo (without factoring in number of servers/requests per server), b) there is a strong correlation of the US-West move to ulsfo earlier this month changing the IPv6 ratio for ulsfo, which in turn suggests that the percentages of IPv6 users in the US are much higher than Asia (or the rest of the world for that matter).

Change 242122 had a related patch set uploaded (by BBlack):
Bump v6 route max_size to 131072 for all

https://gerrit.wikimedia.org/r/242122

Change 242122 merged by Faidon Liambotis:
Bump IPv6 route max_size to 131072 for all

https://gerrit.wikimedia.org/r/242122

FWIW, I did some additional searching and ended up... in Facebook's IPv6 work, which explains well the symptoms we're seeing. Their work has been merged into Linux 4.2-rc1 from what I can see, so things would be considerably improved then — but in the meantime, the fixes max_size increase that was just pushed should alleviate them for our scale.

Let's monitor this for another 24 hours and resolve.

^ +1