meanwhile I've restored the component and uploaded haproxy 2.8.20 there

Closing the task as this has been investigated

I've replicated locally a SSL handshake failure using haproxy with log-format-sd %{+E}o\ [haproxykafka@0\ %[capture.req.hdr(0),json(ascii)]|%HPO|%HQ|%rt]

Yes, sequence numbers are enerated by haproxy itself, even if it results in a SSL handshake error where the sequence number doesn't reach haproxykafka when using haproxy 3.0 because the log format is ignored for that kind of error.

It looks like the root cause is MEDIUM: log/session: handle embryonic session log within sess_log(). A change introduced in HAProxy 3.1 as part of the work done to introduce the log profiles feature.

In T422030#11778885, @Fabfur wrote:

This is most probably due to a deprecation in haproxy configuration directives https://www.haproxy.com/blog/reviewing-every-new-feature-in-haproxy-3-1#deprecation, especially

option     accept-invalid-http-request
option     accept-invalid-http-response

this is a side effect of moving healthchecks on upload from healthcheck.wm.org to upload.wm.o in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1164466

For instance, a recent decommissioning of codfw cp nodes (T419753) left the legacy ats-be service unavailable and caused the (depooled!) nodes to increment varnish_sli_bad as there were no servers left in the site according to them

There is no CI check for validity of lua scripts loaded in trafficserver; some have functional tests but a syntax error should have never passed CI, even in the absence of tests

this is a by-product of switching to Katran and not a feature that we can deploy independently at the moment, so in eqiad and codfw this is currently blocked till we can switch every service to IPIP encapsulation

with the patch applied acme-chief was able to issue the certificate a few hours after reporting the issue to GTS (no active answer from them though)

I've tried to patch our client to skip already validated challenges but I'm running into another issue, this is the request flow performed by acme-chief:

this seems to be tracked as T417278

In T417825#11655005, @MShilova_WMF wrote:

@Vgutierrez , this is part of the API rate limiting rollout prep, and we’re trying to confirm whether everything needed for the March 9 phase is on track.

Do you have a sense of current status and expected timeline for this change? Let me know if there’s anything we should be aware of or help unblock.

could I suggest using a dedicated prefix for API/REST gateway headers? We already use internally X-WMF as a prefix on other layers that aren't related to API/REST gateways.

change has been merged, and it should be live by now

change has been merged, please allow puppet to propagate the change, it could take up to 30 minutes

SSH has been verified out-of-band

got mcollins approval via Slack, we need Data-Engineering approval now (that's @Milimetric / @Ottomata)

waiting for mcollins approval, I've pinged them on Slack cause I've failed to find their phabricator user so far

In T417864#11632385, @Clement_Goubert wrote:

In the merged task, I was proposing not creating a new header, but instead adding to the existing X-Requestctl header using the same convention we use for haproxy applied rules, so something like X-Requestctl: rgw:${x-wmf-ratelimit-class}.

blast radius is big.. I'm wondering if k8s nodes have workloads not exposed to the Internet where having a bigger MTU (thinking about jumbo frames here) could be beneficial or even required in performance terms.

fixed by:

• Retrying if we can't fetch the MAC address
• Reporting the configured MAC address
• Refusing to start if we can't fetch the MAC address after 10 attempts
• Triggering an ARP resolution if the MAC address isn't on the kernel neighbors table

In T417536#11622959, @hashar wrote:

That is great thank you :-]

I am wondering though why reusing connections leads to errors. Apache has the default Debian configuration as far as I can tell:

# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive On

# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 100

# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 5

Can it be that ATS has a much longer timeout than the Apache 5 seconds timeout? If so i imagine:

• ATS picks up a connection that it considers not having timed out (idling less than whatever timeout it has) but is about to expire on the Apache side (it has been idle for almost 5 seconds).
• ATS sends the headers and at the same time, because the connection has been idling for 5 seconds Apache terminates it (tcp FIN?)
• ATS gets the unexpected termination, gives up and emit a 502 response

I managed to trigger this while capturing the traffic between ATS and gerrit2003, in my run it failed fetching https://gerrit.wikimedia.org/r/mediawiki/extensions/RelatedArticles, this is the content of the offending request:

HAProxy 3.0 bumps to lua 5.4, as a consequence HAProxy fails to start cause lua5.4-maxminddb isn't there:

Feb 12 15:51:33 cp4052 haproxy[2379768]: [NOTICE]   (2379768) : haproxy version is 3.0.15-1~bpo11+1
Feb 12 15:51:33 cp4052 haproxy[2379768]: [NOTICE]   (2379768) : path to executable is /usr/sbin/haproxy
Feb 12 15:51:33 cp4052 haproxy[2379768]: [ALERT]    (2379768) : config : parsing [/etc/haproxy/haproxy.cfg:18] : Lua runtime error: /etc/haproxy/lua/maxmind-lookup.lua:3: module 'maxminddb' not found:
Feb 12 15:51:33 cp4052 haproxy[2379768]:         no field package.preload['maxminddb']
Feb 12 15:51:33 cp4052 haproxy[2379768]:         no file '/etc/haproxy/lua/private/maxminddb.lua'
Feb 12 15:51:33 cp4052 haproxy[2379768]:         no file '/usr/local/share/lua/5.4/maxminddb.lua'
Feb 12 15:51:33 cp4052 haproxy[2379768]:         no file '/usr/local/share/lua/5.4/maxminddb/init.lua'
Feb 12 15:51:33 cp4052 haproxy[2379768]:         no file '/usr/local/lib/lua/5.4/maxminddb.lua'
Feb 12 15:51:33 cp4052 haproxy[2379768]:         no file '/usr/local/lib/lua/5.4/maxminddb/init.lua'
Feb 12 15:51:33 cp4052 haproxy[2379768]:         no file '/usr/share/lua/5.4/maxminddb.lua'
Feb 12 15:51:33 cp4052 haproxy[2379768]:         no file '/usr/share/lua/5.4/maxminddb/init.lua'
Feb 12 15:51:33 cp4052 haproxy[2379768]:         no file './maxminddb.lua'
Feb 12 15:51:33 cp4052 haproxy[2379768]:         no file './maxminddb/init.lua'
Feb 12 15:51:33 cp4052 haproxy[2379768]:         no file '/usr/local/lib/lua/5.4/maxminddb.so'
Feb 12 15:51:33 cp4052 haproxy[2379768]:         no file '/usr/lib/x86_64-linux-gnu/lua/5.4/maxminddb.so'
Feb 12 15:51:33 cp4052 haproxy[2379768]:         no file '/usr/lib/lua/5.4/maxminddb.so'
Feb 12 15:51:33 cp4052 haproxy[2379768]:         no file '/usr/local/lib/lua/5.4/loadall.so'
Feb 12 15:51:33 cp4052 haproxy[2379768]:         no file './maxminddb.so'
Feb 12 15:51:33 cp4052 haproxy[2379768]: [ALERT]    (2379768) : config : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
Feb 12 15:51:33 cp4052 haproxy[2379768]: [ALERT]    (2379768) : config : parsing [/etc/haproxy/conf.d/tls.cfg:180] : error detected in proxy 'tls' while parsing 'http-request set-var(req.provenance,ifnotexists,ifnotempty)' rule : unknown fetch method 'lua.fetch_isp'.
Feb 12 15:51:33 cp4052 haproxy[2379768]: [ALERT]    (2379768) : config : Error(s) found in configuration file : /etc/haproxy/conf.d/tls.cfg
Feb 12 15:51:33 cp4052 systemd[1]: haproxy.service: Control process exited, code=exited, status=1/FAILURE
Feb 12 15:51:33 cp4052 systemd[1]: Reload failed for HAProxy Load Balancer.

You just need to expose caniuse.com browser data to hiddenparma.

It's not clear why the net.ipv4.conf.default.rp_filter would need to change to 0 for IPIP and we would like to not do that on k8s nodes to prevent IP spoofing from inside containers (although that would require CAP_NET_ADMIN). If we could keep net.ipv4.conf.default.rp_filter=1 the dynamically created calico interfaces would still inherit that setting while net.ipv4.conf.all.rp_filter=0 would allow the ipip interfaces to also have rp_filter=0 set.

oh I see... you're using https://gerrit.wikimedia.org/ in the healthcheck URL instead of https://healthcheck.wikimedia.org/varnish-fe this needs to be fixed

hmm why these healthchecks from liberica are hitting the backend server in eqiad instead of staying in the cp nodes?

the headers described on https://wikitech.wikimedia.org/wiki/CDN/Backend_api and x-ja3n/x-ja4h should be hitting MediaWiki already

the assessment is OK and the link can be removed safely