I possible scenario can be the following:

• We have mixed servers (k8s and bare metal)
• The old bare metal environment is allowed to talk to public mediawiki API directly (eg. en.wikipedia.org) where the new k8s setup doesnt.
• When we receive a request for a snapshot with an overlay geoshape (lets call it A) the service makes an HTTP API request to itself (lets call this request B)
• In the old bare metal setup the request (B) for the "internal" geoshapes is sent to the public kartotherian endpoint
• It could be the case that one of the initial requests A was handled by a bare metal server and the follow up (B) "internal requests" ended up in k8s which doesn't allow external traffic
• This caused timeouts and because requests were hanging, increase in load

Thanks a lot for the brainbounce :)

From logstash I don't see anything logged by k8s eqiad pods during the outage, if request B failed I'd have expected to see some failures (but maybe kartotherian doesn't currently log it for some reason).

In the use case outlined above, request A generated the internal geoshape request B to the public kartotherian endpoint, that is maps.wikimedia.org. At that point, request B can be served from either a pod or a bare metal node, but both of them should be ok to return the right HTTP response. Am I missing something? In any case we should verify in staging (k8s only) if we can reproduce this case..

I agree that it is something related to the bare metal nodes being left with less resources, because for some reason they cannot use what is offered by k8s pods. I am wondering if we should just depool all bare metals entirely in codfw (that seems less trafficated), after pooling in all the 5 k8s pods, and see if any trouble emerges. Maybe the new version can handle traffic in the right way, meanwhile the old one needs a specific amount of bare metal nodes to be up otherwise it starts failing.

I am not happy that we don't have a clear root cause yet, trying the above procedure seems a shot in the dark and I don't love to work in this way, but maybe it is the only way forward.

Lemme know your thoughts @Jgiannelos :)

The errors not showing up on the k8s side could be because there was actually no error, the ETIMEDOUT is raised in the client side, the server side just hangs.

In T386648#10559641, @Jgiannelos wrote:

The errors not showing up on the k8s side could be because there was actually no error, the ETIMEDOUT is raised in the client side, the server side just hangs.

Something doesn't add up though - the HTTP 504 stacktrace mentions "internalURI":"https://en.wikipedia.org/w/api.php" for the ETIMEDOUT, not maps.wikimedia.org..

I agree, but the only env that could hang on en.wikipedia.org is k8s, maps nodes can talk to that endpoint directly.

I want to make a test via LVS load balancing config, namely put more weights on k8s pods/workers (5 in total in each DC) so that more requests will be served there. If my theory is correct, namely that bare metals can be overwhelmed really easy, we shouldn't see any impact but more requests served by k8s. Otherwise we'll see a raise in timeouts, and a rollback to the previous step should be easy.

I took a quick look this morning after seeing the discussion in IRC, and it looks like none of the k8s worker nodes that have been added to the kartotherian-k8s-ssl service in conftool actually have the LVS IP configured on their loopback device. In terms of service turnup steps, kartotherian-k8s-ssl was never added to profile::lvs::realserver::pools on these hosts.

Not 100% sure what routing the behavior would be in that case ... Perhaps hairpin back to LVS? Worst case, loop? (e.g., there was a comment in IRC about seeing TTL exceeded)

In any case, since the desired end-state is to have all k8s workers be aware of the LVS IP, it probably makes sense to add the service to the pools list in hieradata/role/common/kubernetes/worker.yaml.

For debugging purposes, this is a URL that requests a snapshot with an overlay map from en.wikipedia.org:

https://maps.wikimedia.org/img/osm-intl,a,a,a,300x200.png?lang=en&domain=en.wikipedia.org&title=Alabama&revid=1268325471&groups=_1b46af921bb4e1f090a1b07748a50bd2e1f322fc

From my local env when I try this request this shows up in the trace logs so indeed it should eventually request en.wikipedia.org:

[2025-02-19T05:00:20.053Z] TRACE: kartotherian/20 on 48bfdbb6c162: Outgoing request (request_id=6b03dd00-ee7e-11ef-b32e-cf29ca820246, levelPath=trace/req)
    out_request: {
      "method": "post",
      "uri": "https://en.wikipedia.org/w/api.php",
      "headers": {
        "user-agent": "kartotherian",
        "x-request-id": "6b03dd00-ee7e-11ef-b32e-cf29ca820246"
      },
      "body": {
        "format": "json",
        "formatversion": "2",
        "action": "query",
        "revids": "1268325471",
        "prop": "mapdata",
        "mpdlimit": "max",
        "mpdgroups": [
          "_1b46af921bb4e1f090a1b07748a50bd2e1f322fc"
        ]
      }
    }
    --
    request: {
      "url": "/img/osm-intl,a,a,a,300x200.png?lang=en&domain=en.wikipedia.org&title=Alabama&revid=1268325471&groups=_1b46af921bb4e1f090a1b07748a50bd2e1f322fc",
      "headers": {
        "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:135.0) Gecko/20100101 Firefox/135.0",
        "x-request-id": "6b03dd00-ee7e-11ef-b32e-cf29ca820246"
      },
      "method": "GET",
      "params": {
        "0": "/img/osm-intl,a,a,a,300x200.png"
      },
      "query": {
        "lang": "en",
        "domain": "en.wikipedia.org",
        "title": "Alabama",
        "revid": "1268325471",
        "groups": [
          "_1b46af921bb4e1f090a1b07748a50bd2e1f322fc"
        ]
      },
      "remoteAddress": "192.168.65.1",
      "remotePort": 48165
    }

Change #1120893 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] role::kubernetes::worker: add kartotherian-k8s-ssl to the lvs pools

https://gerrit.wikimedia.org/r/1120893

Change #1120893 merged by Elukey:

[operations/puppet@production] role::kubernetes::worker: add kartotherian-k8s-ssl to the lvs pools

https://gerrit.wikimedia.org/r/1120893

In T386648#10560543, @Scott_French wrote:

I took a quick look this morning after seeing the discussion in IRC, and it looks like none of the k8s worker nodes that have been added to the kartotherian-k8s-ssl service in conftool actually have the LVS IP configured on their loopback device. In terms of service turnup steps, kartotherian-k8s-ssl was never added to profile::lvs::realserver::pools on these hosts.

Not 100% sure what routing the behavior would be in that case ... Perhaps hairpin back to LVS? Worst case, loop? (e.g., there was a comment in IRC about seeing TTL exceeded)

In any case, since the desired end-state is to have all k8s workers be aware of the LVS IP, it probably makes sense to add the service to the pools list in hieradata/role/common/kubernetes/worker.yaml.

This is almost probably the root cause of the problem, we were discussing on IRC why sometimes the Prometheus network probes were failing, and it was due to k8s workers not being able to accept traffic from the load balancer. When I removed bare metals from the LB I caused more pressure on the remaining nodes, that almost probably had timeouts due to k8s workers not being reachable.

@Jgiannelos see the above, my impression is that it was me causing the issue and not the code itself. In the example that you posted there is indeed a call to en.wikipedia.org, but I thought we had in place some rewrite-to-localhost-proxy config for that use case (I am pretty sure we tested it in staging).

I just depooled two bare metal nodes from each DC (like I did on Monday) and everything seems stable, so I think we got the root cause :)

Last step before closing: write the incident report.

Change #1121309 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] services: bump allocated memory for Kartotherian

https://gerrit.wikimedia.org/r/1121309

Change #1121309 merged by Elukey:

[operations/deployment-charts@master] services: bump allocated memory for Kartotherian

https://gerrit.wikimedia.org/r/1121309

Change #1121315 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] services: update cpu resources for kartotherian's mesh/statsd containers

https://gerrit.wikimedia.org/r/1121315

Change #1121315 merged by Elukey:

[operations/deployment-charts@master] services: update cpu resources for kartotherian's mesh/statsd containers

https://gerrit.wikimedia.org/r/1121315

I started reviewing the metrics related to Monday's outage but I didn't find what I expected. Let's start with timings:

https://sal.toolforge.org/production?p=1&q=&d=2025-02-17

• I pooled the wikikube workers in eqiad at around 12:49 UTC
• I deployed https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1120178 to raise the replicas of Kartotherian at around 14:09 UTC. It was just to fix some inconsistency, I didn't observe anything at the time.
• I removed the maps nodes (bare metals) at around 14:15, and after some minutes the impact was really clear (see https://logstash.wikimedia.org/goto/ea66e4c15220f8576dc011ec8ebf438b and ATS metrics.

From 12:49 UTC to 14:09 UTC the eqiad LVS load balancer contained backends (the k8s workers) without the kartotherian's loopback interface deployed, so sometimes a connection to kartotherian.discovery.wmnet:6543 ended up in a connection failure (due to looping between LVS and the wikikube workers, that didn't accept the IP packets for the Kartotherian SVC IP). In theory I'd have expected to see some impact registered on the CDN side as 502/503 for maps.wikimedia.org, but nothing. I am very happy if no harm to external users was caused during that timeframe, but I don't understand why :D

I rechecked the IRC logs to see if I somehow lost any notification of wikikube workers behind the kartotherian LVS not passing health checks, but nothing was registered before the 14:15 mark (after that the maps bare metals started for fail health checks).

Edit: mistery solved after talking with Valentin. ATS heavily reuse connections to the backend, as one can see in this graph. When I pooled the k8s workers, they passed health checks from pybal but ATS failed to connect to them, so it relied on the connection to the "healthy" bare metals. This allowed to reduce the impact by a lot :)

https://wikitech.wikimedia.org/wiki/Incidents/2025-02-19_maps published :)