After a few minutes the webservice is in CrashLoopBackOff again:

tools.scholia@tools-bastion-15:~$ kubectl get all
NAME                           READY   STATUS             RESTARTS      AGE
pod/scholia-7b8d846c5c-wv9h2   0/1     CrashLoopBackOff   5 (64s ago)   6m7s

I stopped it again and opened https://github.com/WDscholia/scholia/issues/2794

Mentioned in SAL (#wikimedia-cloud) [2026-06-21T09:18:49Z] <dhinus> webservice stop (tried restarting, but it keeps crashing) T429738

This triggered again today, around one hour ago, and scholia was stopped (and the alert stopped too).

There's though an increased number of sessions on the frontend, that are hitting now fourohfour:

Sessions on the proxy:

Fourohfour traffic:

You can see here on the proxy, that the 500s that trigger the alet (in red) moved to 400s (in orange):

Another one where you can also see that the traffic on scholia moved to fourohfour, and multiplied at the same time (probably due to retries):

Another one where you can also see that the traffic on scholia moved to fourohfour, and multiplied at the same time (probably due to retries):

Is this chart from one of our dashboards? I could not find it.

I created a new one targeted to this specific issue: https://grafana.wmcloud.org/d/fnhp8st/tool-error-rates

Indeed now with the work by @taavi on T392356: Replace ingress-nginx before upstream EOL date we do have per-tool status codes as seen by istio, i.e. the metric used in https://grafana.wmcloud.org/d/fnhp8st/tool-error-rates based on istio_requests_total

With that in mind we can do per-tool availability as rate(5xx) / rate(all-codes):

1 - (
sum by (destination_service_namespace) (rate(istio_requests_total{response_code=~"5.."}[5m]))
  /
sum by (destination_service_namespace) (rate(istio_requests_total[5m]))
)

Then I think it makes sense to frame toolforge availability as: max( unavailable tools (i.e. availability < 1) )

To illustrate the example, this is availability during the eqiad incident the other day. Shown here as taking top20 tools just for illustration purposes:

Or taken as the max()

Links respectively: https://w.wiki/R$Fv https://w.wiki/R$Fw

The tool availability metric will need to be a recording rule because istio_requests_total is huge. And the same per-tool availability metric can also feed the KR metrics for webservice unavailability we were talking about yesterday (cc @CCiufo-WMF @aputhin)

Moving the alerting to be based on Istio-level metrics means we lose visibility into any potential connectivity issues between HAProxy and Istio, right?

fnegri opened https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/65

ToolforgeWebHighErrorRate: increase threshold

Maybe we could keep the haproxy alert but with a higher threshold? I opened https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/65 that is also a quick fix if we want to restart Scholia.

To clarify my previous comment: maybe we could keep the haproxy alert at a 50% threshold (or higher), but also have an additional alert based on istio, like the one suggested by @fgiunchedi above.

fnegri merged https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/65

ToolforgeWebHighErrorRate: increase threshold

Mentioned in SAL (#wikimedia-cloud) [2026-07-03T13:59:50Z] <dhinus> "webservice -m 1G python3.9 start" (after increasing the alert threshold in T429738)

Restarted the scholia tool, and posted an update in https://github.com/WDscholia/scholia/issues/2794.

In T429738#12085068, @taavi wrote:

Moving the alerting to be based on Istio-level metrics means we lose visibility into any potential connectivity issues between HAProxy and Istio, right?

Yes indeed, if haproxy can't talk to istio at all is I think the most prominent case. The other way around (i.e. istio can't talk to haproxy) should be covered (to be verified) because istio/envoy reports such condition as status flags, I'm assuming as part of a 5xx: https://www.envoyproxy.io/docs/envoy/latest/configuration/advanced/substitution_formatter#config-access-log-format-response-flags (DC I think)

I don't feel strongly either way on which way to go (both alerts, keep the current alert with lower threshold, something else) as long as we're comfortable with it as a paging alert (i.e. high confidence it won't misfire)