| fgiunchedi | |
| Mar 3 2026, 8:15 AM |
| F72481648: image.png | |
| Mar 3 2026, 9:36 AM |
Over the last ~3 days fourohfour has been failing its probes consistently, the first being at 2026-03-01T08:59:51:
metricsinfra-alertmanager-2:~# journalctl -u alertmanager-webhook-logger.service --since -1w --grep http_this_tool_does_not_exist_toolforge_org | wc -l 1183
Scaling up the deployment didn't help, eventually the tool fails its healthcheck and then gets restarted by k8s in a loop, eventually the pods end up in crashloop
On redis, there was an increase in the network traffic around the time the alert started triggering (not exactly, it seems a bit later):
https://grafana.wmcloud.org/goto/efew1jzmafytcc?orgId=1
It also matches an increase on the amount of set commands, and on the time the server spends on those.
Enabled logging on fourohfour, it seems that the did-you-mean urls are taking really long, looking
Update: not only those, others take also >5s :/
Added logs before and after hitting redis cache, and the slowdown does not seem to be there (diff between miss memory-hit redis):
[2026-03-03 10:16:32,815] WARNING in app: all_tools: Miss memory cache [2026-03-03 10:16:32,819] WARNING in app: all_tools: Hit redis cache [2026-03-03 10:16:33,517] WARNING in app: all_tools: Miss memory cache [2026-03-03 10:16:33,520] WARNING in app: all_tools: Hit redis cache [2026-03-03 10:16:33,617] WARNING in app: all_tools: Miss memory cache [2026-03-03 10:16:33,618] WARNING in app: all_tools: Hit redis cache [2026-03-03 10:16:33,717] WARNING in app: all_tools: Miss memory cache [2026-03-03 10:16:33,719] WARNING in app: all_tools: Hit redis cache [2026-03-03 10:16:36,861] WARNING in app: all_tools: Miss memory cache [2026-03-03 10:16:36,863] WARNING in app: all_tools: Hit redis cache [2026-03-03 10:16:37,090] WARNING in app: all_tools: Miss memory cache [2026-03-03 10:16:37,095] WARNING in app: all_tools: Hit redis cache [2026-03-03 10:16:37,748] WARNING in app: all_tools: Miss memory cache [2026-03-03 10:16:37,750] WARNING in app: all_tools: Hit redis cache [2026-03-03 10:16:38,324] WARNING in app: all_tools: Miss memory cache [2026-03-03 10:16:38,329] WARNING in app: all_tools: Hit redis cache [2026-03-03 10:16:38,834] WARNING in app: all_tools: Miss memory cache [2026-03-03 10:16:38,837] WARNING in app: all_tools: Hit redis cache n app: all_tools: Miss memory cache [2026-03-03 10:16:37,657] WARNING in app: all_tools: Miss memory cache [2026-03-03 10:16:37,714] WARNING in app: all_tools: Hit redis cache [2026-03-03 10:16:37,717] WARNING in app: all_tools: Hit redis cache
Some progress, I've added a memory cache for the 'did-you-mean' endpoint (that gets all tools and projects), and that reduced most of the calls made to it from several seconds to msecs
Things seem stable for almos 10m, most of the requests to that endpoint were for the non-existing tool quentinv57-tools.toolforge.org
That tool is redirected from the proxies too:
https://codesearch.wmcloud.org/search/?q=quentinv57-tools.toolforge.org
Change #1247722 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):
[operations/puppet@production] toolforge: Drop legacy redirects for quentinv57-tools
Change #1247722 merged by David Caro:
[operations/puppet@production] toolforge: Drop legacy redirects for quentinv57-tools
There was one restart of one pod during the night that caused an alert email
to be sent.
I merged @bd808's patch to remove that specific tool, and give a look at the
rest to cleanup also some of the old tools.
Change #1247963 had a related patch set uploaded (by David Caro; author: David Caro):
[operations/puppet@production] legacy_redirector: remove some disabled tools
Note that since I enabaled request logging yesterday, from the last 100k requests, >50k are for that specific host name (quentinv57-tools), so cleaning up the rest might not help much, but well, it's something.
Change #1247963 merged by David Caro:
[operations/puppet@production] legacy_redirector: remove some disabled tools
This seems to have been stable for some time, the improvement patches have been merged, I'll reopen if it crashes again.