It would be helpful to have a health check endpoint for our services. Can we update ours to reflect our system 'health' accurately? We have a health check endpoint live somewhere.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | BUG REPORT | Jdforrester-WMF | T406848 wide-scale Python failure | ||
| Resolved | Spike | Jdforrester-WMF | T408899 Change the function-evaluator readiness endpoint from _info to a new one that actually tells k8s if we're healthy for evaluation |
It may be possible for us to set up a user-facing page that behaves as a type of health check; we could use the scripts we use during our deploy process that pings our services that ensures they are alive and responsive. Starting off, the page would be sparse, albeit usable. What do you think we wire this up and then beautify it next year (next fiscal quarter(s))?
Product Q for Laura (can't find her account yet) and cc'ing @DSantamaria
No, the _info endpoint is the health-check for k8s (and doesn't check much health!).
@Jdforrester-WMF @ecarg, Then why do we need a user-facing page for doing a health check in this case?
It's not clear to me what the purpose of this task is. Are we trying to give people visibility? (Which people?) Are we looking for alerting / monitoring? Are we trying to get the system to heal itself?
This task was an action item from one of our team meetings following the Py Evaluator outage; IIRC @DSantamaria's proposition to have a user-facing system availability endpoint set up.
@ecarg @Jdforrester-WMF If I did propose that, I am unsure about the goal of having a user-facing endpoint. My main goal is to have observability around this. If this can be done with an internal endpoint similar to what we have in the _info one, that would definitely work for me, the idea is to have observability and alerting triggered when one of our services stop working and not being alerted by the community as the last outages in the team.
See https://wikitech.wikimedia.org/wiki/Streamlined_Service_Delivery_Design#Readiness for the (slighted-dated) documentation about how to tweak this endpoint.
With https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/456 deployed we can see the live wasmProcessCount at the _info endpoint:
jforrester@deploy2002:/srv/deployment-charts/helmfile.d/services/wikifunctions$ kube_env wikifunctions staging
jforrester@deploy2002:/srv/deployment-charts/helmfile.d/services/wikifunctions$ kubectl get pods -o wide | grep python
function-evaluator-python-evaluator-8b984d9f7-d9cz4 2/2 Running 0 38m 10.64.64.62 kubestage1006.eqiad.wmnet <none> <none>
jforrester@deploy2002:/srv/deployment-charts/helmfile.d/services/wikifunctions$ curl --silent 10.64.64.62:6927/_info | jq
{
"name": "function-evaluator",
"version": "0.0.1",
"description": "A Wikifunctions service to evaluate WikiLambda functions",
"home": "http://meta.wikimedia.org/wiki/Abstract%20Wikipedia",
"wasmProcessCount": 10
}jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/512
info: Return HTTP 503 when no WASM evaluators are ready
jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/512
info: Return HTTP 503 when no WASM evaluators are ready
Change #1271724 had a related patch set uploaded (by Jforrester; author: Jforrester):
[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2026-04-06-224243 to 2026-04-14-215402
Change #1271724 merged by jenkins-bot:
[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2026-04-06-224243 to 2026-04-14-215402
Change #1271915 had a related patch set uploaded (by Jforrester; author: Jforrester):
[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2026-04-06-224243 to 2026-04-14-215402
Change #1271915 merged by jenkins-bot:
[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2026-04-06-224243 to 2026-04-14-215402