What/Why: req-ids in Rust Py Eval in Prod needs fixing
- request_id=unknown-request-id and span.id and trace.id's are null
- some logs show string placeholder, x-request-id: "request_id"
Description
Description
Details
Details
Related Changes in Gerrit:
Customize query in gerrit
Related Changes in GitLab:
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| rustversion: wire in request-id-middleware into router | repos/abstract-wiki/wikifunctions/function-evaluator!566 | ecarg | grace/T426792/register-reqId-middleware | main | |
| fix middleware to propagate request-ids properly | repos/abstract-wiki/rust-service-utils!5 | ecarg | grace/T426792/fix-request-ids | main |
Related Objects
Related Objects
Event Timeline
Comment Actions
ex:
kubectl logs function-evaluator-evaluator-rust-javascript-585d445b6-rvsk8
Defaulted container "function-evaluator-evaluator-rust-javascript" out of: function-evaluator-evaluator-rust-javascript, function-evaluator-evaluator-rust-javascript-tls-proxy
{"@timestamp":"2026-05-19T14:51:29.402416505+00:00","http.method":null,"http.status_code":null,"log.level":"INFO","message":"rust evaluator logger initialized","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T14:51:29.402569550+00:00","http.method":null,"http.status_code":null,"log.level":"INFO","message":"rust evaluator service starting","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T14:51:31.713495083+00:00","http.method":null,"http.status_code":null,"log.level":"ERROR","message":"\"Failed during the export process\"","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T21:28:36.908639960+00:00","http.method":null,"http.status_code":null,"log.level":"INFO","message":"received request for evaluation request_id=unknown-request-id","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
state machine log:
state machine log: calling execute in executor..., original time: 2026-05-19T21:28:36.910Z
state machine log:
state machine log:
state machine log: ...finished calling execute in executor, original time: 2026-05-19T21:28:36.911Z
state machine log:
{"@timestamp":"2026-05-19T21:28:41.711551961+00:00","http.method":null,"http.status_code":null,"log.level":"ERROR","message":"\"Failed during the export process\"","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
ecarg@deploy1003:~$ kubectl logs function-evaluator-evaluator-rust-python-75fcdc788-q882q
Defaulted container "function-evaluator-evaluator-rust-python" out of: function-evaluator-evaluator-rust-python, function-evaluator-evaluator-rust-python-tls-proxy
{"@timestamp":"2026-05-19T14:51:43.407903255+00:00","http.method":null,"http.status_code":null,"log.level":"INFO","message":"rust evaluator logger initialized","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T14:51:43.408053786+00:00","http.method":null,"http.status_code":null,"log.level":"INFO","message":"rust evaluator service starting","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T14:51:43.595171131+00:00","http.method":null,"http.status_code":null,"log.level":"ERROR","message":"\"Failed during the export process\"","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T14:51:44.286470712+00:00","http.method":null,"http.status_code":null,"log.level":"ERROR","message":"","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T14:51:48.595993929+00:00","http.method":null,"http.status_code":null,"log.level":"ERROR","message":"\"Failed during the export process\"","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T14:54:01.238355760+00:00","http.method":null,"http.status_code":null,"log.level":"INFO","message":"received request for evaluation request_id=unknown-request-id","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
state machine log: calling execute in executor..., original time: 2026-05-19T14:54:01.241032
state machine log: ...finished calling execute in executor, original time: 2026-05-19T14:54:01.243123
{"@timestamp":"2026-05-19T14:54:03.596314815+00:00","http.method":null,"http.status_code":null,"log.level":"ERROR","message":"\"Failed during the export process\"","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T21:37:56.136814572+00:00","http.method":null,"http.status_code":null,"log.level":"INFO","message":"received request for evaluation request_id=unknown-request-id","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
state machine log: calling execute in executor..., original time: 2026-05-19T21:37:56.139517
state machine log: ...finished calling execute in executor, original time: 2026-05-19T21:37:56.141278
{"@timestamp":"2026-05-19T21:37:58.595575687+00:00","http.method":null,"http.status_code":null,"log.level":"ERROR","message":"\"Failed during the export process\"","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}Comment Actions
Update: generateFunctionsMetrics: true is already set in staging, so x-request-id should be forwarded correctly once Orch is wired to route to the Rust evaluators. The evaluatorUri values in the staging Helm config currently point to localhost:6512/6513 (NodeJS Evals).
Comment Actions
Is this just the first half of T417870: Specify Rust and Node versions of the evaluators in prod and staging configs?
Comment Actions
sry I should have updated this and missed James's' comment! this was to test if request-ids come through properly after/when we first deployed to Staging. I don't think this is rly valid anymore bc we should test in Prod, which I was planning to do yesterday and didn't get to yet!
Comment Actions
ecarg opened https://gitlab.wikimedia.org/repos/abstract-wiki/rust-service-utils/-/merge_requests/5
Draft: fix middleware to propagate request-ids properly
Comment Actions
ecarg opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/566
Draft: rustversion: import + register reqId midware
Comment Actions
jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/rust-service-utils/-/merge_requests/5
fix middleware to propagate request-ids properly
Comment Actions
ecarg opened https://gitlab.wikimedia.org/repos/abstract-wiki/rust-service-utils/-/merge_requests/6
Draft: add changelog + bump vers
Comment Actions
jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/rust-service-utils/-/merge_requests/6
Release v0.2.1
Comment Actions
ecarg opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/567
update rust-service-utils vers
Comment Actions
jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/567
rustversion: Update rust-service-utils vers to v0.2.1
Comment Actions
jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/566
rustversion: wire in request-id-middleware into router
Comment Actions
Change #1303433 had a related patch set uploaded (by Jforrester; author: Jforrester):
[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2026-06-09-174730 to 2026-06-16-205705
Comment Actions
Change #1303433 merged by jenkins-bot:
[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2026-06-09-174730 to 2026-06-16-205705
Comment Actions
Change #1303482 had a related patch set uploaded (by Jforrester; author: Jforrester):
[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2026-06-09-174730 to 2026-06-17-154210
Comment Actions
Change #1303482 merged by jenkins-bot:
[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2026-06-09-174730 to 2026-06-17-184727
Comment Actions
Confirmed resolved in Prod:
- request_id=prod-test-1 ✅ (no longer unknown-request-id)
- x-request-id":"prod-test-1" ✅ (actual value, not literal string, "request_id")
- http.method":"POST" ✅
- http.status_code":"400" on "request complete" ✅
{"@timestamp":"2026-06-18T13:14:43.865963630+00:00","http.method":"POST","http.status_code":null,"log.level":"INFO","message":"received request for evaluation request_id=prod-test-1","service.name":"wikifunctions-evaluator-python","span.id":"5048d4f3914c59b5","trace.id":"340777a2807a373fe61d56b0f58457c8","x-request-id":"prod-test-1"}
{"@timestamp":"2026-06-18T13:14:43.866330015+00:00","http.method":"POST","http.status_code":null,"log.level":"WARN","message":"Returning HTTP status 400 due to error in Evaluator request_id=prod-test-1","service.name":"wikifunctions-evaluator-python","span.id":"5048d4f3914c59b5","trace.id":"340777a2807a373fe61d56b0f58457c8","x-request-id":"prod-test-1"}
{"@timestamp":"2026-06-18T13:14:43.866408675+00:00","http.method":"POST","http.status_code":"400","log.level":"INFO","message":"request complete","service.name":"wikifunctions-evaluator-python","span.id":"5048d4f3914c59b5","trace.id":"340777a2807a373fe61d56b0f58457c8","x-request-id":"prod-test-1"}