Findings so far:

• In /function-orchestrator/app.js, the system doesn't like req, res, or next, the params in app.all(). Anytime I try to log them, the logs get held up.
• Therefore or also, we never get inside app.all( '*', ( req, res, next ) => { ... which calls sUtil.initAndLogRequest( req, app ); that calls the metrics that are currently not being emitted

In /function-orchestrator/app.js, the system doesn't like req, res, or next, the params in app.all(). Anytime I try to log them, the logs get held up.

Do you have a branch where we can see this? I'll be happy to investigate if you can show me how to repro locally.

ecarg opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/181

Draft: Prod monitoring: debugging orch metrics

Apologies, I did not notice this comment last wk T_T!

I'll just push up all of the console.log's I have atm here: https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/181

Update: successful 🍐 w/ James today. Moving the two metrics out of the unused function,req.issueRequest = async ( request ) => {... solves this bug. If I observe which metrics were triggered locally, I can see that all four have been:

...
function_orchestrator_router_request_duration_seconds_count{service="function-orchestrator",path="--domain/v1/evaluate",method="POST",status="200"} 1

# HELP function_orchestrator_incomingrequestcount Wikifunctions function-orchestrator request count
# TYPE function_orchestrator_incomingrequestcount counter
function_orchestrator_incomingrequestcount 1

# HELP function_orchestrator_nonrequesterror Wikifunctions function-orchestrator non-request error count
# TYPE function_orchestrator_nonrequesterror counter
function_orchestrator_nonrequesterror 1

# HELP function_orchestrator_outgoingresponsecount Wikifunctions function-orchestrator response count
# TYPE function_orchestrator_outgoingresponsecount counter
function_orchestrator_outgoingresponsecount 1

jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/181

Prod monitoring: fix so we can emit all four metrics

Merged and available in image 2024-06-06-134535 and later. General plan is to deploy on Wednesdays in our regular slot – think it's OK to wait here?

Change #1042269 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2024-06-05-003919 to 2024-06-11-223956

https://gerrit.wikimedia.org/r/1042269

Change #1042269 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2024-06-05-003919 to 2024-06-11-223956

https://gerrit.wikimedia.org/r/1042269