Page MenuHomePhabricator

Improve the logging we're doing in the orchestrator and evaluator to have a better idea of where the slowness is coming from
Closed, ResolvedPublic

Description

Our logging from the orchestrator and evaluators is not great.

  • Consistently use msg or message or whatever is WMF-standard and adjust log dashboards according
    • Determine standard
    • Adjust code to standard
    • Deploy image with new code
    • Adjust dashboard columns/filters, if needed
  • Have payload data actually show up, with the same things everywhere
    • The x-request-id must be in every request-specific log message
  • log API fetch boundaries with request IDs and timestamps
    • evaluator: request received, response sent
    • orchestrator: request received, response sent
    • orchestrator: request sent to evaluator, response received from evaluator
    • orchestrator: request sent to wikilambda_fetch, response received from wikilambda_fetch

Details

Related Changes in Gerrit:
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Update function-schemata sub-module to HEAD (18c7b06)repos/abstract-wiki/wikifunctions/wikilambda-cli!44jforrestersync-function-schematamain
Update function-schemata sub-module to HEAD (18c7b06)repos/abstract-wiki/wikifunctions/function-orchestrator!206jforrestersync-function-schematamain
Update function-schemata sub-module to HEAD (18c7b06)repos/abstract-wiki/wikifunctions/function-evaluator!238jforrestersync-function-schematamain
LoggerWrapper: Set x-request-id, not requestId, in log inforepos/abstract-wiki/wikifunctions/function-schemata!138jforresterlogger-x-request-idmain
Draft: [BREAKING CHANGE] Use new createZObjectKey.repos/abstract-wiki/wikifunctions/function-orchestrator!200apineapine-no-stable-stringifymain
Update function-schemata sub-module to HEAD (eb689ce)repos/abstract-wiki/wikifunctions/wikilambda-cli!43jforrestersync-function-schematamain
Update function-schemata sub-module to HEAD (eb689ce)repos/abstract-wiki/wikifunctions/function-evaluator!231jforrestersync-function-schematamain
Update function-schemata sub-module to HEAD (eb689ce)repos/abstract-wiki/wikifunctions/function-orchestrator!198jforrestersync-function-schematamain
Adjust existing logging to new changes in schemata/LoggerWrapperrepos/abstract-wiki/wikifunctions/function-evaluator!230ecarggrace/T364413/begin-update-logging-to-standardmain
Adjust logging to new LoggerWrapper changes in function-schematarepos/abstract-wiki/wikifunctions/function-orchestrator!197ecarggrace/T364413/begin-update-logging-to-standardmain
Update function-schemata sub-module to HEAD (851131f)repos/abstract-wiki/wikifunctions/function-orchestrator!193jforrestersync-function-schematamain
Update function-schemata sub-module to HEAD (851131f)repos/abstract-wiki/wikifunctions/function-evaluator!227jforrestersync-function-schematamain
Update function-schemata sub-module to HEAD (851131f)repos/abstract-wiki/wikifunctions/wikilambda-cli!41jforrestersync-function-schematamain
[BREAKING CHANGE] Refactor LoggerWrapper for better outputs to LogStashrepos/abstract-wiki/wikifunctions/function-schemata!127ecarggrace/T364413/update-logging-to-standardmain
Add new logs for debuggingrepos/abstract-wiki/wikifunctions/function-evaluator!226ecarggrace/T364413/update-logging-to-standardmain
Add new logs for debuggingrepos/abstract-wiki/wikifunctions/function-orchestrator!187ecarggrace/T364413/update-logging-to-standardmain
Adjust logger to reflect MW standardrepos/abstract-wiki/wikifunctions/function-schemata!123ecarggrace/T364413/update-logging-to-standardmain
Draft: Adjust our logging to fit MW standardrepos/abstract-wiki/wikifunctions/function-orchestrator!186ecarggrace/T364413/update-logging-to-standardmain
Show related patches Customize query in GitLab

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ecarg subscribed.

Updates/notes:

Personal Qs TODO:

  • Are we able to view THE logs (that we feed into our Logstash) in our local docker logs??
ecarg changed the task status from Open to In Progress.Jun 10 2024, 6:43 PM

Note/findings:

  • TIL there is a Beta LogStash:
  • Either way, level and severity levels are not aligned and contradict one another. We need to have both labels be identical. How is 'severity' being defined? I can't see this in Prod LogStash though, while in Beta, formatting is much clearer

Change #1051731 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (851131f)

https://gerrit.wikimedia.org/r/1051731

Change #1051807 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2024-06-17-221517 to 2024-07-03-155425

https://gerrit.wikimedia.org/r/1051807

Change #1051808 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2024-06-11-161031 to 2024-07-03-153821

https://gerrit.wikimedia.org/r/1051808

Change #1051807 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2024-06-17-221517 to 2024-07-03-155425

https://gerrit.wikimedia.org/r/1051807

Change #1051808 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2024-06-11-161031 to 2024-07-03-153821

https://gerrit.wikimedia.org/r/1051808

@ecarg: Heads-up that your change to the logger wrapper is now (finally!) deployed.

Change #1051731 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (851131f)

https://gerrit.wikimedia.org/r/1051731

Change #1053622 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (eb689ce)

https://gerrit.wikimedia.org/r/1053622

Change #1054886 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2024-07-09-155027 to 2024-07-17-140123

https://gerrit.wikimedia.org/r/1054886

Change #1054886 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2024-07-09-155027 to 2024-07-17-140123

https://gerrit.wikimedia.org/r/1054886

Change #1054909 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2024-07-09-154549 to 2024-07-17-145805

https://gerrit.wikimedia.org/r/1054909

Change #1053622 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (a0fd03c)

https://gerrit.wikimedia.org/r/1053622

Change #1054909 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2024-07-09-154549 to 2024-07-17-145805

https://gerrit.wikimedia.org/r/1054909

Change #1056264 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (18c7b06)

https://gerrit.wikimedia.org/r/1056264

Change #1056264 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (18e9ed3)

https://gerrit.wikimedia.org/r/1056264

Change #1064031 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2024-07-19-164024 to 2024-08-13-135124

https://gerrit.wikimedia.org/r/1064031

Change #1064032 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2024-07-23-225548 to 2024-08-16-153209

https://gerrit.wikimedia.org/r/1064032

Change #1064031 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2024-07-19-164024 to 2024-08-13-135124

https://gerrit.wikimedia.org/r/1064031

Change #1064032 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2024-07-23-225548 to 2024-08-16-153209

https://gerrit.wikimedia.org/r/1064032

Jdforrester-WMF updated the task description. (Show Details)