Improve the logging we're doing in the orchestrator and evaluator to have a better idea of where the slowness is coming from
Open, In Progress, HighPublic
Actions

Assigned To

Authored By

	Jdforrester-WMF
	May 7 2024, 5:37 PM

Description

Our logging from the orchestrator and evaluators is not great.

Consistently use msg or message or whatever is WMF-standard and adjust log dashboards according
- Determine standard
- Adjust code to standard
- Deploy image with new code
- Adjust dashboard columns/filters, if needed
Have payload data actually show up, with the same things everywhere
- The x-request-id must be in every request-specific log message
- ???
???

Details

Subject	Repo	Branch	Lines +/-
Update function-schemata sub-module to HEAD (851131f)	mediawiki/extensions/WikiLambda	master	+1 -1
wikifunctions: Upgrade evaluators from 2024-06-11-161031 to 2024-07-03-153821	operations/deployment-charts	master	+2 -2
wikifunctions: Upgrade orchestrator from 2024-06-17-221517 to 2024-07-03-155425	operations/deployment-charts	master	+1 -1

Customize query in gerrit

Title	Reference	Author	Source Branch	Dest Branch
Update function-schemata sub-module to HEAD (851131f)	repos/abstract-wiki/wikifunctions/function-orchestrator!193	jforrester	sync-function-schemata	main
Update function-schemata sub-module to HEAD (851131f)	repos/abstract-wiki/wikifunctions/function-evaluator!227	jforrester	sync-function-schemata	main
Update function-schemata sub-module to HEAD (851131f)	repos/abstract-wiki/wikifunctions/wikilambda-cli!41	jforrester	sync-function-schemata	main
Refactor LoggerWrapper for better outputs to LogStash	repos/abstract-wiki/wikifunctions/function-schemata!127	ecarg	grace/T364413/update-logging-to-standard	main
Draft: Improve logging in evaluator	repos/abstract-wiki/wikifunctions/function-evaluator!226	ecarg	grace/T364413/update-logging-to-standard	main
Draft: Improve logging in orchestrator	repos/abstract-wiki/wikifunctions/function-orchestrator!187	ecarg	grace/T364413/update-logging-to-standard	main
Adjust logger to reflect MW standard	repos/abstract-wiki/wikifunctions/function-schemata!123	ecarg	grace/T364413/update-logging-to-standard	main
Draft: Adjust our logging to fit MW standard	repos/abstract-wiki/wikifunctions/function-orchestrator!186	ecarg	grace/T364413/update-logging-to-standard	main

Customize query in GitLab

Related Objects
Search...

Status	Assigned	Task
In Progress	ecarg	T364413 Improve the logging we're doing in the orchestrator and evaluator to have a better idea of where the slowness is coming from
In Progress	ecarg	T369001 Logging epic nice-to-have: Adding stack trace for every log output
In Progress	ecarg	T369213 Logging epic nice-to-have: add equivalent logs for docker logs

Event Timeline

Jdforrester-WMF created this task.May 7 2024, 5:37 PM

Jdforrester-WMF renamed this task from Improve the logging we're doing in the orchestrator and evaluator to have to Improve the logging we're doing in the orchestrator and evaluator to have a better idea of where the slowness is coming from.May 7 2024, 6:53 PM

Jdforrester-WMF removed Mcastro as the assignee of this task.May 8 2024, 4:34 PM

Jdforrester-WMF triaged this task as High priority.

Jdforrester-WMF moved this task from Incoming to Ready, Features on the Abstract Wikipedia team (24Q4 (Apr–Jun)) board.

Jdforrester-WMF added a subscriber: Mcastro.

Updates/notes:

Looks like 'standard' is message, according to: https://wikitech.wikimedia.org/wiki/Logstash/Common_Logging_Schema. However code-wide MW search shows WAY more examples of the message not being saved to a variable at all. Well, we can stick to message anyhow.
Confirmed by other MW engs:)

Personal Qs TODO:

Are we able to view THE logs (that we feed into our Logstash) in our local docker logs??

ecarg opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/186

Draft: Adjust our logging to fit MW standard

ecarg closed https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/186

Draft: Adjust our logging to fit MW standard

Maintenance_bot removed a project: Patch-For-Review.Mon, Jun 10, 6:31 PM

ecarg opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/123

Draft: Update logging

ecarg changed the task status from Open to In Progress.Mon, Jun 10, 6:43 PM

ecarg opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/187

Draft: Improve logging in orchestrator

Jdforrester-WMF assigned this task to ecarg.Tue, Jun 11, 1:57 PM

Jdforrester-WMF moved this task from Ready, Features to In Progress on the Abstract Wikipedia team (24Q4 (Apr–Jun)) board.

jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/123

Adjust logger to reflect MW standard

ecarg opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/226

Draft: Improve logging in evaluator

Jdforrester-WMF removed a parent task: T363387: [24Q4] Improve production monitoring of Wikifunctions services.Thu, Jun 20, 6:52 PM

Jdforrester-WMF moved this task from 24Q4 (Apr–Jun) to 25Q1 (Jul–Sep) on the Abstract Wikipedia team board.Tue, Jun 25, 6:15 PM

Jdforrester-WMF edited projects, added Abstract Wikipedia team (25Q1 (Jul–Sep)); removed Abstract Wikipedia team (24Q4 (Apr–Jun)).

Jdforrester-WMF moved this task from Incoming to In Progress on the Abstract Wikipedia team (25Q1 (Jul–Sep)) board.

Note/findings:

TIL there is a Beta LogStash:
Either way, level and severity levels are not aligned and contradict one another. We need to have both labels be identical. How is 'severity' being defined? I can't see this in Prod LogStash though, while in Beta, formatting is much clearer

ecarg opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/127

Draft: Refactor our logging wrapper

ecarg added a subtask: T369001: Logging epic nice-to-have: Adding stack trace for every log output.Tue, Jul 2, 6:30 AM

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/wikilambda-cli/-/merge_requests/41

Update function-schemata sub-module to HEAD (rOPUP851131f6e1c8)

Change #1051731 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (851131f)

https://gerrit.wikimedia.org/r/1051731

Change #1051807 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2024-06-17-221517 to 2024-07-03-155425

https://gerrit.wikimedia.org/r/1051807

Change #1051808 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2024-06-11-161031 to 2024-07-03-153821

https://gerrit.wikimedia.org/r/1051808

Change #1051807 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2024-06-17-221517 to 2024-07-03-155425

https://gerrit.wikimedia.org/r/1051807

Change #1051808 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2024-06-11-161031 to 2024-07-03-153821

https://gerrit.wikimedia.org/r/1051808

@ecarg: Heads-up that your change to the logger wrapper is now (finally!) deployed.

ecarg added a subtask: T369213: Logging epic nice-to-have: add equivalent logs for docker logs.Wed, Jul 3, 7:13 PM

ecarg changed the status of subtask T369001: Logging epic nice-to-have: Adding stack trace for every log output from Open to In Progress.

ecarg changed the status of subtask T369213: Logging epic nice-to-have: add equivalent logs for docker logs from Open to In Progress.Wed, Jul 3, 7:15 PM

dmartin merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/wikilambda-cli/-/merge_requests/41