What/Why:
Logs in Logstash recently show that the logs we've been adding to the Evaluator and Orchestrator last couple quarters are not coming through. The only logs we see are the response messages.
How:
[edit] Yes, related issue.
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| increase log sev levels since change to ecs | repos/abstract-wiki/wikifunctions/function-orchestrator!281 | ecarg | grace/T383927/increase-severity-for-lower-level-logs-no-longer-coming-thru | main | |
| adjust log severity levels to see if they show up | repos/abstract-wiki/wikifunctions/function-evaluator!309 | ecarg | grace/T383927/change-log-levels-for-winston | main | |
| adjust log severity levels to see if they show up | repos/abstract-wiki/wikifunctions/function-orchestrator!277 | ecarg | grace/T383927/change-log-levels-for-winston | main | |
| upgrade service-utils version for ecs compatability | repos/abstract-wiki/wikifunctions/function-orchestrator!273 | ecarg | grace/T383927/service-utils-update-ecs-formatting | main | |
| fix winston error | repos/abstract-wiki/wikifunctions/function-orchestrator!271 | ecarg | grace/T383927/fix-orchestrator-logs | main |
notes:
End of the year before code freeze
ecarg opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/271
fix winston error
jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/271
fix winston error
ecarg opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/273
upgrade service-utils version for ecs compatability
jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/273
upgrade service-utils version for ecs compatability
Change #1113571 had a related patch set uploaded (by Jforrester; author: Jforrester):
[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-01-15-052609 to 2025-01-22-203140
Change #1113571 merged by jenkins-bot:
[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-01-15-052609 to 2025-01-22-203140
ecarg opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/277
adjust log severity levels to see if they show up
ecarg opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/309
adjust log severity levels to see if they show up
notes:
do we need to adjust our main dashboard to see the logs we see here on ecs? hm
jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/277
adjust log severity levels to see if they show up
jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/309
adjust log severity levels to see if they show up
Change #1115028 had a related patch set uploaded (by Jforrester; author: Jforrester):
[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-01-22-203140 to 2025-01-28-144249
Change #1115029 had a related patch set uploaded (by Jforrester; author: Jforrester):
[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2025-01-22-212306 to 2025-01-29-140344
Change #1115032 had a related patch set uploaded (by Cory Massaro; author: Cory Massaro):
[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from version: 2025-01-22-203140 to 2025-01-28-144249
Change #1115032 abandoned by Cory Massaro:
[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from version: 2025-01-22-203140 to 2025-01-28-144249
Reason:
already done
Change #1115028 merged by jenkins-bot:
[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-01-22-203140 to 2025-01-28-144249
Change #1115029 merged by jenkins-bot:
[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2025-01-22-212306 to 2025-01-29-140344
This has been solved now! https://logstash.wikimedia.org/goto/ffe503957373208394c62be9cfe60fea
However directs us to another task because turns out our Logstash is using legacy visualizations, not ecs-compliant. We need to update our board or something.
ecarg opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/281
increase log sev levels since change to ecs
dmartin merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/281
increase log sev levels since change to ecs
Hey @ecarg
Are we seeing the missing logs now? If so, should we proceed to "Resolve" this?
Change #1119094 had a related patch set uploaded (by Cory Massaro; author: Cory Massaro):
[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-02-03-215824 to 2025-02-11-155417.
Change #1119094 merged by jenkins-bot:
[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-02-03-215824 to 2025-02-11-155417.
Change #1119191 had a related patch set uploaded (by Jforrester; author: Jforrester):
[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-02-03-215824 to 2025-02-12-171406
Change #1119191 merged by jenkins-bot:
[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-02-03-215824 to 2025-02-12-171406