Page MenuHomePhabricator

temporarily disable custom tracer logic
Closed, ResolvedPublic

Description

Let's temporarily remove the Span creation wrapper from the major functions in both services. This was deployed around the same time as when batching got turned on and we want clarity on what is pushing the Orchestrator over the ledge to OOM.

Details

Related Changes in Gerrit:
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
temporarily remove some custom span creationrepos/abstract-wiki/wikifunctions/function-orchestrator!420ecarggrace/T400652/temp-remove-custom-spansmain
temporarily remove custom span creationrepos/abstract-wiki/wikifunctions/function-evaluator!368ecarggrace/T400652/temp-remove-custom-spansmain
Customize query in GitLab

Event Timeline

OOMs seem to have calmed down, per Cory; do we want to still try this, or abandon?

Jdforrester-WMF changed the task status from Open to In Progress.Jul 30 2025, 3:41 PM
Jdforrester-WMF triaged this task as High priority.

Yes still temp remove them!

Change #1180651 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2025-08-05-075031 to 2025-08-20-203801

https://gerrit.wikimedia.org/r/1180651

Change #1180652 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-08-13-113934 to 2025-08-20-182519

https://gerrit.wikimedia.org/r/1180652

Change #1180651 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2025-08-05-075031 to 2025-08-20-203801

https://gerrit.wikimedia.org/r/1180651

Change #1180652 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-08-13-113934 to 2025-08-20-210742

https://gerrit.wikimedia.org/r/1180652

Custom spans in the services are now turned off