Page MenuHomePhabricator
Create Task
Maniphest T400652

temporarily disable custom tracer logic
Closed, ResolvedPublic
Actions

Description

Let's temporarily remove the Span creation wrapper from the major functions in both services. This was deployed around the same time as when batching got turned on and we want clarity on what is pushing the Orchestrator over the ledge to OOM.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
wikifunctions: Upgrade orchestrator from 2025-08-13-113934 to 2025-08-20-210742operations/deployment-chartsmaster+1 -1
wikifunctions: Upgrade evaluators from 2025-08-05-075031 to 2025-08-20-203801operations/deployment-chartsmaster+2 -2
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
temporarily remove some custom span creationrepos/abstract-wiki/wikifunctions/function-orchestrator!420ecarggrace/T400652/temp-remove-custom-spansmain
temporarily remove custom span creationrepos/abstract-wiki/wikifunctions/function-evaluator!368ecarggrace/T400652/temp-remove-custom-spansmain
Customize query in GitLab

Related Objects

Event Timeline

ecarg created this task.Jul 28 2025, 8:52 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 28 2025, 8:52 PM
ecarg removed a subscriber: Aklapper.Jul 28 2025, 8:54 PM
Maintenance_bot added a project: Abstract Wikipedia team.Jul 28 2025, 9:31 PM
CodeReviewBot added a project: Patch-For-Review.Jul 28 2025, 10:30 PM

ecarg opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/368

temporarily remove custom span creation

CodeReviewBot added a comment.Jul 28 2025, 10:34 PM

ecarg opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/420

temporarily remove some custom span creation

Jdforrester-WMF subscribed.Jul 30 2025, 2:57 PM

OOMs seem to have calmed down, per Cory; do we want to still try this, or abandon?

Jdforrester-WMF changed the task status from Open to In Progress.Jul 30 2025, 3:41 PM
Jdforrester-WMF triaged this task as High priority.
Jdforrester-WMF edited projects, added Abstract Wikipedia team (26Q1 (Jul–Sep)); removed Abstract Wikipedia team.
Jdforrester-WMF moved this task from Incoming to In Engineering on the Abstract Wikipedia team (26Q1 (Jul–Sep)) board.
ecarg added a comment.Aug 2 2025, 1:18 AM

Yes still temp remove them!

CodeReviewBot added a comment.Aug 13 2025, 5:10 PM

apine merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/420

temporarily remove some custom span creation

CodeReviewBot added a comment.Aug 13 2025, 5:14 PM

jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/368

temporarily remove custom span creation

Maintenance_bot removed a project: Patch-For-Review.Aug 13 2025, 5:32 PM
gerritbot added a comment.Aug 20 2025, 8:59 PM

Change #1180651 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2025-08-05-075031 to 2025-08-20-203801

https://gerrit.wikimedia.org/r/1180651

gerritbot added a project: Patch-For-Review.Aug 20 2025, 8:59 PM

Change #1180652 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-08-13-113934 to 2025-08-20-182519

https://gerrit.wikimedia.org/r/1180652

gerritbot added a comment.Aug 20 2025, 9:10 PM

Change #1180651 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2025-08-05-075031 to 2025-08-20-203801

https://gerrit.wikimedia.org/r/1180651

gerritbot added a comment.Aug 20 2025, 9:22 PM

Change #1180652 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-08-13-113934 to 2025-08-20-210742

https://gerrit.wikimedia.org/r/1180652

Maintenance_bot removed a project: Patch-For-Review.Aug 20 2025, 9:31 PM
DSantamaria added a project: Essential-Work.Aug 26 2025, 9:11 AM
ecarg closed this task as Resolved.Sep 3 2025, 10:49 PM

Custom spans in the services are now turned off

Jdforrester-WMF mentioned this in rMSFEa07d3ebd6654: temporarily remove custom span creation.Oct 3 2025, 9:44 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits