Page MenuHomePhabricator
Create Task
Maniphest T406976

Z28602 often, but not always, fails with a Z570
Closed, ResolvedPublicBUG REPORT
Actions

Description

Steps to replicate the issue (include links if applicable):

What happens?:
It fails with a Z570/Reached rate limit in orchestrator with orchestrator rate limit: "600"

It doesn't do so consistently for the same noun. Sometimes a noun passes, sometimes it doesn't (and on repeat, I hit the cache).

What should have happened instead?:
Hopefully pass and show the HTML

Software version (on Special:Version page; skip for WMF-hosted wikis like Wikipedia):

Other information (browser name/version, screenshots, etc.):

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
wikifunctions: Upgrade orchestrator from 2025-10-28-205854 to 2025-11-04-215809operations/deployment-chartsmaster+1 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedcmassaroT406346 [26Q2] Orchestrator memory squashing
 ResolvedBUG REPORTcmassaroT406976 Z28602 often, but not always, fails with a Z570

Event Timeline

DVrandecic created this task.Oct 10 2025, 11:32 AM
Jdforrester-WMF triaged this task as High priority.Oct 15 2025, 5:50 PM
Jdforrester-WMF added a parent task: T406346: [26Q2] Orchestrator memory squashing.
Jdforrester-WMF edited projects, added Abstract Wikipedia team (26Q2 (Oct–Dec)); removed Abstract Wikipedia team.
Jdforrester-WMF added a project: OKR-Work.
Jdforrester-WMF moved this task from Incoming to Ready on the Abstract Wikipedia team (26Q2 (Oct–Dec)) board.
cmassaro subscribed.Oct 15 2025, 5:52 PM

This is being debugged by committee. I'm assigning it to myself, but it's actually a group effort among several of us.

cmassaro claimed this task.Oct 15 2025, 5:52 PM
cmassaro moved this task from Ready to In Engineering on the Abstract Wikipedia team (26Q2 (Oct–Dec)) board.
DVrandecic added a comment.Oct 18 2025, 4:26 PM

I hope I am not messing up the debugging effort, but I created a code implementation ( Z28931 ) that works much faster than the previous implementation ( Z28604 ).

The mentioned error is still reproducible on implementation Z28604

https://www.wikifunctions.org/view/en/Z28604

CodeReviewBot added a project: Patch-For-Review.Oct 30 2025, 8:46 PM

apine opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/475

Make maximum simultaneous executions controllable by an environment variable.

CodeReviewBot added a comment.Nov 3 2025, 5:07 PM

ecarg merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/475

Make maximum simultaneous executions controllable by an environment variable.

Maintenance_bot removed a project: Patch-For-Review.Nov 3 2025, 5:31 PM
gerritbot added a comment.Nov 5 2025, 2:25 PM

Change #1202149 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-10-28-205854 to 2025-11-04-215809

https://gerrit.wikimedia.org/r/1202149

gerritbot added a project: Patch-For-Review.Nov 5 2025, 2:25 PM
gerritbot added a comment.Nov 5 2025, 3:12 PM

Change #1202149 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-10-28-205854 to 2025-11-04-215809

https://gerrit.wikimedia.org/r/1202149

Maintenance_bot removed a project: Patch-For-Review.Nov 5 2025, 3:31 PM
Jdforrester-WMF subscribed.Nov 5 2025, 4:01 PM

After the deploy, I've just triggered Z28602 for Herz in ~6 rare languages in a row, and all worked. Is this just happenstance/luck, or is this ameliorated/fixed?

DSantamaria changed the task status from Open to In Progress.Nov 12 2025, 7:00 AM
ecarg closed this task as Resolved.Nov 19 2025, 11:35 PM
ecarg subscribed.

Resolving now since:

  • original implementation has changed
  • maxSimultaneousExecutions has been configured and increased

(Note: we are still using this implementation for local/debug discovery purposes!)

Jdforrester-WMF moved this task from In Engineering to Needs Sign-off on the Abstract Wikipedia team (26Q2 (Oct–Dec)) board.Nov 20 2025, 3:02 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits