Orchestrator tries to run Z981 as an evaluated implementation
Open, HighPublicBUG REPORT
Actions

Assigned To

None

Authored By

	cmassaro
	Mar 25 2025, 2:15 PM

Description

Steps to reproduce (step by step instructions, with links, commands and necessary data to reproduce the error)

look here

Observed behavior

an error comes from the Evaluated implementation (src/implementation.js)

Expected behavior/Acceptance criteria (returned value, expected error, performance expectations, etc.)

Z981 is a built-in implementation and should never be run as an evaluated implementation

Completion checklist

Before closing this task, review one by one the checklist available here: https://www.mediawiki.org/wiki/Abstract_Wikipedia_team/Definition_of_Done#Back-end_Task/Bug_completion_checklist

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	wikifunctions: Update orchestrator from 2025-04-02-124609 to 2025-04-08-183631	operations/deployment-charts	master	+1 -1

Customize query in gerrit

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	Add logging to figure out why Z981 gets run as evaluated.	repos/abstract-wiki/wikifunctions/function-orchestrator!338	apine	apine-why-why	main

Customize query in GitLab

Event Timeline

cmassaro created this task.Mar 25 2025, 2:15 PM

Jdforrester-WMF awarded a token.Mar 25 2025, 2:22 PM

Which implementation is this arising from? (Sorry I don't have access to the logs linked.)

Jdforrester-WMF triaged this task as High priority.Mar 26 2025, 4:53 PM

Jdforrester-WMF moved this task from To Triage to Bugs to investigate on the Abstract Wikipedia team board.

It's not clear yet. It claims to be Z981, but that makes no sense. We'll hopefully be able to repro this tomorrow.

This is now affecting Z201 and Z6921, as well: https://logstash.wikimedia.org/app/dashboards#/view/1ed06310-decd-11ef-859d-f36bcd343d41?_g=h@dcc6a94&_a=h@8778ca4 (checked 20250401, ~17:00 UTC)

apine opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/338

Add logging to figure out why Z981 gets run as evaluated.

jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/338

Add logging to figure out why Z981 gets run as evaluated.

Maintenance_bot removed a project: Patch-For-Review.Apr 8 2025, 3:16 PM

Change #1135424 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Update orchestrator from 2025-04-02-124609 to 2025-04-08-183631

https://gerrit.wikimedia.org/r/1135424

gerritbot added a project: Patch-For-Review.Apr 9 2025, 1:01 PM

Change #1135424 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Update orchestrator from 2025-04-02-124609 to 2025-04-08-183631

https://gerrit.wikimedia.org/r/1135424

Now deployed. Does this give us a sign of what's broken?

Maintenance_bot removed a project: Patch-For-Review.Apr 9 2025, 2:31 PM

Here's a link to logstash to see messages from MR338:

https://logstash.wikimedia.org/goto/0983492220a67e2e4cb0049ffa8bdb26

At this time I see 398 occurrences for the last 24 hours. They indicate calls to several different functions, including: Z18475, Z24207, Z12681, Z13224, Z13366, Z12767, Z12668.

Here are 2 examples:

Evaluated function failed for {"Z1K1":{"Z1K1":"Z9","Z9K1":"Z7"},"Z7K1":{"Z1K1":"Z9","Z9K1":"Z18475"},"Z18475K1":{"Z1K1":{"Z1K1":{"Z1K1":"Z9","Z9K1":"Z7"},"Z7K1":{"Z1K1":"Z9","Z9K1":"Z881"},"Z881K1":{"Z1K1":"Z9","Z9K1":"Z1"}},"K1":{"Z1K1":{"Z1K1":"Z9","Z9K1":"Z6091"},"Z6091K1":{"Z1K1":"Z6","Z6K1":"Q146786"}},"K2":{"Z1K1":{"Z1K1":{"Z1K1":"Z9","Z9K1":"Z7"},"Z7K1":{"Z1K1":"Z9","Z9K1":"Z881"},"Z881K1":{"Z1K1":"Z9","Z9K1":"Z1"}}}}}

evaluated function failed for {"z1k1":{"z1k1":"z9","z9k1":"z7"},"z7k1":{"z1k1":"z9","z9k1":"z12755"},"z12755k1":{"z1k1":{"z1k1":"z9","z9k1":"z18"},"z18k1":{"z1k1":"z6","z6k1":"z14779k2"}}}

There are also some cases where the function called is spelled out as a Z8 literal.

Orchestrator tries to run Z981 as an evaluated implementationOpen, HighPublicBUG REPORTActions

Description

Description

Completion checklist

Details

Event Timeline

Orchestrator tries to run Z981 as an evaluated implementation
Open, HighPublicBUG REPORT
Actions