Page MenuHomePhabricator
Create Task
Maniphest T423693

function-orchestrator: TypeError: Cannot read properties of undefined (reading 'z1k1')
Open, Needs TriagePublic
Actions

Description

Log error observed in the function-orchestrator service.

Error: TypeError: Cannot read properties of undefined (reading 'z1k1')

Request ID: af1ff260-7f80-4229-b35c-ca27010947d8

Logstash: https://logstash.wikimedia.org/app/dashboards#/view/1ed06310-decd-11ef-859d-f36bcd343d41?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1d,to:now))&_a=(filters:!(('$state':(store:appState),meta:(key:orchestrator.namespace,params:(query:wikifunctions)),query:(match_phrase:(orchestrator.namespace:wikifunctions))),('$state':(store:appState),meta:(key:labels.deployment,params:(query:function-orchestrator)),query:(match_phrase:(labels.deployment:function-orchestrator))),('$state':(store:appState),meta:(key:http.request.id,params:(query:af1ff260-7f80-4229-b35c-ca27010947d8)),query:(match_phrase:(http.request.id:af1ff260-7f80-4229-b35c-ca27010947d8)))),title:'wikifunctions%20backend',viewMode:view)

Function call that triggered the error:

json
{
  "Z1K1": "Z7",
  "Z7K1": "Z808",
  "Z808K1": {
    "Z1K1": "Z7",
    "Z7K1": {
      "Z1K1": "Z8",
      "Z8K1": ["Z17", {
        "Z1K1": "Z17",
        "Z17K1": "Z1",
        "Z17K2": "Z30617K1",
        "Z17K3": {"Z1K1": "Z12", "Z12K1": ["Z11", {"Z1K1": "Z11", "Z11K1": "Z1002", "Z11K2": "reference"}]}
      }],
      "Z8K2": {"Z1K1": "Z7", "Z7K1": "Z881", "Z881K1": {"Z1K1": "Z7", "Z7K1": "Z882", "Z882K1": "Z39", "Z882K2": "Z1"}},
      "Z8K3": ["Z20", "Z30618", "Z30630"],
      "Z8K4": ["Z14", "Z30619"],
      "Z8K5": "Z30617"
    },
    "Z30617K1": "Z17404"
  }
}

Related Objects

Event Timeline

DSmit-WMF created this task.Apr 17 2026, 11:08 AM
DMartin-WMF subscribed.Apr 23 2026, 6:06 AM

Another recent case of the same failure was recorded in T423960, which I am closing. Here is the request from that ticket:

request: {"Z1K1":"Z7","Z7K1":{"Z1K1":"Z8","Z8K1":["Z17",{"Z1K1":"Z17","Z17K1":"Z6","Z17K2":"Z11602K1","Z17K3":{"Z1K1":"Z12","Z12K1":["Z11",{"Z1K1":"Z11","Z11K1":"Z1002","Z11K2":"value"},{"Z1K1":"Z11","Z11K1":"Z1014","Z11K2":"Uru"},{"Z1K1":"Z11","Z11K1":"Z1430","Z11K2":"Wert"}]}}],"Z8K2":"Z6","Z8K3":["Z20","Z11604","Z11605","Z11606"],"Z8K4":["Z14","Z32142"],"Z8K5":"Z11602"},"Z11602K1":""}

DMartin-WMF merged a task: T423960: [v2] 'Orchestration failed with...' errors.Apr 23 2026, 6:24 AM
DMartin-WMF added subscribers: ecarg, Aklapper.
DSmit-WMF added a comment.Apr 23 2026, 8:11 AM

I ran this through the profiler (the first one in the main ticket description) but didn't get the z1k1 error?

{
	"Z1K1": "Z22",
	"Z22K1": "Z24",
	"Z22K2": {
		"Z1K1": {
			"Z1K1": "Z7",
			"Z7K1": "Z883",
			"Z883K1": "Z6",
			"Z883K2": "Z1"
		},
		"K1": [{
			"Z1K1": "Z7",
			"Z7K1": "Z882",
			"Z882K1": "Z6",
			"Z882K2": "Z1"
		}, {
			"Z1K1": {
				"Z1K1": "Z7",
				"Z7K1": "Z882",
				"Z882K1": "Z6",
				"Z882K2": "Z1"
			},
			"K1": "errors",
			"K2": {
				"Z1K1": "Z5",
				"Z5K1": "Z511",
				"Z5K2": {
					"Z1K1": {
						"Z1K1": "Z7",
						"Z7K1": "Z885",
						"Z885K1": "Z511"
					},
					"Z511K1": {
						"Z1K1": "Z39",
						"Z39K1": "Z39K1"
					},
					"Z511K2": {
						"Z1K1": "Z99",
						"Z99K1": {
							"Z1K1": {
								"Z1K1": "Z9",
								"Z9K1": "Z17402"
							},
							"Z17402K1": {
								"Z1K1": "Z9",
								"Z9K1": "Z17404"
							}
						}
					}
				}
			}
		}, {
			"Z1K1": {
				"Z1K1": "Z7",
				"Z7K1": "Z882",
				"Z882K1": "Z6",
				"Z882K2": "Z1"
			},
			"K1": "implementationType",
			"K2": "Z14K2"
		}, {
			"Z1K1": {
				"Z1K1": "Z7",
				"Z7K1": "Z882",
				"Z882K1": "Z6",
				"Z882K2": "Z1"
			},
			"K1": "implementationId",
			"K2": {
				"Z1K1": "Z6",
				"Z6K1": "Z30619"
			}
		}, {
			"Z1K1": {
				"Z1K1": "Z7",
				"Z7K1": "Z882",
				"Z882K1": "Z6",
				"Z882K2": "Z1"
			},
			"K1": "orchestrationMemoryUsage",
			"K2": "157.63 MiB"
		}, {
			"Z1K1": {
				"Z1K1": "Z7",
				"Z7K1": "Z882",
				"Z882K1": "Z6",
				"Z882K2": "Z1"
			},
			"K1": "orchestrationCpuUsage",
			"K2": "158.011 ms"
		}, {
			"Z1K1": {
				"Z1K1": "Z7",
				"Z7K1": "Z882",
				"Z882K1": "Z6",
				"Z882K2": "Z1"
			},
			"K1": "orchestrationStartTime",
			"K2": "2026-04-23T08:00:33.189Z"
		}, {
			"Z1K1": {
				"Z1K1": "Z7",
				"Z7K1": "Z882",
				"Z882K1": "Z6",
				"Z882K2": "Z1"
			},
			"K1": "orchestrationEndTime",
			"K2": "2026-04-23T08:00:33.467Z"
		}, {
			"Z1K1": {
				"Z1K1": "Z7",
				"Z7K1": "Z882",
				"Z882K1": "Z6",
				"Z882K2": "Z1"
			},
			"K1": "orchestrationDuration",
			"K2": "278 ms"
		}, {
			"Z1K1": {
				"Z1K1": "Z7",
				"Z7K1": "Z882",
				"Z882K1": "Z6",
				"Z882K2": "Z1"
			},
			"K1": "orchestrationHostname",
			"K2": "6b129926c519"
		}]
	}
}

I also tried to run the one @DMartin-WMF mentioned and there I do See the error:

{
	"Z1K1": "Z22",
	"Z22K1": "Z24",
	"Z22K2": {
		"Z1K1": {
			"Z1K1": "Z7",
			"Z7K1": "Z883",
			"Z883K1": "Z6",
			"Z883K2": "Z1"
		},
		"K1": [{
			"Z1K1": "Z7",
			"Z7K1": "Z882",
			"Z882K1": "Z6",
			"Z882K2": "Z1"
		}, {
			"Z1K1": {
				"Z1K1": "Z7",
				"Z7K1": "Z882",
				"Z882K1": "Z6",
				"Z882K2": "Z1"
			},
			"K1": "errors",
			"K2": {
				"Z1K1": "Z5",
				"Z5K1": "Z507",
				"Z5K2": {
					"Z1K1": {
						"Z1K1": "Z7",
						"Z7K1": "Z885",
						"Z885K1": "Z507"
					},
					"Z507K1": {
						"Z1K1": "Z99",
						"Z99K1": {
							"Z1K1": "Z7",
							"Z7K1": {
								"Z1K1": "Z8",
								"Z8K1": ["Z17", {
									"Z1K1": "Z17",
									"Z17K1": "Z6",
									"Z17K2": "Z11602K1",
									"Z17K3": {
										"Z1K1": "Z12",
										"Z12K1": ["Z11", {
											"Z1K1": "Z11",
											"Z11K1": "Z1002",
											"Z11K2": "value"
										}, {
											"Z1K1": "Z11",
											"Z11K1": "Z1014",
											"Z11K2": "Uru"
										}, {
											"Z1K1": "Z11",
											"Z11K1": "Z1430",
											"Z11K2": "Wert"
										}]
									}
								}],
								"Z8K2": "Z6",
								"Z8K3": ["Z20", "Z11604", "Z11605", "Z11606"],
								"Z8K4": ["Z14", "Z32142"],
								"Z8K5": "Z11602"
							},
							"Z11602K1": ""
						}
					},
					"Z507K2": {
						"Z1K1": "Z5",
						"Z5K1": "Z500",
						"Z5K2": {
							"Z1K1": {
								"Z1K1": "Z7",
								"Z7K1": "Z885",
								"Z885K1": "Z500"
							},
							"Z500K1": "Orchestration failed with unidentified error: TypeError: Cannot read properties of undefined (reading 'Z1K1')."
						}
					}
				}
			}
		}, {
			"Z1K1": {
				"Z1K1": "Z7",
				"Z7K1": "Z882",
				"Z882K1": "Z6",
				"Z882K2": "Z1"
			},
			"K1": "implementationType",
			"K2": "Z14K2"
		}, {
			"Z1K1": {
				"Z1K1": "Z7",
				"Z7K1": "Z882",
				"Z882K1": "Z6",
				"Z882K2": "Z1"
			},
			"K1": "implementationId",
			"K2": {
				"Z1K1": "Z6",
				"Z6K1": "Z32142"
			}
		}, {
			"Z1K1": {
				"Z1K1": "Z7",
				"Z7K1": "Z882",
				"Z882K1": "Z6",
				"Z882K2": "Z1"
			},
			"K1": "orchestrationMemoryUsage",
			"K2": "157.72 MiB"
		}, {
			"Z1K1": {
				"Z1K1": "Z7",
				"Z7K1": "Z882",
				"Z882K1": "Z6",
				"Z882K2": "Z1"
			},
			"K1": "orchestrationCpuUsage",
			"K2": "14.367 ms"
		}, {
			"Z1K1": {
				"Z1K1": "Z7",
				"Z7K1": "Z882",
				"Z882K1": "Z6",
				"Z882K2": "Z1"
			},
			"K1": "orchestrationStartTime",
			"K2": "2026-04-23T08:03:42.608Z"
		}, {
			"Z1K1": {
				"Z1K1": "Z7",
				"Z7K1": "Z882",
				"Z882K1": "Z6",
				"Z882K2": "Z1"
			},
			"K1": "orchestrationEndTime",
			"K2": "2026-04-23T08:03:42.650Z"
		}, {
			"Z1K1": {
				"Z1K1": "Z7",
				"Z7K1": "Z882",
				"Z882K1": "Z6",
				"Z882K2": "Z1"
			},
			"K1": "orchestrationDuration",
			"K2": "42 ms"
		}, {
			"Z1K1": {
				"Z1K1": "Z7",
				"Z7K1": "Z882",
				"Z882K1": "Z6",
				"Z882K2": "Z1"
			},
			"K1": "orchestrationHostname",
			"K2": "6b129926c519"
		}]
	}
}
DSmit-WMF added a comment.Apr 23 2026, 8:13 AM

The fact that this again contains a Z17 makes me think might again be related to https://phabricator.wikimedia.org/T423853

DSmit-WMF added a comment.Apr 23 2026, 8:36 AM

Did some debugging: WFBase.from() in orchestrator code receives an undefined zobject param.
here this.Z1K1 is undefined: WfObject.js -> let fullType = await WFBase.from( this.Z1K1, this.wfConfig );

DSmit-WMF moved this task from To Triage to Bugs to investigate on the Abstract Wikipedia team board.Apr 23 2026, 8:44 AM
DSmit-WMF added a subscriber: DSantamaria.
cmassaro claimed this task.Jun 3 2026, 4:04 PM
cmassaro added a comment.Jun 3 2026, 4:07 PM

This falls into the same bucket as https://phabricator.wikimedia.org/T423694 : content issue in a nested function is not being propagated correctly.

In this case, I believe the issue is Z30619. That function's tests are failing. I think we need some kind of general "add stack information" task to handle these kinds of issues.

cmassaro mentioned this in T416142: Consider adding stack trace to all Z5s, other kinds of nested metadata.Jun 15 2026, 1:27 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits