What/Why:
We see this error Call tuples failed in returnOnFirstError. Error: TypeError: Cannot read properties of null (reading 'Z9K1')., time: 2024-10-18T04:20:25.238Z, reqId: 80f5f313-2004-95fb-835f-0be9e390ba8e, trace: at orchestrate (/srv/service/src/orchestrate.js:85:21) on a very regular basis on our dashboard. We should resolve this non-fatal error that causes so much noise.
How:
We're also getting argumentDict.argument.asJSON is not a function, e.g. around 11:53 on 2025-02-14: https://logstash.wikimedia.org/app/discover#/doc/0fade920-6712-11eb-8327-370b46f9e7a5/ecs-k8s-1-1.11.0-7-2025.07?id=KBNOBJUB1cXMpx2miMja
{
"_index": "ecs-k8s-1-1.11.0-7-2025.07",
"_id": "KBNOBJUB1cXMpx2miMja",
"_score": 1,
"_source": {
"labels": {
"container_name": "function-orchestrator-main-orchestrator",
"release": "main-orchestrator",
"deployment": "function-orchestrator"
},
"service": {
"name": "function-orchestrator",
"type": "input-file-kubernetes"
},
"orchestrator": {
"type": "kubernetes",
"cluster": {
"url": "https://kubemaster.svc.codfw.wmnet:6443"
},
"resource": {
"name": "function-orchestrator-main-orchestrator-74d8d4c66c-md6tk"
},
"namespace": "wikifunctions"
},
"message": "function implementation: Z903 error: TypeError: argumentDict.argument.asJSON is not a function",
"@timestamp": "2025-02-14T11:53:30.379Z",
"@version": "1",
"host": {
"name": "wikikube-worker2238"
},
"ecs": {
"version": "8.10.0"
},
"normalized": {
"message": "function implementation: z903 error: typeerror: argumentdict.argument.asjson is not a function",
"dropped": {
"no_such_field": [
"requestId",
"rsyslog",
"kubernetes",
"metadata",
"stream",
"program",
"docker",
"logsource",
"time",
"timestamp",
"normalized_message"
]
}
},
"log": {
"level": "ERROR",
"syslog": {
"facility": {
"code": 16,
"name": "local0"
},
"severity": {
"code": 3,
"name": "error"
},
"priority": 131
}
},
"tags": [
"input-kafka-k8s",
"rsyslog-shipper",
"kafka",
"es",
"k8s_containerd_msg_field_parsed",
"k8s_docker_log_field_parsed",
"normalized_message_untrimmed"
]
},
"fields": {
"@timestamp": [
"2025-02-14T11:53:30.379Z"
]
}
}This co-occurs with the Cannot read properties of null error. Taken together, these issues might point to an un-normalized Z9 being passed around, or null being passed around where we expect a Z9.