Page MenuHomePhabricator

Scopes Are Not Always Attached to ZObjects Correctly
Closed, ResolvedPublicBUG REPORT

Description

Description

Steps to reproduce (step by step instructions, with links, commands and necessary data to reproduce the error)

Here is a test function call. DO NOT RUN IT ON BETA CLUSTER OR IN CI:

{
	"Z1K1": "Z7",
	"Z7K1": {
		"Z1K1": "Z8",
		"Z8K1": [
			"Z17",
			{
				"Z1K1": "Z17",
				"Z17K1": "Z8",
				"Z17K2": "Z10043K1",
				"Z17K3": {
					"Z1K1": "Z12",
					"Z12K1": [
						"Z11"
					]
				}
			}
		],
		"Z8K2": "Z1",
		"Z8K3": [
			"Z20"
		],
		"Z8K4": [
			"Z14",
			{
				"Z1K1": "Z14",
				"Z14K1": "Z10043",
				"Z14K2": {
					"Z1K1": "Z7",
					"Z7K1": {
						"Z1K1": "Z18",
						"Z18K1": "Z10043K1"
					},
					"K1": {
						"Z1K1": "Z18",
						"Z18K1": "Z10043K1"
					}
				}
			}
		],
		"Z8K5": "Z10043"
	},
	"Z10043K1": {
		"Z1K1": "Z8",
		"Z8K1": [
			"Z17",
			{
				"Z1K1": "Z17",
				"Z17K1": "Z8",
				"Z17K2": "Z10044K1",
				"Z17K3": {
					"Z1K1": "Z12",
					"Z12K1": [
						"Z11"
					]
				}
			}
		],
		"Z8K2": "Z1",
		"Z8K3": [
			"Z20"
		],
		"Z8K4": [
			"Z14",
			{
				"Z1K1": "Z14",
				"Z14K1": "Z10044",
				"Z14K2": {
					"Z1K1": "Z7",
					"Z7K1": {
						"Z1K1": "Z18",
						"Z18K1": "Z10044K1"
					},
					"K1": {
						"Z1K1": "Z18",
						"Z18K1": "Z10044K1"
					}
				}
			}
		],
		"Z8K5": "Z10044"
	}
}

This is similar to the following self-recursive Python code:

(lambda function(function))(lambda function(function))

SO:

  1. in function-orchestrator:src/frame.js::Frame::constructor(), add this.size_ the number of frames (i.e., this.size_ = this.lastFrame_ === null ? 0 : this.lastFrame_.size_ + 1)
  2. add console.log(this.size_)
  3. spin up a local development environment
  4. run the above self-recursive code IN THE LOCAL DEVELOPMENT ENVIRONMENT

Observed behavior

  • your environment will blow up
  • the function-orchestrator logs will show that the stack size did not grow unbounded but remain around 0/1
  • this is probably because the new scopes are not being attached properly
    • probably, when the Z7 compositions are executed as functions, their Z7K1 Functions keep the enclosing scope, rather than the new scope

Expected behavior/Acceptance criteria (returned value, expected error, performance expectations, etc.)

  • the stack size should have grown!

Completion checklist

Details

Related Changes in Gerrit:
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Scopes attachment fix: add/use parent pointer for ZWrappers with no scoperepos/abstract-wiki/wikifunctions/function-orchestrator!172ecarggrace/T337589/z-obj-scopes-issuemain
Customize query in GitLab

Event Timeline

Un-assigning myself for now. This is one of the quality compromises we'll have to make for launch: we can't feasibly fix this now.

If the status on this is still the same, would it make sense to mark this as 'Low'? Sry to intrude, just scouring for a good BE task to pick up!

@ecarg I just saw your note. Not intruding at all. Some related issues have started to cause bugs in prod, so (in my opinion) this would absolutely be a worthwhile thing to look at for fix-it week!

As I work on the metadata nesting stuff, I am learning a lot about scope attachment problems.

  1. When we call setScope on an object, we don't down-propagate this to its child objects (as we do when we first create a ZWrapper). This means that nested objects still refer to the enclosing scope. A possible fix here would be as follow: instead of setting the scope on every nested ZWrapper, we just give ZWrappers a reference to their parent (i.e., the ZWrapper that encloses them). That way, if the current ZWrapper doesn't have an attached scope, it can consult its parent (grandparent, etc.) to find a scope. When we later call setScope on an object, all of its child objects will be updated.
  2. We sometimes don't set scopes in a ZWrapper at all, which is just an error. We should figure out all the places we're creating ZWrappers without scopes and add scopes there (even just an EmptyFrame would be fine).

Change #1037084 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2024-05-13-145903 to 2024-05-23-164021

https://gerrit.wikimedia.org/r/1037084

Change #1037084 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2024-05-13-145903 to 2024-05-23-164021

https://gerrit.wikimedia.org/r/1037084

Currently doing some tests/debugging to check if this requires a follow-up task/MR!

Updates:

  • Running above Z7 function call w/ logs, I get below in API Sandbox result:
{
    "query": {
        "wikilambda_function_call": {
            "success": true,
            "data": "{\"Z1K1\":\"Z22\",\"Z22K1\":\"Z24\",\"Z22K2\":{\"Z1K1\":{\"Z1K1\":\"Z7\",\"Z7K1\":\"Z883\",\"Z883K1\":\"Z6\",\"Z883K2\":\"Z1\"},\"K1\":[{\"Z1K1\":\"Z7\",\"Z7K1\":\"Z882\",\"Z882K1\":\"Z6\",\"Z882K2\":\"Z1\"},{\"Z1K1\":{\"Z1K1\":\"Z7\",\"Z7K1\":\"Z882\",\"Z882K1\":\"Z6\",\"Z882K2\":\"Z1\"},\"K1\":\"errors\",\"K2\":{\"Z1K1\":\"Z5\",\"Z5K1\":\"Z558\",\"Z5K2\":\"Orchestration generally failed.\"}}]}}"
        }
    }
}
  • then I go to routes/evaluate.js where that error message lives and add logs everywhere but nothing shows up in my Orchestrator logs. It's as if that file is not even hit, yet the above output is returned

NVM, was NOT pointing to local orch in my extension.json, TY Cory.

Ok new error logs:

SyntaxError: Unexpected token ''', ...": [
function-orchestrator-1          |       'Z610', 'Z"... is not valid JSON
function-orchestrator-1          |     at JSON.parse (<anonymous>)
function-orchestrator-1          |     at getEvaluatorConfigsForEnvironment (/srv/service/src/utils.js:361:25)
function-orchestrator-1          |     at /srv/service/routes/evaluate.js:100:28
function-orchestrator-1          |     at /srv/service/lib/util.js:128:26
function-orchestrator-1          |     at tryCatcher (/srv/service/node_modules/bluebird/js/release/util.js:16:23)
function-orchestrator-1          |     at Promise.attempt.Promise.try (/srv/service/node_modules/bluebird/js/release/method.js:39:29)
function-orchestrator-1          |     at layer.handle (/srv/service/lib/util.js:128:18)
function-orchestrator-1          |     at Layer.handle [as handle_request] (/srv/service/node_modules/express/lib/router/layer.js:95:5)
function-orchestrator-1          |     at next (/srv/service/node_modules/express/lib/router/route.js:149:13)
function-orchestrator-1          |     at Route.dispatch (/srv/service/node_modules/express/lib/router/route.js:119:3)
function-orchestrator-1          |     at Layer.handle [as handle_request] (/srv/service/node_modules/express/lib/router/layer.js:95:5)
function-orchestrator-1          |     at /srv/service/node_modules/express/lib/router/index.js:284:15
function-orchestrator-1          |     at Function.process_params (/srv/service/node_modules/express/lib/router/index.js:346:12)
function-orchestrator-1          |     at next (/srv/service/node_modules/express/lib/router/index.js:280:10)
function-orchestrator-1          |     at Function.handle (/srv/service/node_modules/express/lib/router/index.js:175:3)
function-orchestrator-1          |     at router (/srv/service/node_modules/express/lib/router/index.js:47:12)
function-orchestrator-1          | {"name":"function-orchestrator","hostname":"fbdd4949a5f7","pid":1,"level":10,"err":{"message":"","name":"function-orchestrator","stack":"ReferenceError: allConfig is not defined\n    at getEvaluatorConfigsForEnvironment (/srv/service/src/utils.js:367:25)\n    at /srv/service/routes/evaluate.js:100:28\n    at /srv/service/lib/util.js:128:26\n    at tryCatcher (/srv/service/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise.attempt.Promise.try (/srv/service/node_modules/bluebird/js/release/method.js:39:29)\n    at layer.handle (/srv/service/lib/util.js:128:18)\n    at Layer.handle [as handle_request] (/srv/service/node_modules/express/lib/router/layer.js:95:5)\n    at next (/srv/service/node_modules/express/lib/router/route.js:149:13)\n    at Route.dispatch (/srv/service/node_modules/express/lib/router/route.js:119:3)\n    at Layer.handle [as handle_request] (/srv/service/node_modules/express/lib/router/layer.js:95:5)\n    at /srv/service/node_modules/express/lib/router/index.js:284:15\n    at Function.process_params (/srv/service/node_modules/express/lib/router/index.js:346:12)\n    at next (/srv/service/node_modules/express/lib/router/index.js:280:10)\n    at Function.handle (/srv/service/node_modules/express/lib/router/index.js:175:3)\n    at router (/srv/service/node_modules/express/lib/router/index.js:47:12)\n    at Layer.handle [as handle_request] (/srv/service/node_modules/express/lib/router/layer.js:95:5)","levelPath":"trace"},"msg":"allConfig is not defined","time":"2024-05-31T15:50:36.867Z","v":0}

Turns out my docker-compose.override.yml was using single quotes which caused above error; now I receive below in the API SB result:

{
    "query": {
        "wikilambda_function_call": {
            "success": true,
            "data": "{\"Z1K1\":\"Z22\",\"Z22K1\":\"Z24\",\"Z22K2\":{\"Z1K1\":{\"Z1K1\":\"Z7\",\"Z7K1\":\"Z883\",\"Z883K1\":\"Z6\",\"Z883K2\":\"Z1\"},\"K1\":[{\"Z1K1\":\"Z7\",\"Z7K1\":\"Z882\",\"Z882K1\":\"Z6\",\"Z882K2\":\"Z1\"},{\"Z1K1\":{\"Z1K1\":\"Z7\",\"Z7K1\":\"Z882\",\"Z882K1\":\"Z6\",\"Z882K2\":\"Z1\"},\"K1\":\"errors\",\"K2\":{\"Z1K1\":\"Z5\",\"Z5K1\":\"Z570\",\"Z5K2\":{\"Z1K1\":{\"Z1K1\":\"Z7\",\"Z7K1\":\"Z885\",\"Z885K1\":\"Z570\"},\"Z570K1\":\"100\"}}},{\"Z1K1\":{\"Z1K1\":\"Z7\",\"Z7K1\":\"Z882\",\"Z882K1\":\"Z6\",\"Z882K2\":\"Z1\"},\"K1\":\"implementationType\",\"K2\":\"Z14K2\"},{\"Z1K1\":{\"Z1K1\":\"Z7\",\"Z7K1\":\"Z882\",\"Z882K1\":\"Z6\",\"Z882K2\":\"Z1\"},\"K1\":\"orchestrationMemoryUsage\",\"K2\":\"164.31 MiB\"},{\"Z1K1\":{\"Z1K1\":\"Z7\",\"Z7K1\":\"Z882\",\"Z882K1\":\"Z6\",\"Z882K2\":\"Z1\"},\"K1\":\"orchestrationCpuUsage\",\"K2\":\"1106.329 ms\"},{\"Z1K1\":{\"Z1K1\":\"Z7\",\"Z7K1\":\"Z882\",\"Z882K1\":\"Z6\",\"Z882K2\":\"Z1\"},\"K1\":\"orchestrationStartTime\",\"K2\":\"2024-05-31T16:38:42.453Z\"},{\"Z1K1\":{\"Z1K1\":\"Z7\",\"Z7K1\":\"Z882\",\"Z882K1\":\"Z6\",\"Z882K2\":\"Z1\"},\"K1\":\"orchestrationEndTime\",\"K2\":\"2024-05-31T16:38:43.391Z\"},{\"Z1K1\":{\"Z1K1\":\"Z7\",\"Z7K1\":\"Z882\",\"Z882K1\":\"Z6\",\"Z882K2\":\"Z1\"},\"K1\":\"orchestrationDuration\",\"K2\":\"938 ms\"},{\"Z1K1\":{\"Z1K1\":\"Z7\",\"Z7K1\":\"Z882\",\"Z882K1\":\"Z6\",\"Z882K2\":\"Z1\"},\"K1\":\"orchestrationHostname\",\"K2\":\"baf3059ec03e\"}]}}"
        }
    }
}

Environment did NOT blow up BUT stack size did not grow

Ok I updated from main and I get another result from API SB today:

{
    "query": {
        "wikilambda_function_call": {
            "success": false,
            "data": {
                "Z1K1": "Z22",
                "Z22K2": {
                    "Z1K1": {
                        "Z1K1": "Z7",
                        "Z7K1": "Z883",
                        "Z883K1": "Z6",
                        "Z883K2": "Z1"
                    },
                    "K1": [
                        {
                            "Z1K1": "Z7",
                            "Z7K1": "Z882",
                            "Z882K1": "Z6",
                            "Z882K2": "Z1"
                        },
                        {
                            "Z1K1": {
                                "Z1K1": "Z7",
                                "Z7K1": "Z882",
                                "Z882K1": "Z6",
                                "Z882K2": "Z1"
                            },
                            "K1": "errors",
                            "K2": {
                                "Z1K1": "Z5",
                                "Z5K1": "Z507",
                                "Z5K2": {
                                    "Z1K1": {
                                        "Z1K1": "Z7",
                                        "Z7K1": "Z885",
                                        "Z885K1": "Z507"
                                    },
                                    "K1": {
                                        "Z1K1": "Z99",
                                        "Z99K1": {
                                            "Z1K1": "Z7",
                                            "Z7K1": {
                                                "Z1K1": "Z8",
                                                "Z8K1": [
                                                    "Z17",
                                                    {
                                                        "Z1K1": "Z17",
                                                        "Z17K1": "Z8",
                                                        "Z17K2": "Z10043K1",
                                                        "Z17K3": {
                                                            "Z1K1": "Z12",
                                                            "Z12K1": [
                                                                "Z11"
                                                            ]
                                                        }
                                                    }
                                                ],
                                                "Z8K2": "Z1",
                                                "Z8K3": [
                                                    "Z20"
                                                ],
                                                "Z8K4": [
                                                    "Z14",
                                                    {
                                                        "Z1K1": "Z14",
                                                        "Z14K1": "Z10043",
                                                        "Z14K2": {
                                                            "Z1K1": "Z7",
                                                            "Z7K1": {
                                                                "Z1K1": "Z18",
                                                                "Z18K1": "Z10043K1"
                                                            },
                                                            "K1": {
                                                                "Z1K1": "Z18",
                                                                "Z18K1": "Z10043K1"
                                                            }
                                                        }
                                                    }
                                                ],
                                                "Z8K5": "Z10043"
                                            },
                                            "Z10043K1": {
                                                "Z1K1": "Z8",
                                                "Z8K1": [
                                                    "Z17",
                                                    {
                                                        "Z1K1": "Z17",
                                                        "Z17K1": "Z8",
                                                        "Z17K2": "Z10044K1",
                                                        "Z17K3": {
                                                            "Z1K1": "Z12",
                                                            "Z12K1": [
                                                                "Z11"
                                                            ]
                                                        }
                                                    }
                                                ],
                                                "Z8K2": "Z1",
                                                "Z8K3": [
                                                    "Z20"
                                                ],
                                                "Z8K4": [
                                                    "Z14",
                                                    {
                                                        "Z1K1": "Z14",
                                                        "Z14K1": "Z10044",
                                                        "Z14K2": {
                                                            "Z1K1": "Z7",
                                                            "Z7K1": {
                                                                "Z1K1": "Z18",
                                                                "Z18K1": "Z10044K1"
                                                            },
                                                            "K1": {
                                                                "Z1K1": "Z18",
                                                                "Z18K1": "Z10044K1"
                                                            }
                                                        }
                                                    }
                                                ],
                                                "Z8K5": "Z10044"
                                            }
                                        }
                                    },
                                    "K2": {
                                        "Z1K1": "Z5",
                                        "Z5K1": "Z500",
                                        "Z5K2": {
                                            "Z1K1": {
                                                "Z1K1": "Z7",
                                                "Z7K1": "Z885",
                                                "Z885K1": "Z500"
                                            },
                                            "K1": "Internal Server Error"
                                        }
                                    }
                                }
                            }
                        }
                    ]
                }
            }
        }
    }
}