Page MenuHomePhabricator

[spike] find out where the delay is coming from
Closed, ResolvedPublic

Description

What/Why:
Whenever we request large objects, namely items and lexemes, there is significant delay in the Orchestrator's response, upwards of 3 seconds. We need to find out why or where this is occurring so we can continue with Wikipedia integration work.

How:

  • observe metrics, esp the consumption of NodeJS heap/memory usage (XX probably not the culprit)
  • observe behavior with and without pre-validation (XX probably not the culprit)
  • observe behavior without Schema validation ( ... still checking)

Details

Related Changes in Gerrit:
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Update function-schemata sub-module to HEAD (c17a559)repos/abstract-wiki/wikifunctions/wikilambda-cli!66jforrestersync-function-schematamain
Update function-schemata sub-module to HEAD (c17a559)repos/abstract-wiki/wikifunctions/function-evaluator!328jforrestersync-function-schematamain
[BREAKING CHANGE] Update function-schemata sub-module to HEAD (c17a559)repos/abstract-wiki/wikifunctions/function-orchestrator!315jforrestersync-function-schematamain
Start to eliminate JSON schema validators.repos/abstract-wiki/wikifunctions/function-schemata!200apineapine-no-schema-in-normorcanonmain
Optimize shortcuts for Z881.repos/abstract-wiki/wikifunctions/function-orchestrator!299apineapine-Z881main
Add test with trace information in the call tree.repos/abstract-wiki/wikifunctions/function-orchestrator!298apineapine-call-tree-visualizermain
Customize query in GitLab

Event Timeline

👋
Is this what @cmassaro is working on? Should this be assigned to him, and should the status be updated to "In progress" and moved to the column "In Engineering"?

ecarg changed the task status from Open to In Progress.Feb 5 2025, 3:52 PM
ecarg assigned this task to cmassaro.

So far, removing JSON schema validation has been shown to have very little effect (sometimes positive, sometimes negative). However, that was before recent changes to pass the isValidated and isFullyResolved states throughout the orchestrator. I think it's still worth revisiting this work, particularly if we can rely more on those new bits of state. Ideally, we'd rewrite ALL validator functions in pure JS, and these functions would use type comparison in combination with those states to avoid doing repeated work. For example, if an object of type ZXXXX1 has a member of type ZXXXX2, then the validator function could look at that member and run the following algorithm:

if ( typeCompares( member, 'ZXXXX2' ) ) {
    if ( !member.isValidated() ) {
        validation = runValidator( member, 'ZXXXX2' );
        if ( responseEnvelopeContainsError( validation ) ) {
            return someError; // include the key for the member along with the validation's Z22K2, in order to percolate the error upward with tracing information
        } else {
            member.setIsValidated(); // or the runValidator function might do this directly
        }
    }
} else {
    // possibly perform dditional processing in case this member is a Z7, Z9, Z18, etc.
}

Change #1126996 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Update orchestrator from 2025-03-05-140247 to 2025-03-11-234105

https://gerrit.wikimedia.org/r/1126996

Change #1126996 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Update orchestrator from 2025-03-05-140247 to 2025-03-11-234105

https://gerrit.wikimedia.org/r/1126996

Change #1128843 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (c17a559)

https://gerrit.wikimedia.org/r/1128843

Change #1129367 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Update evaluators from 2025-03-11-234147 to 2025-03-19-125950

https://gerrit.wikimedia.org/r/1129367

Change #1129368 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Update orchestrator from 2025-03-11-234105 to 2025-03-19-203723

https://gerrit.wikimedia.org/r/1129368

Change #1129367 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Update evaluators from 2025-03-11-234147 to 2025-03-19-125950

https://gerrit.wikimedia.org/r/1129367

Change #1129368 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Update orchestrator from 2025-03-11-234105 to 2025-03-19-203723

https://gerrit.wikimedia.org/r/1129368

Change #1128843 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (c17a559)

https://gerrit.wikimedia.org/r/1128843

The findings of this work can be found in the "Performance Problems" document.

The findings of this work can be found in the "Performance Problems" document.

There's still (at least) the sub-task to reply/resolve, ideally. :-)