Bumping this task. We would like to have a performance readiness review performed in Q4 FY 2020-2021. Exact projected requested start date to be determined and agreed upon within the Abstract Wikipedia team.

Heads up @Gilles, request for performance readiness review Q4. As you probably know, the architecture has changed, with the WikiLambda extension (still) in MediaWiki and a service orchestrator / choreographer component in Node.js and programming language evaluator componentry behind that; it may change in some ways although that would likely be after first launch dependent on real world observed workload behavior.

We'll need to update the task Description to better reflect reality. But first, I wanted to get this in the queue for Q4. Looking forward to it!

Sounds good to me, thanks for requesting it early!

@Gilles I touched up the Description some. As emailed, we'd like review from the 1st of June (Tuesday) to the 14th of June (Tuesday), 2021, inclusive.

Thanks!

As promised, I've updated the main docker-compose instructions to hopefully cover both the main testing use case and the likely variants that you'll encounter, and confirmed in a brand-new check-out that it works as I expect, using:

function-orchestrator:
  image: docker-registry.wikimedia.org/wikimedia/mediawiki-services-function-orchestrator:2021-05-26-191706-production
  ports:
    - 6254:6254
function-evaluator:
  image: docker-registry.wikimedia.org/wikimedia/mediawiki-services-function-evaluator:2021-05-27-170058-production
  ports:
    - 6927:6927

Special:CreateZObject page

• A fair amount of content jumping around can be avoided by matching the no-JS placeholder's sizing to the actual final UI's. This should be optimised for the common case (user has JS). The same is true for Special:EvaluateFunctionCall
• The language list query doesn't appear to be cached at all (Cache-Control: private, must-revalidate, max-age=0 response). I don't expect the list of languages to change that frequently? Even caching this for a few minutes server-sider and client-side would help.
• The loaded JS files coming from Ace are uncached a well (mode-json.js theme-chrome.js worker-json.js)

I don't know why Ace needs a service worker at all, but I didn't see an obvious way to avoid it in their documentation. I guess that odd architecture choice is something we'll have to live with as long as Ace is what's used for syntax highlighting.

Orchestrator and evaluator

I benchmarked the orchestrator and evaluator by hammering the API endpoint with ab. There doesn't seem to be any memory leak in either service.

In terms of performance, the drawback of the single-threaded nature of node starts to show pretty quickly as the concurrency increases. With 10 concurrent requests, the median response time is around 600ms. As expected with 100 concurrent requests, it becomes 6s.

Having a dynamically sized pool of evaluator services is desirable for the production deployment. I assume that the orchestrator sends requests to the evaluator asynchronously and as a result shouldn't need to be pooled as well.

The issue might be solvable inside the service as well, if the actual running of functions is done inside a Web Worker instead of directly on the main thread. But attempting to solve this in-service has downsides: less control over the concurrency and observability becomes more complex as well.

I would like to test the timeout abilities, but I really can't figure out the UI in order to create something that would call a custom JS function. Is there a step-by-step tutorial I could follow in order to do that?

Hi, Gilles, a few questions/comments.

I assume that the orchestrator sends requests to the evaluator asynchronously and as a result shouldn't need to be pooled as well

Correct, all communication with the orchestrator and evaluator is asynchronous. Out of curiosity, with these concurrent requests, how much time is spent in the orchestrator vs. the evaluator?

The issue might be solvable inside the service as well, if the actual running of functions is done inside a Web Worker instead of directly on the main thread

I did not know about this! Are you recommending this for the evaluator or the orchestrator?

I really can't figure out the UI in order to create something that would call a custom JS function

Do you want to call this from the wiki or insert your request somewhere deeper in the stack (e.g. orchestrator, evaluator)? If you check out the API sandbox, there is an example request that will let you run JS code. As for a step-by-step tutorial ... hmm, there isn't, but there should be. At present, I am happy to discuss this with you if you like!

I haven't checked what's causing the single-threaded behaviour. I presume it's happening in the evaluator, as you're probably running code on the main thread there?

I've tried modifying the python3 API sandbox example, but I suspect it's actually not fully working for me, as the output values don't have the result of the sum. I get:

{
    "query": {
        "wikilambda_function_call": {
            "Orchestrated": {
                "success": "",
                "data": "{\"Z1K1\":\"Z5\",\"Z5K1\":{\"Z1K1\":\"Z402\",\"Z402K1\":{\"Z1K1\":\"Z421\",\"Z421K1\":null}}}"
            }
        }
    }
}

Is this the expected output?

And trying to modify the python code to call python's time.sleep() doesn't seem to sleep at all. I have a feeling that the python code is actually not being run and there's no visible error in the output. Is there an error log somewhere from the evaluator that I could inspect?

Hmm, I also notice that nothing is working at the moment. We've made a bunch of changes to the orchestrator recently, so I assume that's the reason. I will investigate today.

probably running code on the main thread

All code is run within promises, but in the main thread, yes.

error log somewhere from the evaluator

Not yet; that is something I would very much like to have.

Following up from off thread discussion: we'll plan on the following based on what we know today:

1. Loading to beta cluster. Work will be happening over the next short while to get there.
2. Sometime later in Q2, further performance readiness review.
3. In 2022, after working through pre-launch production considerations, MVP production launch.