I think that for RESTBase simply going through all the endpoints is a wrong approach. We need to test up different behaviours/codepaths of RESTRouter, not the same exact codepath with different data. So, instead I'm going to go through the following behaviours:

1. Fetching HTML and Summary from storage - the most common codepath
2. Fetching HTML and Summary from storage with no-cache - most common update codepath
3. Fetching math formulae - very common read path
4. Fetching PDF - extra long backend response time
5. Fetching HTML and Summary with non-standard language variant - multi-step codepath with many backend services contacted
6. Fetching some PCS content - simple proxy with no storage and reasonably quick backend response time
7. Transform endpoint

I believe that these tests should be more than enough to figure out the initial limits. We can adjust as we go.

Fetching HTML from storage. C1-30, n1000

Fetching summary from storage, C1-30, n1000

Fetching HTML with no-cache, C1-5-30, n100

Fetching summary with no-cache, C1-5-30, n500

These suggest quite a strong and obvious pattern, the more we wait for backends to generate the content, the less memory/cpu we require.

These also agree quite well with what we're seeing in production with the real traffic. I will continue the experiments, but I think that 1CPU per worker and 750M per worker should be reasonable limits, to begin with judging from what production sees.

After more load testing here's the numbers I propose with explanation:

num_workers: 2

Starting up RESTRouter takes time. Quite a long time. So, we want to lower the probability of having a dangling master, thus 2 workers.

requests.cpu=1600m

According to load testing, we have 2 fundamentally different kinds of requests - those that hit storage and those that just proxy to backend services and both have completely different CPU requirements. When we serve requests from storage, we can max out CPU (1 CPU per worker) given appropriate , while requests that are just a proxy use hardly any CPU since we're mostly waiting for IO. In production 80% of requests are served from storage, thus if we had a minimum viable number of pods (having 0 capacity over) we would've run under 80% CPU. The hard limit on possible CPU per pod is 2000m since node is single-threaded and we have 2 workers per pod. Thus, 2000*0.8=1600

requests.memory=800Mi

Currently in production the mean memory consumption is 400Mi per worker. Thus 2 workers per pod = 800Mi

limits.cpu=2

The hard limit for CPU consumption of a node worker is 1, thus 2 workers = 2.

limits.memory=1500Mi

service-runner kills RESTBase worker when 700Mi is per worker is reached. We're running 2 workers, plus a little room for master.