Hi. I think you can add more replicas with the --replicas webservice flag, the default quota is 10 pods so you should be able to fit some extra replicas comfortably there. Right now it seems like the pod has its CPU and RAM allocations increased from the defaults so you would need more quota to do several pods of that size, I would suggest starting from trying several smaller pods and if that does not work we can increase the quota to have several larger pods.

I also see you're still running Python 2 which has been unsupported for several years at this point, I would suggest moving to a modern Python versions as those have various performance improvements and other fixes that you could take advantage of.

Yeah, the Python 3 backend is already deployed over on the montage-dev tool. Peak traffic isn't an ideal time to migrate.

And yeah, I tried running more replicas, but I guess the hardware resource constraints limit Montage to only two pods. We actually need the higher memory per pod because while most requests are small, some actually do consume that much memory (loading tens of thousands of images, etc.). We do run multiple workers per pod to account for that pattern.

@mahmoud the hardware resources (technically VM resources) should not be a constraint, as replicas can spread across different nodes. I think the CPU and Memory quotas are the current constraints. Would it be useful to double those to cpu: 4 and memory: 16Gi?

(venv)tools.montage@tools-sgebastion-10:~$ toolforge jobs quota
Running jobs                                  Used    Limit
--------------------------------------------  ------  -------
Total running jobs at once (Kubernetes pods)  2       10
Running one-off and cron jobs                 0       15
CPU                                           2       2
Memory                                        8Gi     8Gi

Per-job limits    Limit
----------------  -------
CPU               1
Memory            4Gi

Yeah, after a few days of monitoring, there's definite improvement.

User complaints have gone down, but I'm still getting perf-related error notifications from our monitoring systems. Less often since upping to two pods, but still 1-2 a day. I'd definitely be interested in increasing resources, especially since I'm getting a message on restart which I think is saying something to the effect of "montage is running so close to resource limits that we can't do rolling restarts." If you can allocate at least enough resources we can get graceful restarts back, that would be great.

As an aside, I'm mostly monitoring the web workers; it occurs to me that too many workers could also overload the db resources. If there are some recommended practices for monitoring those, I'd love a link or two. Thanks!

I'd definitely be interested in increasing resources

@taavi can I get your +1 on doubling the CPU and Memory quotas for this tool?

it occurs to me that too many workers could also overload the db resources. If there are some recommended practices for monitoring those, I'd love a link or two.

What is the DB you are writing to? If it's ToolsDB, we only have overall metrics, but we currently don't have metrics showing the resources (connections, CPU, disk, etc.) used by a specific tool. If you have a dedicated database (Trove or CloudVPS), it might be easier.

In T348894#9269025, @fnegri wrote:

I'd definitely be interested in increasing resources

@taavi can I get your +1 on doubling the CPU and Memory quotas for this tool?

+1

it occurs to me that too many workers could also overload the db resources. If there are some recommended practices for monitoring those, I'd love a link or two.

I think currently the best approach would be to add logging for slow queries to the app itself.

Great, thank you for the resources and tips! Will circle back if there are more reports of issues. Thanks again!