signal 9 is SIGKILL. OOM killer?

I see "/usr/lib/python2.7/dist-packages/toollabs/webservice/backends/kubernetesbackend.py"

'python': {
    'cls': PythonWebService,
    'image': 'toollabs-python-web',
    'resources': {
         'limits': {
            # Pods will be killed if they go over memory limit
            'memory': '2Gi',
            # Pods can still burst to more than cpu limit
            'cpu': '2',
         },
         'requests': {
            # Pods are guaranteed at least this many resources
            'memory': '256Mi',
            'cpu': '0.125'
         }
    }
},

With 2 Gi I might run into problems. Does the number of workers (where the default i 4 as far as I know) mean that there is only 500 MB for each service?

I do not know how to start it with few workers. Isn't the number 4 hardcoded into /usr/lib/python2.7/dist-packages/toollabs/webservice/services/uwsgiwebservice.py?

The tool is this one: http://tools.wmflabs.org/wembedder/
At the moment it performs fine.

I have seen it become slow overnight. I restarted it this morning with webservice --backend=kubernetes python start.

On my Flask development server, Python3 is using 1.2 GB fairly constantly (somewhat more than I anticipated). I suppose that I run into memory problems. I wonder how I can get around this.

I still get "502 Bad Gateway". It seems to be tied to one worker. Currently, I get "Respawned uWSGI worker 1" several times, while the other workers (that I see with different pid) does not seem to run into problems.

Is there are problem with applications that use so much memory. I was under the impression that the service would be able to handle up to 4 GB.

By default most webservices running on our Kubernetes cluster are allowed a hard limit of 2GB of ram. Java gets 4GB by default because Java. We have a process for requesting more quota that is documented on wikitech. Looking at the backend code, it seems that implementing quota bumps for Kubernetes has not been done yet.

Is it possible for me to see the load off the computer? I tried with top, qstat or kubectl.

It depends on what context you want to see.

• kubectl describe nodes will show you the status of all the exec nodes in the Kubernetes cluster (slow and verbose).
• kubectl describe node <hostname> will show you a single exec node. You can find out which node your pod is running on with kubectl get pod <pod> -o json|jq '.spec.nodeName'
• You can get a shell inside a running pod with kubectl exec -i -t <pod> /bin/bash. The containers that are used to run things on the Kubernetes grid are pretty bare-bones, so you may have difficulty from there.

webservice --backend=kubernetes python start takes several minutes, while on my computer the setup of the application takes a few seconds.

Starting the pod requires finding a free space in the exec nodes, downloading a Docker image, starting that Docker image, attaching various NFS exports, and finally starting your Python process from an NFS share. This actually happens pretty quickly for all the moving parts, but it is certainly not as fast as starting a local process.