Current workaround for people having this problem:
- webservice stop
- rm $HOME/service.manifest
- webservice [add other args here as needed] start
The commons-mass-description tool ran into this problem today. @Urbanecm was trying to start using webservice --backend=kubernetes python start and got the error message:
Looks like you already have another webservice running, with a gridengine backend You should stop that webservice by issuing: webservice --backend=gridengine stop And then start it again with backend kubernetes by issuing: webservice --backend=kubernetes start
qstat showed no process running at that instant, but service.log recorded 2017-04-19T17:13:22.646841 No running webservice job found, attempting to start it and then a grid job spawned. This job died as soon as it started with this uwsgi.log output:
*** Starting uWSGI 1.9.17.1-debian (64bit) on [Wed Apr 19 17:13:55 2017] *** compiled with version: 4.8.2 on 23 March 2014 17:15:32 os: Linux-3.13.0-100-generic #147-Ubuntu SMP Tue Oct 18 16:48:51 UTC 2016 nodename: tools-webgrid-generic-1404 machine: x86_64 clock source: unix pcre jit disabled detected number of CPU cores: 4 current working directory: /mnt/nfs/labstore-secondary-tools-project/commons-mas s-description detected binary path: /usr/bin/uwsgi-core your processes number limit is 63707 your process address space limit is 4294967296 bytes (4096 MB) your memory page size is 4096 bytes detected max file descriptor number: 1024 lock engine: pthread robust mutexes thunder lock: disabled (you can enable it with --thunder-lock) uwsgi socket 0 bound to TCP address :52476 fd 3 Python version: 2.7.6 (default, Oct 26 2016, 20:33:43) [GCC 4.8.4] Set PythonHome to /data/project/commons-mass-description/www/python/venv ImportError: No module named site
This error is caused by the $HOME/www/python/venv being built for Python3 on Kubernetes and the app being executed by Python2 on grid engine.
Multiple attempts to stop the grid engine webservice backend with webservice stop just printed Your webservice is not running. The service.manifest file however still contained:
# This file is used by toollabs infrastructure. # Please do not edit manually at this time. backend: gridengine version: 2 web: uwsgi-python
Ultimately, using rm service.manifest stopped the webservice reconciliation loop from trying to restart the service on the job grid and allowed it to be started on kubernetes with webservice --backend=kubernetes python start.