Change Details

The commons-mass-description tool ran into this problem today. @Urbanecm was trying to start using `webservice --backend=kubernetes python start` and got the error message: ``` Looks like you already have another webservice running, with a gridengine backend You should stop that webservice by issuing: webservice --backend=gridengine stop And then start it again with backend kubernetes by issuing: webservice --backend=kubernetes start ``` `qstat` showed no process running at that instant, but `service.log` recorded `2017-04-19T17:13:22.646841 No running webservice job found, attempting to start it` and then a grid job spawned. This job died as soon as it started with this `uwsgi.log` output: ``` *** Starting uWSGI 1.9.17.1-debian (64bit) on [Wed Apr 19 17:13:55 2017] *** compiled with version: 4.8.2 on 23 March 2014 17:15:32 os: Linux-3.13.0-100-generic #147-Ubuntu SMP Tue Oct 18 16:48:51 UTC 2016 nodename: tools-webgrid-generic-1404 machine: x86_64 clock source: unix pcre jit disabled detected number of CPU cores: 4 current working directory: /mnt/nfs/labstore-secondary-tools-project/commons-mas s-description detected binary path: /usr/bin/uwsgi-core your processes number limit is 63707 your process address space limit is 4294967296 bytes (4096 MB) your memory page size is 4096 bytes detected max file descriptor number: 1024 lock engine: pthread robust mutexes thunder lock: disabled (you can enable it with --thunder-lock) uwsgi socket 0 bound to TCP address :52476 fd 3 Python version: 2.7.6 (default, Oct 26 2016, 20:33:43) [GCC 4.8.4] Set PythonHome to /data/project/commons-mass-description/www/python/venv ImportError: No module named site ``` This error is caused by the `$HOME/www/python/venv` being built for Python3 on Kubernetes and the app being executed by Python2 on grid engine. Multiple attempts to stop the grid engine webservice backend with `webservice stop` just printed `Your webservice is not running`. The `service.manifest` file however still contained: ``` # This file is used by toollabs infrastructure. # Please do not edit manually at this time. backend: gridengine version: 2 web: uwsgi-python ``` Ultimately, using `rm service.manifest` stopped the `webservice` reconciliation loop from trying to restart the service on the job grid and allowed it to be started on kubernetes with `webservice --backend=kubernetes python start`.

Current workaround for people having this problem: * `webservice stop` * `rm $HOME/service.manifest` * `webservice [add other args here as needed] start` --- The commons-mass-description tool ran into this problem today. @Urbanecm was trying to start using `webservice --backend=kubernetes python start` and got the error message: ``` Looks like you already have another webservice running, with a gridengine backend You should stop that webservice by issuing: webservice --backend=gridengine stop And then start it again with backend kubernetes by issuing: webservice --backend=kubernetes start ``` `qstat` showed no process running at that instant, but `service.log` recorded `2017-04-19T17:13:22.646841 No running webservice job found, attempting to start it` and then a grid job spawned. This job died as soon as it started with this `uwsgi.log` output: ``` *** Starting uWSGI 1.9.17.1-debian (64bit) on [Wed Apr 19 17:13:55 2017] *** compiled with version: 4.8.2 on 23 March 2014 17:15:32 os: Linux-3.13.0-100-generic #147-Ubuntu SMP Tue Oct 18 16:48:51 UTC 2016 nodename: tools-webgrid-generic-1404 machine: x86_64 clock source: unix pcre jit disabled detected number of CPU cores: 4 current working directory: /mnt/nfs/labstore-secondary-tools-project/commons-mas s-description detected binary path: /usr/bin/uwsgi-core your processes number limit is 63707 your process address space limit is 4294967296 bytes (4096 MB) your memory page size is 4096 bytes detected max file descriptor number: 1024 lock engine: pthread robust mutexes thunder lock: disabled (you can enable it with --thunder-lock) uwsgi socket 0 bound to TCP address :52476 fd 3 Python version: 2.7.6 (default, Oct 26 2016, 20:33:43) [GCC 4.8.4] Set PythonHome to /data/project/commons-mass-description/www/python/venv ImportError: No module named site ``` This error is caused by the `$HOME/www/python/venv` being built for Python3 on Kubernetes and the app being executed by Python2 on grid engine. Multiple attempts to stop the grid engine webservice backend with `webservice stop` just printed `Your webservice is not running`. The `service.manifest` file however still contained: ``` # This file is used by toollabs infrastructure. # Please do not edit manually at this time. backend: gridengine version: 2 web: uwsgi-python ``` Ultimately, using `rm service.manifest` stopped the `webservice` reconciliation loop from trying to restart the service on the job grid and allowed it to be started on kubernetes with `webservice --backend=kubernetes python start`.