Currently the code is a large number of shell scripts (webservice2, tool-lighttpd, lighttpd-started, tool-tomcat, tomcat-starter, tool-nodejs, tool-uwsgi-python, portgrabber, portgrabber.py, symlink to webservice) that all call each other and are written in a combination of python and bash.
Unify this into a single python package with a proper, nicer design.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | yuvipanda | T129309 Goal: Allow using k8s instead of GridEngine as a backend for webservices | |||
| Resolved | yuvipanda | T98440 Deploy new unified webservice code | |||
| Resolved | yuvipanda | T98441 Convert lighttpd-starter from bash to python | |||
| Resolved | yuvipanda | T98442 Convert tomcat-starter to python | |||
| Resolved | yuvipanda | T98818 Deprecate #no-default-php in .lighttpd.conf | |||
| Resolved | bd808 | T132987 Write documentation on new webservice code |
So ^ is the beginning of webservice-new for interactive use and webservice-runner for actually starting things on the webservice nodes. Also contains implementations of tool-nodejs and tool-python-uwsgi as appropriate extensible modules.
To recap, the code is mostly done, but the big NFS outage in June took vastly bigger priority over this. Just needs some finishing touches and support in webservicemonitor and then we can switch to this.
After thinking about this some more -- should we spend time on this, or is it better to document which webservices use 'generic' (i.e. which webservices are not restarted by webservicewatcher) and then just move everything to k8s when it is ready?
I spent some time making webservicemonitor work with webservice-new, and it does work fine now. webservice-new will add a 'version: 2' field in the service.manifest, and webservicemonitor will look for this.
Testing the different command invocations people use for it...
+-----------------------------------------------------------------+-------+ | event_commandline | count | +-----------------------------------------------------------------+-------+ | /usr/local/bin/webservice --release trusty uwsgi-python restart | 97985 | | /usr/local/bin/webservice --release trusty nodejs restart | 80363 | | /usr/local/bin/webservice --release precise lighttpd restart | 28658 | | /usr/local/bin/webservice restart | 28077 | | /usr/local/bin/webservice --release trusty lighttpd restart | 24803 | | /usr/local/bin/webservice uwsgi-python start | 10958 | | /usr/local/bin/webservice uwsgi-python stop | 10914 | | /usr/local/bin/webservice --release trusty generic restart | 7345 | | /usr/local/bin/webservice --release trusty uwsgi-plain restart | 1908 | | /usr/local/bin/webservice start | 438 | | /usr/local/bin/webservice2 uwsgi-python restart | 320 | | /usr/local/bin/webservice stop | 292 | | /usr/local/bin/webservice status | 229 | | /usr/local/bin/webservice2 uwsgi-python start | 216 | | /usr/local/bin/webservice2 uwsgi-plain restart | 196 | | /usr/local/bin/webservice2 uwsgi-python stop | 189 | | /usr/local/bin/webservice uwsgi-python restart | 136 | | /usr/local/bin/webservice2 tomcat start | 91 | | /usr/local/bin/webservice2 uwsgi-plain stop | 90 | | /usr/local/bin/webservice2 uwsgi-plain start | 72 | | /usr/local/bin/webservice --release trusty tomcat restart | 39 | | /usr/local/bin/webservice2 stop | 33 | | /usr/local/bin/webservice2 start | 28 | | /usr/local/bin/webservice2 restart | 22 | | /usr/local/bin/webservice2 nodejs start | 18 | | /usr/local/bin/webservice | 18 | | /usr/local/bin/webservice2 status | 16 | | /usr/local/bin/webservice --release trusty restart | 14 | | /usr/local/bin/webservice2 uwsgi-python status | 11 | | /usr/local/bin/webservice --release trusty start | 11 | | /usr/local/bin/webservice lighttpd restart | 11 | | /usr/local/bin/webservice2 nodejs restart | 11 | | /usr/local/bin/webservice2 nodejs stop | 9 | | /usr/local/bin/webservice --help | 8 | | /usr/local/bin/webservice start --release precise | 7 | | /usr/local/bin/webservice -h | 6 | | /usr/local/bin/webservice --release precise start | 5 | | /usr/local/bin/webservice --release trusty lighttpd start | 4 | | /usr/local/bin/webservice2 | 4 | | /usr/local/bin/webservice python-uwsgi restart | 3 | | /usr/local/bin/webservice2 uwsgi-plain status | 3 | | /usr/local/bin/webservice uswsgi-python restart | 2 | | /usr/local/bin/webservice uwsgi-python status | 2 | | /usr/local/bin/webservice --release trusty lighttpd status | 2 | | /usr/local/bin/webservice start --release=precise | 2 | | /usr/local/bin/webservice --release trusty lighttpd stop | 2 | | /usr/local/bin/webservice --release trusty stop | 2 | | /usr/local/bin/webservice lighttpd start | 2 | | /usr/local/bin/webservice2 uwsgi-python restrart | 2 | | /usr/local/bin/webservice2 --help | 2 | | /usr/local/bin/webservice2 uwsgi-python -h | 2 | | /usr/local/bin/webservice statu | 2 | | /usr/local/bin/webservice2 uwsgi-python restrat | 1 | | /usr/local/bin/webservice reset | 1 | | /usr/local/bin/webservice2 --release trusty restart | 1 | | /usr/local/bin/webservice --status | 1 | | /usr/local/bin/webservice restart uwsgi-python | 1 | | /usr/local/bin/webservice uwsgi start | 1 | | /usr/local/bin/webservice -release trusty on | 1 | | /usr/local/bin/webservice2 uwsgi-plain start -v | 1 | | /usr/local/bin/webservice2 nodejs top | 1 | | /usr/local/bin/webservice start --release precise -v | 1 | | /usr/local/bin/webservice --release=trusty start | 1 | | /usr/local/bin/webservice2 tomcat stop | 1 | | /usr/local/bin/webservice kill | 1 | | /usr/local/bin/webservice --release precise lighttpd start | 1 | | /usr/local/bin/webservice uwsgi-python step | 1 | | /usr/local/bin/webservice --release trusty | 1 | | /usr/local/bin/webservice2 wsgi-python restrart | 1 | | /usr/local/bin/webservice strt uwsgi-python | 1 | | /usr/local/bin/webservice2 nodejs sart | 1 | | /usr/local/bin/webservice2 -h | 1 | | /usr/local/bin/webservice lighttpd start --release trusty | 1 | | /usr/local/bin/webservice start uwsgi | 1 | | /usr/local/bin/webservice -v status | 1 | | /usr/local/bin/webservice2 start --release trusty | 1 | | /usr/local/bin/webservice help | 1 | | /usr/local/bin/webservice2 uwsgistatus | 1 | | /usr/local/bin/webservice -help | 1 | | /usr/local/bin/webservice start --version=3 | 1 | +-----------------------------------------------------------------+-------+ 80 rows in set (6.80 sec)
Without tools-services (aka webservicemonitor):
mysql:research@analytics-store.eqiad.wmnet [log]> select event_commandline, COUNT(*) as count FROM CommandInvocation_15243810 WHERE event_command = 'webservice' AND event_hostname NOT LIKE 'tools-services-%' GROUP BY event_commandline ORDER BY count DESC; +-----------------------------------------------------------------+-------+ | event_commandline | count | +-----------------------------------------------------------------+-------+ | /usr/local/bin/webservice restart | 20487 | | /usr/local/bin/webservice uwsgi-python start | 10958 | | /usr/local/bin/webservice uwsgi-python stop | 10914 | | /usr/local/bin/webservice start | 438 | | /usr/local/bin/webservice2 uwsgi-python restart | 320 | | /usr/local/bin/webservice stop | 292 | | /usr/local/bin/webservice status | 229 | | /usr/local/bin/webservice2 uwsgi-python start | 216 | | /usr/local/bin/webservice2 uwsgi-plain restart | 196 | | /usr/local/bin/webservice2 uwsgi-python stop | 189 | | /usr/local/bin/webservice --release trusty nodejs restart | 137 | | /usr/local/bin/webservice uwsgi-python restart | 136 | | /usr/local/bin/webservice2 tomcat start | 91 | | /usr/local/bin/webservice2 uwsgi-plain stop | 90 | | /usr/local/bin/webservice --release trusty uwsgi-python restart | 75 | | /usr/local/bin/webservice2 uwsgi-plain start | 72 | | /usr/local/bin/webservice2 stop | 33 | | /usr/local/bin/webservice2 start | 28 | | /usr/local/bin/webservice2 restart | 22 | | /usr/local/bin/webservice2 nodejs start | 18 | | /usr/local/bin/webservice | 18 | | /usr/local/bin/webservice2 status | 16 | | /usr/local/bin/webservice --release trusty restart | 14 | | /usr/local/bin/webservice --release trusty start | 11 | | /usr/local/bin/webservice2 nodejs restart | 11 | | /usr/local/bin/webservice lighttpd restart | 11 | | /usr/local/bin/webservice2 uwsgi-python status | 11 | | /usr/local/bin/webservice2 nodejs stop | 9 | | /usr/local/bin/webservice --help | 8 | | /usr/local/bin/webservice --release trusty lighttpd restart | 7 | | /usr/local/bin/webservice start --release precise | 7 | | /usr/local/bin/webservice -h | 6 | | /usr/local/bin/webservice --release precise start | 5 | | /usr/local/bin/webservice2 | 4 | | /usr/local/bin/webservice --release trusty lighttpd start | 4 | | /usr/local/bin/webservice --release precise lighttpd restart | 4 | | /usr/local/bin/webservice2 uwsgi-plain status | 3 | | /usr/local/bin/webservice python-uwsgi restart | 3 | | /usr/local/bin/webservice start --release=precise | 2 | | /usr/local/bin/webservice --release trusty lighttpd stop | 2 | | /usr/local/bin/webservice --release trusty stop | 2 | | /usr/local/bin/webservice lighttpd start | 2 | | /usr/local/bin/webservice2 uwsgi-python restrart | 2 | | /usr/local/bin/webservice2 --help | 2 | | /usr/local/bin/webservice2 uwsgi-python -h | 2 | | /usr/local/bin/webservice statu | 2 | | /usr/local/bin/webservice uswsgi-python restart | 2 | | /usr/local/bin/webservice uwsgi-python status | 2 | | /usr/local/bin/webservice --release trusty lighttpd status | 2 | | /usr/local/bin/webservice --release trusty | 1 | | /usr/local/bin/webservice2 wsgi-python restrart | 1 | | /usr/local/bin/webservice2 nodejs sart | 1 | | /usr/local/bin/webservice2 -h | 1 | | /usr/local/bin/webservice lighttpd start --release trusty | 1 | | /usr/local/bin/webservice start uwsgi | 1 | | /usr/local/bin/webservice -v status | 1 | | /usr/local/bin/webservice2 start --release trusty | 1 | | /usr/local/bin/webservice help | 1 | | /usr/local/bin/webservice2 uwsgistatus | 1 | | /usr/local/bin/webservice strt uwsgi-python | 1 | | /usr/local/bin/webservice -help | 1 | | /usr/local/bin/webservice start --version=3 | 1 | | /usr/local/bin/webservice2 uwsgi-python restrat | 1 | | /usr/local/bin/webservice reset | 1 | | /usr/local/bin/webservice2 --release trusty restart | 1 | | /usr/local/bin/webservice --status | 1 | | /usr/local/bin/webservice restart uwsgi-python | 1 | | /usr/local/bin/webservice uwsgi start | 1 | | /usr/local/bin/webservice -release trusty on | 1 | | /usr/local/bin/webservice2 uwsgi-plain start -v | 1 | | /usr/local/bin/webservice2 nodejs top | 1 | | /usr/local/bin/webservice start --release precise -v | 1 | | /usr/local/bin/webservice --release=trusty start | 1 | | /usr/local/bin/webservice2 tomcat stop | 1 | | /usr/local/bin/webservice kill | 1 | | /usr/local/bin/webservice --release precise lighttpd start | 1 | | /usr/local/bin/webservice uwsgi-python step | 1 | +-----------------------------------------------------------------+-------+ 77 rows in set (5.75 sec)
Some of it is weird - for example, tools.jembot was responsible for 19487 of the webservice restarts in non-services hosts...
Change 285582 had a related patch set uploaded (by Yuvipanda):
Don't barf on --release for non-lighttpd usage
Change 285624 had a related patch set uploaded (by Yuvipanda):
Use webservice not webservice-new
Ok, so new webservice code is deployed and available in /usr/bin/webservice now! webservicemonitor will use it for version: 2 manifests.
Things left to do:
Change 285656 had a related patch set uploaded (by Yuvipanda):
tools: Switch to newer webservice comand
I've restarted all webgrid jobs and they're all running new code now! \o/
I need to now get rid of all the old webservice related code and we can call this done!
Change 285926 had a related patch set uploaded (by Yuvipanda):
tools: Remove all webservice related code