Page MenuHomePhabricator

Deploy new unified webservice code
Closed, ResolvedPublic

Description

Currently the code is a large number of shell scripts (webservice2, tool-lighttpd, lighttpd-started, tool-tomcat, tomcat-starter, tool-nodejs, tool-uwsgi-python, portgrabber, portgrabber.py, symlink to webservice) that all call each other and are written in a combination of python and bash.

Unify this into a single python package with a proper, nicer design.

Event Timeline

yuvipanda claimed this task.
yuvipanda raised the priority of this task from to Needs Triage.
yuvipanda updated the task description. (Show Details)
yuvipanda added a project: Toolforge.
yuvipanda added a subscriber: yuvipanda.
valhallasw triaged this task as Medium priority.May 10 2015, 7:50 PM
valhallasw added a subscriber: valhallasw.

Change 210196 had a related patch set uploaded (by Yuvipanda):
Initial commit

https://gerrit.wikimedia.org/r/210196

So ^ is the beginning of webservice-new for interactive use and webservice-runner for actually starting things on the webservice nodes. Also contains implementations of tool-nodejs and tool-python-uwsgi as appropriate extensible modules.

Ricordisamoa removed a project: Patch-For-Review.
Ricordisamoa set Security to None.
Ricordisamoa removed a subscriber: gerritbot.

I need to pick this back up again

To recap, the code is mostly done, but the big NFS outage in June took vastly bigger priority over this. Just needs some finishing touches and support in webservicemonitor and then we can switch to this.

After thinking about this some more -- should we spend time on this, or is it better to document which webservices use 'generic' (i.e. which webservices are not restarted by webservicewatcher) and then just move everything to k8s when it is ready?

yuvipanda renamed this task from Unify / simplify webservice code to Deploy new unified webservice code.Mar 21 2016, 1:05 AM

I spent some time making webservicemonitor work with webservice-new, and it does work fine now. webservice-new will add a 'version: 2' field in the service.manifest, and webservicemonitor will look for this.

Testing the different command invocations people use for it...

+-----------------------------------------------------------------+-------+
| event_commandline                                               | count |
+-----------------------------------------------------------------+-------+
| /usr/local/bin/webservice --release trusty uwsgi-python restart | 97985 |
| /usr/local/bin/webservice --release trusty nodejs restart       | 80363 |
| /usr/local/bin/webservice --release precise lighttpd restart    | 28658 |
| /usr/local/bin/webservice restart                               | 28077 |
| /usr/local/bin/webservice --release trusty lighttpd restart     | 24803 |
| /usr/local/bin/webservice uwsgi-python start                    | 10958 |
| /usr/local/bin/webservice uwsgi-python stop                     | 10914 |
| /usr/local/bin/webservice --release trusty generic restart      |  7345 |
| /usr/local/bin/webservice --release trusty uwsgi-plain restart  |  1908 |
| /usr/local/bin/webservice start                                 |   438 |
| /usr/local/bin/webservice2 uwsgi-python restart                 |   320 |
| /usr/local/bin/webservice stop                                  |   292 |
| /usr/local/bin/webservice status                                |   229 |
| /usr/local/bin/webservice2 uwsgi-python start                   |   216 |
| /usr/local/bin/webservice2 uwsgi-plain restart                  |   196 |
| /usr/local/bin/webservice2 uwsgi-python stop                    |   189 |
| /usr/local/bin/webservice uwsgi-python restart                  |   136 |
| /usr/local/bin/webservice2 tomcat start                         |    91 |
| /usr/local/bin/webservice2 uwsgi-plain stop                     |    90 |
| /usr/local/bin/webservice2 uwsgi-plain start                    |    72 |
| /usr/local/bin/webservice --release trusty tomcat restart       |    39 |
| /usr/local/bin/webservice2 stop                                 |    33 |
| /usr/local/bin/webservice2 start                                |    28 |
| /usr/local/bin/webservice2 restart                              |    22 |
| /usr/local/bin/webservice2 nodejs start                         |    18 |
| /usr/local/bin/webservice                                       |    18 |
| /usr/local/bin/webservice2 status                               |    16 |
| /usr/local/bin/webservice --release trusty restart              |    14 |
| /usr/local/bin/webservice2 uwsgi-python status                  |    11 |
| /usr/local/bin/webservice --release trusty start                |    11 |
| /usr/local/bin/webservice lighttpd restart                      |    11 |
| /usr/local/bin/webservice2 nodejs restart                       |    11 |
| /usr/local/bin/webservice2 nodejs stop                          |     9 |
| /usr/local/bin/webservice --help                                |     8 |
| /usr/local/bin/webservice start --release precise               |     7 |
| /usr/local/bin/webservice -h                                    |     6 |
| /usr/local/bin/webservice --release precise start               |     5 |
| /usr/local/bin/webservice --release trusty lighttpd start       |     4 |
| /usr/local/bin/webservice2                                      |     4 |
| /usr/local/bin/webservice python-uwsgi restart                  |     3 |
| /usr/local/bin/webservice2 uwsgi-plain status                   |     3 |
| /usr/local/bin/webservice uswsgi-python restart                 |     2 |
| /usr/local/bin/webservice uwsgi-python status                   |     2 |
| /usr/local/bin/webservice --release trusty lighttpd status      |     2 |
| /usr/local/bin/webservice start --release=precise               |     2 |
| /usr/local/bin/webservice --release trusty lighttpd stop        |     2 |
| /usr/local/bin/webservice --release trusty stop                 |     2 |
| /usr/local/bin/webservice lighttpd start                        |     2 |
| /usr/local/bin/webservice2 uwsgi-python restrart                |     2 |
| /usr/local/bin/webservice2 --help                               |     2 |
| /usr/local/bin/webservice2 uwsgi-python -h                      |     2 |
| /usr/local/bin/webservice statu                                 |     2 |
| /usr/local/bin/webservice2 uwsgi-python restrat                 |     1 |
| /usr/local/bin/webservice reset                                 |     1 |
| /usr/local/bin/webservice2 --release trusty restart             |     1 |
| /usr/local/bin/webservice --status                              |     1 |
| /usr/local/bin/webservice restart uwsgi-python                  |     1 |
| /usr/local/bin/webservice uwsgi start                           |     1 |
| /usr/local/bin/webservice -release trusty on                    |     1 |
| /usr/local/bin/webservice2 uwsgi-plain start -v                 |     1 |
| /usr/local/bin/webservice2 nodejs top                           |     1 |
| /usr/local/bin/webservice start --release precise -v            |     1 |
| /usr/local/bin/webservice --release=trusty start                |     1 |
| /usr/local/bin/webservice2 tomcat stop                          |     1 |
| /usr/local/bin/webservice kill                                  |     1 |
| /usr/local/bin/webservice --release precise lighttpd start      |     1 |
| /usr/local/bin/webservice uwsgi-python step                     |     1 |
| /usr/local/bin/webservice --release trusty                      |     1 |
| /usr/local/bin/webservice2 wsgi-python restrart                 |     1 |
| /usr/local/bin/webservice strt uwsgi-python                     |     1 |
| /usr/local/bin/webservice2 nodejs sart                          |     1 |
| /usr/local/bin/webservice2 -h                                   |     1 |
| /usr/local/bin/webservice lighttpd start --release trusty       |     1 |
| /usr/local/bin/webservice start uwsgi                           |     1 |
| /usr/local/bin/webservice -v status                             |     1 |
| /usr/local/bin/webservice2 start --release trusty               |     1 |
| /usr/local/bin/webservice help                                  |     1 |
| /usr/local/bin/webservice2 uwsgistatus                          |     1 |
| /usr/local/bin/webservice -help                                 |     1 |
| /usr/local/bin/webservice start --version=3                     |     1 |
+-----------------------------------------------------------------+-------+
80 rows in set (6.80 sec)

Without tools-services (aka webservicemonitor):

mysql:research@analytics-store.eqiad.wmnet [log]> select event_commandline, COUNT(*) as count FROM CommandInvocation_15243810 WHERE event_command = 'webservice' AND event_hostname NOT LIKE 'tools-services-%' GROUP BY event_commandline ORDER BY count DESC;
+-----------------------------------------------------------------+-------+
| event_commandline                                               | count |
+-----------------------------------------------------------------+-------+
| /usr/local/bin/webservice restart                               | 20487 |
| /usr/local/bin/webservice uwsgi-python start                    | 10958 |
| /usr/local/bin/webservice uwsgi-python stop                     | 10914 |
| /usr/local/bin/webservice start                                 |   438 |
| /usr/local/bin/webservice2 uwsgi-python restart                 |   320 |
| /usr/local/bin/webservice stop                                  |   292 |
| /usr/local/bin/webservice status                                |   229 |
| /usr/local/bin/webservice2 uwsgi-python start                   |   216 |
| /usr/local/bin/webservice2 uwsgi-plain restart                  |   196 |
| /usr/local/bin/webservice2 uwsgi-python stop                    |   189 |
| /usr/local/bin/webservice --release trusty nodejs restart       |   137 |
| /usr/local/bin/webservice uwsgi-python restart                  |   136 |
| /usr/local/bin/webservice2 tomcat start                         |    91 |
| /usr/local/bin/webservice2 uwsgi-plain stop                     |    90 |
| /usr/local/bin/webservice --release trusty uwsgi-python restart |    75 |
| /usr/local/bin/webservice2 uwsgi-plain start                    |    72 |
| /usr/local/bin/webservice2 stop                                 |    33 |
| /usr/local/bin/webservice2 start                                |    28 |
| /usr/local/bin/webservice2 restart                              |    22 |
| /usr/local/bin/webservice2 nodejs start                         |    18 |
| /usr/local/bin/webservice                                       |    18 |
| /usr/local/bin/webservice2 status                               |    16 |
| /usr/local/bin/webservice --release trusty restart              |    14 |
| /usr/local/bin/webservice --release trusty start                |    11 |
| /usr/local/bin/webservice2 nodejs restart                       |    11 |
| /usr/local/bin/webservice lighttpd restart                      |    11 |
| /usr/local/bin/webservice2 uwsgi-python status                  |    11 |
| /usr/local/bin/webservice2 nodejs stop                          |     9 |
| /usr/local/bin/webservice --help                                |     8 |
| /usr/local/bin/webservice --release trusty lighttpd restart     |     7 |
| /usr/local/bin/webservice start --release precise               |     7 |
| /usr/local/bin/webservice -h                                    |     6 |
| /usr/local/bin/webservice --release precise start               |     5 |
| /usr/local/bin/webservice2                                      |     4 |
| /usr/local/bin/webservice --release trusty lighttpd start       |     4 |
| /usr/local/bin/webservice --release precise lighttpd restart    |     4 |
| /usr/local/bin/webservice2 uwsgi-plain status                   |     3 |
| /usr/local/bin/webservice python-uwsgi restart                  |     3 |
| /usr/local/bin/webservice start --release=precise               |     2 |
| /usr/local/bin/webservice --release trusty lighttpd stop        |     2 |
| /usr/local/bin/webservice --release trusty stop                 |     2 |
| /usr/local/bin/webservice lighttpd start                        |     2 |
| /usr/local/bin/webservice2 uwsgi-python restrart                |     2 |
| /usr/local/bin/webservice2 --help                               |     2 |
| /usr/local/bin/webservice2 uwsgi-python -h                      |     2 |
| /usr/local/bin/webservice statu                                 |     2 |
| /usr/local/bin/webservice uswsgi-python restart                 |     2 |
| /usr/local/bin/webservice uwsgi-python status                   |     2 |
| /usr/local/bin/webservice --release trusty lighttpd status      |     2 |
| /usr/local/bin/webservice --release trusty                      |     1 |
| /usr/local/bin/webservice2 wsgi-python restrart                 |     1 |
| /usr/local/bin/webservice2 nodejs sart                          |     1 |
| /usr/local/bin/webservice2 -h                                   |     1 |
| /usr/local/bin/webservice lighttpd start --release trusty       |     1 |
| /usr/local/bin/webservice start uwsgi                           |     1 |
| /usr/local/bin/webservice -v status                             |     1 |
| /usr/local/bin/webservice2 start --release trusty               |     1 |
| /usr/local/bin/webservice help                                  |     1 |
| /usr/local/bin/webservice2 uwsgistatus                          |     1 |
| /usr/local/bin/webservice strt uwsgi-python                     |     1 |
| /usr/local/bin/webservice -help                                 |     1 |
| /usr/local/bin/webservice start --version=3                     |     1 |
| /usr/local/bin/webservice2 uwsgi-python restrat                 |     1 |
| /usr/local/bin/webservice reset                                 |     1 |
| /usr/local/bin/webservice2 --release trusty restart             |     1 |
| /usr/local/bin/webservice --status                              |     1 |
| /usr/local/bin/webservice restart uwsgi-python                  |     1 |
| /usr/local/bin/webservice uwsgi start                           |     1 |
| /usr/local/bin/webservice -release trusty on                    |     1 |
| /usr/local/bin/webservice2 uwsgi-plain start -v                 |     1 |
| /usr/local/bin/webservice2 nodejs top                           |     1 |
| /usr/local/bin/webservice start --release precise -v            |     1 |
| /usr/local/bin/webservice --release=trusty start                |     1 |
| /usr/local/bin/webservice2 tomcat stop                          |     1 |
| /usr/local/bin/webservice kill                                  |     1 |
| /usr/local/bin/webservice --release precise lighttpd start      |     1 |
| /usr/local/bin/webservice uwsgi-python step                     |     1 |
+-----------------------------------------------------------------+-------+
77 rows in set (5.75 sec)

Some of it is weird - for example, tools.jembot was responsible for 19487 of the webservice restarts in non-services hosts...

Change 285582 had a related patch set uploaded (by Yuvipanda):
Don't barf on --release for non-lighttpd usage

https://gerrit.wikimedia.org/r/285582

Just sent out an announcement email saying I'm doing this today \o/

Change 285582 merged by Yuvipanda:
Don't barf on --release for non-lighttpd usage

https://gerrit.wikimedia.org/r/285582

Change 285624 had a related patch set uploaded (by Yuvipanda):
Use webservice not webservice-new

https://gerrit.wikimedia.org/r/285624

Ok, so new webservice code is deployed and available in /usr/bin/webservice now! webservicemonitor will use it for version: 2 manifests.

Things left to do:

  1. Replace current /usr/local/bin/webservice with symlink
  2. Restart all webservices so they start using the new setup!

Change 285656 had a related patch set uploaded (by Yuvipanda):
tools: Switch to newer webservice comand

https://gerrit.wikimedia.org/r/285656

Change 285656 merged by Yuvipanda:
tools: Switch to newer webservice comand

https://gerrit.wikimedia.org/r/285656

Change 285624 merged by Yuvipanda:
Use webservice not webservice-new

https://gerrit.wikimedia.org/r/285624

I've restarted all webgrid jobs and they're all running new code now! \o/

I need to now get rid of all the old webservice related code and we can call this done!

Change 285926 had a related patch set uploaded (by Yuvipanda):
tools: Remove all webservice related code

https://gerrit.wikimedia.org/r/285926

Change 285926 merged by Yuvipanda:
tools: Remove all webservice related code

https://gerrit.wikimedia.org/r/285926

I'm going to call this done! \o/