Page MenuHomePhabricator

Stretch grid problem: cannot migrate tomcat webservice
Closed, ResolvedPublic

Description

While migrating isbn to the Stretch job grid I ran into this problem:

tomcat is running fine on trusty:

$ webservice status
Your webservice of type tomcat is running
$ curl -s -i https://tools.wmflabs.org/isbn/ | head -1
HTTP/1.1 200 OK

After stopping the webservice on trusty, I tried to start it on stretch, but it dies after the start:

$ webservice --backend=gridengine tomcat start
Starting webservice............
$ webservice status
Your webservice is not running

(Going back to trusty, I can start the webservice without problems.)

Log:

$ tail -1 service.log
2019-03-01T12:02:59.037776 No running webservice job found, attempting to start it

Event Timeline

From $HOME/error.log for the isbn tool:

Cannot find /usr/share/tomcat7/bin/setclasspath.sh
This file is needed to run this program
Cannot find /usr/share/tomcat7/bin/setclasspath.sh
This file is needed to run this program

The Debian Stretch grid has tomcat8 installed rather than tomcat7. We need to update the setup-tomcat initialization script and add instructions on migrating an existing tomcat7 setup. Alternately, we need to find a proper solution for T141396: Provide a migration path for tools running tomcat.

Change 494641 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[labs/toollabs@master] misctools: update setup=tomcat for Stretch grid

https://gerrit.wikimedia.org/r/494641

Change 494641 merged by jenkins-bot:
[labs/toollabs@master] misctools: update setup-tomcat for Stretch grid

https://gerrit.wikimedia.org/r/494641

Mentioned in SAL (#wikimedia-cloud) [2019-03-07T00:34:16Z] <zhuyifei1999_> begin building misctools 1.37 using debuild T217406

Mentioned in SAL (#wikimedia-cloud) [2019-03-07T00:38:29Z] <zhuyifei1999_> published misctools 1.37 T217406

tools.zhuyifei1999-test@tools-sgebastion-08:~$ setup-tomcat
Setting up your public_tomcat directory...
All done.
You can edit the configuration in /data/project/zhuyifei1999-test/public_tomcat/conf/server.xml as needed.
tools.zhuyifei1999-test@tools-sgebastion-08:~$ tree public_tomcat
public_tomcat
├── bin
│   ├── setenv.sh
│   ├── shutdown.sh
│   └── startup.sh
├── conf
│   ├── catalina.properties
│   ├── context.xml
│   ├── jaspic-providers.xml
│   ├── logging.properties
│   ├── server.xml
│   ├── tomcat-users.xml
│   └── web.xml
├── logs
├── temp
├── webapps
└── work

6 directories, 10 files

Mentioned in SAL (#wikimedia-cloud) [2019-03-07T00:49:49Z] <zhuyifei1999_> clushed misctools 1.37 upgrade on @bastion,@cron,@bastion-stretch T217406

@Ireas I think you should be able to move your Trusty/tomcat7 $HOME/public_tomcat out of the way and then use setup-tomcat from login.tools.wmflabs.org (now the Stretch bastion) to build a new Stretch/tomcat8 $HOME/public_tomcat now.

@bd808 Thanks for looking into this! There seems to be a problem with the available memory, see this error log:

Mentioned in SAL (#wikimedia-cloud) [2019-03-13T00:22:27Z] <bd808> Raise web-memlimit for isbn tool to 6G for tomcat8 (T217406)

Mentioned in SAL (#wikimedia-cloud) [2019-03-13T00:22:27Z] <bd808> Raise web-memlimit for isbn tool to 6G for tomcat8 (T217406)

@Ireas I have added a config file (/data/project/.system/config/isbn.web-memlimit) that will give your isbn tool's webservice a 6G memory limit. Give that a try. We can raise it a bit more if that is not enough to make the JVM and Tomcat happy.

It seems to start succesfully now. Thanks for your help!