Page MenuHomePhabricator

Provide a migration path for tools running tomcat
Open, LowPublic

Description

Currently we have 9 3 tools running tomcat webservices. In the long term we'd like to not be responsible for maintaining tomcat - need to find an alternative that works for all these tools. Ideally they'd use an embedded HTTP server like Jetty (for new users) and for older ones we find a equivalent that can work.

Running tomcat job list as of 2021-05-24T18:13Z based on https://sge-status.toolforge.org/ and the "tomcat-..." naming convention:

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

For folks who just have a war file that they want to run, https://www.eclipse.org/jetty/documentation/9.4.x/runner.html might be a lighter and easier to script replacement for tomcat.

scfc triaged this task as Low priority.Feb 16 2017, 7:45 PM
scfc moved this task from Triage to Backlog on the Toolforge board.

In my case, my Java tool replacer is prepared to be packaged as a WAR to be run in Tomcat, or as an SpringBoot standalone JAR.

I have been running the tool as a WAR so far. On the other side, when running the tool as a JAR in the lab server, I am not able to access it from the outside (https://tools.wmflabs.org/replacer/), I guess because of the port used.

Any hint about the port to use so the tool can be accessed in a regular browser ?

In my case, my Java tool replacer is prepared to be packaged as a WAR to be run in Tomcat, or as an SpringBoot standalone JAR.

I have been running the tool as a WAR so far. On the other side, when running the tool as a JAR in the lab server, I am not able to access it from the outside (https://tools.wmflabs.org/replacer/), I guess because of the port used.

Any hint about the port to use so the tool can be accessed in a regular browser ?

For a job grid webservice, a PORT environment variable will be exported by the webservice-runner process that is spawned to start your webservice.

For a Kubernetes webservice, the expected port is always 8000.

I have tried the port 8000 with no success.

I have started the Kubernetes JDK8 shell:

webservice --backend=kubernetes jdk8 shell

Then I have started the standalone JAR with an embedded Tomcat server:

java -jar replacer.jar

In the log displayed, I can read the following message:

Tomcat started on port(s): 8000 (http)

But when I try to access the tool as usual in https://tools.wmflabs.org/replacer/ :

The URI you have requested, /replacer/, is not currently serviced.

I have tried the port 8000 with no success.

I have started the Kubernetes JDK8 shell:

webservice --backend=kubernetes jdk8 shell

The shell action does not set the attributes on the Kubernetes pod which are required for our ingress solution to connect the frontend proxy to the pod. It took a bit of digging for me to figure out how the jdk8 backend is expected to work. Eventually I determined that it is setup to use our "generic" startup mechanism.

This process should work to start your tool as a Kubernetes webservice:

  • Create a wrapper script that will start your java process. I'm going to put this example at $HOME/replace.sh in the test environment I'm using:
$HOME/replacer.sh
#!/bin/sh
cd $HOME
exec java -jar $HOME/replacer.jar
  • Tell webservice to run this script inside a jdk8 container:
$ webservice --backend=kubernetes jdk8 start $HOME/replacer.sh

It would be a bit nicer if you could give the whole java -jar replacer.jar command directly to webservice, but the options parsing the webservice script is doing right now has problems with any "extra" args that start with '-'.

The proposed solution has worked for me, so I will stop working with Tomcat. This will allow me also to finally migrate to Java 8.

For the record, besides changing the port, I have had to also change the context path in the Spring Boot configuration file:

server:
  port: 8000
  servlet.context-path: /replacer/

Thanks a lot!

Is there anything left to do here? @bd808 is there some magic not documented anywhere to see if we have any tomcat tools remaining?

There are four tomcat-xxx jobs currently running (sge-status), mine included.

Tool pbbot has been moved to k8s. I'm a bit surprised that it stays well within default memory limits (512 Mi), while this would not be possible in sge (needs 5-6 Gi).

Tool pbbot has been moved to k8s. I'm a bit surprised that it stays well within default memory limits (512 Mi), while this would not be possible in sge (needs 5-6 Gi).

Glad to hear that!

Part of the problem with JVM based tools on the grid engine is that grid jobs are not contained in a cgroup or any other operating system level constraint. This in turn means that the JVM thinks that it has all the cpu cores and ram of the underlying grid exec node to itself, and so it asks the kernel for resources based on that assumption. When running on Kubernetes, the JVM process is started within a Kubernetes managed container (currently Docker) which means that when the JVM interrogates it's environment it sees the limits set for that container and responds appropriately by requesting resources that "fit" inside the container's limits.