Page MenuHomePhabricator

Help to run Java8 web-app
Closed, ResolvedPublic

Description

Hello, first of all sorry if this is not the appropriate channel to make this kind of questions.

I am maintaining a tool for some years in Toolforge coded in Java 7. This tool is a webapp, it exposes a simple web interface to interact with the tool, so it is deployed in the Tomcat server. I use Java 7 because it is the supported version (https://wikitech.wikimedia.org/wiki/Help:Toolforge/Java#Java_versions).

I wonder if it is possible to upgrade the tool to use Java 8. I have tried with the jdk8 Kubernetes service (https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Play_and_similar_JVM-based_frameworks), but I am really lost. Once in the jdk8 shell:

  • When needed, I point directly to the Java8 executable (T205774).
  • Not sure if it is possible to run a Tomcat with Java8
  • I have modified the app to be an auto-executable JAR and it deploys, but I cannot achieve to access the webapp from outside.

Running the app in Java7 is not so bad (I maintain worse thinks at work), but it is a little older and lack some cool features (https://en.wikipedia.org/wiki/Java_version_history).

Thanks a lot for your help,

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 30 2018, 9:11 AM

What have you tried and what error messages you get?

First I have "logged in" the JDK8 container:

webservice --backend=kubernetes jdk8 shell

Tomcat is not available:

tools.replacer@interactive:~$ webservice tomcat status
Traceback (most recent call last):
  File "/usr/bin/webservice", line 199, in <module>
    if job.get_state() != Backend.STATE_STOPPED:
  File "/usr/lib/python2.7/dist-packages/toollabs/webservice/backends/gridenginebackend.py", line 101, in get_state
    job = self._get_job_xml()
  File "/usr/lib/python2.7/dist-packages/toollabs/webservice/backends/gridenginebackend.py", line 61, in _get_job_xml
    output = subprocess.check_output(['qstat', '-xml'])
  File "/usr/lib/python2.7/subprocess.py", line 212, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 390, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1024, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

I cannot run Tomcat from the executable either:

tools.replacer@interactive:~$ ./public_tomcat/bin/startup.sh 
./public_tomcat/bin/startup.sh: 3: ./public_tomcat/bin/startup.sh: /usr/share/tomcat7/bin/startup.sh: not found

Then I have packaged my web-app as a runnable Spring Boot JAR with embebed Tomcat, and run it. It deploys correctly with no special error message, but I cannot access the application from the outside as usual, maybe I need some special configuration of the port or similar?

Thanks in advance,

tools.replacer@interactive:~$ webservice tomcat status
Traceback (most recent call last):
[...]
  File "/usr/lib/python2.7/dist-packages/toollabs/webservice/backends/gridenginebackend.py", line 61, in _get_job_xml
    output = subprocess.check_output(['qstat', '-xml'])
[...]
OSError: [Errno 2] No such file or directory

Try with --backend=kubernetes. It's querying the grid with qstat.

Try with --backend=kubernetes. It's querying the grid with qstat.

Thank you for your help. It seems that Tomcat is not available as an option in kubernetes:

tools.replacer@interactive:~$ webservice --backend=kubernetes tomcat status
type must be one of  golang,jdk8,nodejs,php5.6,php7.2,python,python2,ruby2,tcl

Is tomcat one of 'Play Framework projects (and other JVM-based projects that have one executable to start the application)'? If not,it cannot be run on k8s, but on grid. (I don't really know how tomcat works):

The old trusty servers run java 7:

$ java -version
java version "1.7.0_181"
OpenJDK Runtime Environment (IcedTea 2.6.14) (7u181-2.6.14-0ubuntu0.3)
OpenJDK 64-Bit Server VM (build 24.181-b01, mixed mode)

but the newer stretch servers run java 11:

$ java --version
openjdk 11.0.1 2018-10-16
OpenJDK Runtime Environment (build 11.0.1+13-Debian-2bpo91)
OpenJDK 64-Bit Server VM (build 11.0.1+13-Debian-2bpo91, mixed mode, sharing)

Does that suit your needs?

Tomcat is an application server which is already pre-configured on grid, so after your answer I am afraid that you are confirming me that it cannot be used on k8s (yet).

On the other side, I can configure my Java web-app to run as one executable (not Play Framework, but Spring Boot, which I suppose it is kind of similar). However, after executing the JAR (Java executable), I cannot access the web-app from the outside. I have tried several times tweaking the values of the port and the context (as in the example for Play Framework) with no success. Any additional help with these values or others ?

I am really not interested in configuring my tool to run in Play Framework. So if I am not able to deploy a self-executing Java Spring Boot web-app, I will keep on running the web-app in the good-old Java 7 Tomcat.

In any case, thanks a lot for your help,

Sorry, my focus was on 'JVM-based projects that have one executable to start the application' part, not specifically Play Framework. When you execute the JAR, is it executed on the bastion or in a k8s instance? It must be executed on k8s for the routing to work.

I may also be wrong in thinking of requiring single-executable; I was pinged on IRC that in theory Tomcat should be able to run on k8s anyways. Do you have some code I could experiment with to setup Tomcat webservice on k8s?

Also, I might have been unclear that there are two grids available right now. One is the old grid of Ubuntu Trusty with Java 7, and this grid will retire in a few months; another is on Debian Stretch with Java 11. I was wondering Tomcat works on the new grid.

Also, I might have been unclear that there are two grids available right now. One is the old grid of Ubuntu Trusty with Java 7, and this grid will retire in a few months; another is on Debian Stretch with Java 11. I was wondering Tomcat works on the new grid.

These are good news. For the moment, I have logged in the Stretch machine and tried (with no success):

  • Compile the project with Maven (issue with the number of threads)
  • Prepare the Tomcat folder (running setup_tomcat)

On the other hand, I usually compile the project (and start the Tomcat server) not in the login machine, but in the dev one: dev.tools.wmflabs.org Do you know if there will be a Stretch alternative for this machine ?

Do you know if there will be a Stretch alternative for this machine ?

There will be, probably. I don't see an alternate bastion built yet.

For the moment, I have logged in the Stretch machine and tried (with no success):

Do you have the commands and the error message?

For the moment, I have logged in the Stretch machine and tried (with no success):

Do you have the commands and the error message?

For the Maven command:

  • It works if I compile the code for Java 7
  • It fails if I compile the code for Java 8:
tools.replacer@tools-sgebastion-06:~/replacer$ mvn clean package -DskipTests
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/usr/share/maven/lib/guice.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building replacer 1.6.3
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:3.0.0:clean (default-clean) @ replacer ---
[INFO] Deleting /mnt/nfs/labstore-secondary-tools-project/replacer/replacer/target
[INFO] 
[INFO] --- maven-resources-plugin:3.0.2:resources (default-resources) @ replacer ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 22 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ replacer ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 50 source files to /mnt/nfs/labstore-secondary-tools-project/replacer/replacer/target/classes
[INFO] 
[INFO] --- maven-resources-plugin:3.0.2:testResources (default-testResources) @ replacer ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.7.0:testCompile (default-testCompile) @ replacer ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 25 source files to /mnt/nfs/labstore-secondary-tools-project/replacer/replacer/target/test-classes
[INFO] 
[INFO] --- maven-surefire-plugin:2.21.0:test (default-test) @ replacer ---
[12.912s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11.619 s
[INFO] Finished at: 2019-01-30T09:37:17+00:00
[INFO] Final Memory: 44M/164M
[INFO] ------------------------------------------------------------------------
[ERROR] unable to create native thread: possibly out of memory or process/resource limits reached -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError
[13.010s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.

On the other hand, when I try to setup the Tomcat instance, via command setup-tomcat, there is no output, the command simply takes a long long time with no response, and I have to kill it manually via Ctrl + C.

Thanks a lot for your help,

[12.912s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.

EAGAIN 11 Resource temporarily unavailable

EAGAIN Insufficient resources to create another thread.

EAGAIN A  system-imposed limit on the number of threads was encountered.  There are a number of limits that may trigger this error:
       the RLIMIT_NPROC soft resource limit (set via setrlimit(2)), which limits the number of processes and  threads  for  a  real
       user  ID,  was reached; the kernel's system-wide limit on the number of processes and threads, /proc/sys/kernel/threads-max,
       was reached (see proc(5)); or the maximum number of PIDs, /proc/sys/kernel/pid_max, was reached (see proc(5)).

The error code does seem to suggest running out of s certain resource, but I don't see what that could be.

Trusty:

tools.zhuyifei1999-test@tools-bastion-02:~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 63807
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 512
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) 43200
max user processes              (-u) 30
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Stretch:

tools.zhuyifei1999-test@tools-sgebastion-06:~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 64113
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 512
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) 43200
max user processes              (-u) 30
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Memory is unlimited and process is 30 which should be enough for normal usage. How many processes & threads does maven create? Also, you you mind if I run that maven command as your tool to debug it?

I have eventually been able to run the Maven packaging by reducing the amount of memory to be used:

export MAVEN_OPTS="-Xmx128m"

On the other hand, I cannot create a Tomcat environment.

It seems the setup-tomcat script has not been updated:

tools.replacer@tools-sgebastion-07:~/replacer$ cat /usr/bin/setup-tomcat 
#! /bin/bash

prefix=$(/bin/cat /etc/wmflabs-project)
tool=$(/usr/bin/id -nu|sed -e "s/^$prefix.//")
user="$prefix.$tool"

home=$(getent passwd $user | cut -d : -f 6 | sed -e 's/\/$//')
if [ "$(getent group $user | cut -d : -f 1)" != "$user" ]; then
    echo "$0: $tool does not appear to be a tool" >&2
    exit 1
fi

if [ -d "$home/public_tomcat" ]; then
    echo "This tool already has a public_tomcat directory" >&2
    exit 1
fi

echo "Setting up your public_tomcat directory..."
/usr/bin/qsub -sync y -o /dev/null -e /dev/null -i /dev/null -q "webgrid-generic" -l h_vmem=512m -l release=trusty -b y -N "setup-$tool" tomcat7-instance-create public_tomcat >/dev/null

conf="$home/public_tomcat/conf/server.xml"
echo "All done."
echo "You can edit the configuration in $conf as needed."
bd808 assigned this task to Benjavalero.Feb 22 2019, 9:29 PM
bd808 closed this task as Resolved.
bd808 added a subscriber: bd808.

Copying comments from related task:

In my case, my Java tool replacer is prepared to be packaged as a WAR to be run in Tomcat, or as an SpringBoot standalone JAR.

I have been running the tool as a WAR so far. On the other side, when running the tool as a JAR in the lab server, I am not able to access it from the outside (https://tools.wmflabs.org/replacer/), I guess because of the port used.

Any hint about the port to use so the tool can be accessed in a regular browser ?

In my case, my Java tool replacer is prepared to be packaged as a WAR to be run in Tomcat, or as an SpringBoot standalone JAR.

I have been running the tool as a WAR so far. On the other side, when running the tool as a JAR in the lab server, I am not able to access it from the outside (https://tools.wmflabs.org/replacer/), I guess because of the port used.

Any hint about the port to use so the tool can be accessed in a regular browser ?

For a job grid webservice, a PORT environment variable will be exported by the webservice-runner process that is spawned to start your webservice.

For a Kubernetes webservice, the expected port is always 8000.

I have tried the port 8000 with no success.

I have started the Kubernetes JDK8 shell:

webservice --backend=kubernetes jdk8 shell

Then I have started the standalone JAR with an embedded Tomcat server:

java -jar replacer.jar

In the log displayed, I can read the following message:

Tomcat started on port(s): 8000 (http)

But when I try to access the tool as usual in https://tools.wmflabs.org/replacer/ :

The URI you have requested, /replacer/, is not currently serviced.

I have tried the port 8000 with no success.

I have started the Kubernetes JDK8 shell:

webservice --backend=kubernetes jdk8 shell

The shell action does not set the attributes on the Kubernetes pod which are required for our ingress solution to connect the frontend proxy to the pod. It took a bit of digging for me to figure out how the jdk8 backend is expected to work. Eventually I determined that it is setup to use our "generic" startup mechanism.

This process should work to start your tool as a Kubernetes webservice:

  • Create a wrapper script that will start your java process. I'm going to put this example at $HOME/replace.sh in the test environment I'm using: `name=$HOME/replacer.sh, lang=sh #!/bin/sh cd $HOME exec java -jar $HOME/replacer.jar `
  • Tell webservice to run this script inside a jdk8 container: ` $ webservice --backend=kubernetes jdk8 start $HOME/replacer.sh `

    It would be a bit nicer if you could give the whole java -jar replacer.jar command directly to webservice, but the options parsing the webservice script is doing right now has problems with any "extra" args that start with '-'.

The proposed solution has worked for me, so I will stop working with Tomcat. This will allow me also to finally migrate to Java 8.

For the record, besides changing the port, I have had to also change the context path in the Spring Boot configuration file:

server:
  port: 8000
  servlet.context-path: /replacer/

Thanks a lot!