Page MenuHomePhabricator

Webservice for vcat does not start (Tomcat on Kubernetes) since 2020-02-10 due to "unrecognized arguments" error
Closed, ResolvedPublic

Description

I run the vcat tool in my own Tomcat installation using Kubernetes. All I configured was for the Tomcat to listen on port 8000, and I use the following command to start it:

webservice --backend=kubernetes jdk8 start '/data/project/vcat/tomcat/bin/catalina.sh run'

This has been working fine for months, until yesterday. According to the access logs the last request was served at 10/Feb/2020 21:45:12 UTC.

The output from "kubectl logs ..." shows this error message:

usage: webservice-runner [-h]
                         [--type {tomcat,nodejs,generic,lighttpd,lighttpd-plain,uwsgi-plain,uwsgi-python}]
                         [--extra_args EXTRA_ARGS] [--port PORT]
                         [--register-proxy]
webservice-runner: error: unrecognized arguments: d a t a / p r o j e c t / v c a t / t o m c a t / b i n / c a t a l i n a . s h   r u n

Looks like somehow the parameters are not handed over to webservice-runner correctly. service.manifest seems fine to me:

# This file is used by Toolforge infrastructure.
# Please do not edit manually at this time.
backend: kubernetes
distribution: debian
version: 2
web: jdk8
web::extra_args: /data/project/vcat/tomcat/bin/catalina.sh run

I have tried the following so far, with no noticable changes:

  • Using jdk11 instead of jdk8.
  • Creating a separate start script that requires no parameters.

According to "webservice status" the webservice is supposed to be running right now and I have disabled all cron jobs that would attempt to restart it.

Event Timeline

Dapete created this task.Feb 11 2020, 5:24 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 11 2020, 5:24 PM

I have tried to start it again, still the same error message.

doctaxon added a subscriber: doctaxon.EditedFeb 22 2020, 7:58 PM

@Aklapper Das Tool wird häufig genutzt und ist auch nicht unwichtig. Kannst Du diesen Task bitte assignen und/oder die Tags anpassen? Danke sehr ...

Aklapper renamed this task from Webservice for vcat does not start (Tomcat on Kubernetes) to Webservice for vcat does not start (Tomcat on Kubernetes) since 2020-02-10 due to "unrecognized arguments" error.Feb 22 2020, 9:05 PM

@doctaxon: Hi, people decide on their own what they plan to work on so I am not going to assign tasks, and the tags look correct to me.

Change 574236 had a related patch set uploaded (by Dapete; owner: Dapete):
[operations/software/tools-webservice@master] Partially revert changes to improve support for extra_args

https://gerrit.wikimedia.org/r/574236

The cause seems to be this change: https://gerrit.wikimedia.org/r/#/c/operations/software/tools-webservice/+/496567/

It is supposed to allow multiple extra_args as an array and changes the existing .append(...) calls to .extend(...). I assume it is still a string, so .extend(...) iterates over each character and adds them as individual parameters. This would explain the spaces in the error message.

However, only gridenginebackend.py and kubernetesbackend.py were changed, and while I know know next to nothing about how all this works, I believe at least the webservice and webservice-runner scripts would have to be changed to support this a well. I have tried it with the syntax as descriped in the commit message, this does not work (the webservice script fails immediately).

It worked once when I manually changed web::extra_args into an array in service.manifest but I have not been able to repeat this.

I have pushed a Gerrit change to revert back from .extent(...) to .append(...) in the two files that were changed: https://gerrit.wikimedia.org/r/#/c/operations/software/tools-webservice/+/574236/

bd808 claimed this task.Feb 27 2020, 1:31 AM
bd808 edited projects, added cloud-services-team (Kanban); removed Patch-For-Review.

Change 575110 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/software/tools-webservice@master] 3rd try at making extra_args handling "better"

https://gerrit.wikimedia.org/r/575110

Change 575110 merged by jenkins-bot:
[operations/software/tools-webservice@master] 3rd try at making extra_args handling "better"

https://gerrit.wikimedia.org/r/575110

Change 574236 abandoned by BryanDavis:
Partially revert changes to improve support for extra_args

Reason:
Superseded by If7c3c9ba10db3a151c99b64bcf67fdf2f00a0317 and If933ead6e2b16b9f3914d67ec5fa27d576a4915d. I apologize for not seeing this patch and the related bug report sooner.

https://gerrit.wikimedia.org/r/574236

Bstorm added a subscriber: Bstorm.Feb 27 2020, 11:25 PM

@Dapete I've got vcat running after using it as the test case for @bd808's latest code. I'm deploying the changes now. From here in you'll want to not use quotes around the end of the command, making it more like webservice --backend=kubernetes jdk8 start /data/project/vcat/tomcat/bin/catalina.sh run

Bstorm triaged this task as High priority.Feb 27 2020, 11:35 PM
bd808 closed this task as Resolved.Mar 8 2020, 4:11 AM