Page MenuHomePhabricator

"webservice shell" fails with "No such file or directory" (with php5.6)
Closed, ResolvedPublic

Description

tools.templatecheck@tools-bastion-03:~$ webservice --backend=kubernetes php5.6 start
Starting webservice..
tools.templatecheck@tools-bastion-03:~$ webservice --backend=kubernetes php5.6 shell
Traceback (most recent call last):
  File "/usr/local/bin/webservice", line 163, in <module>
    job.shell()
  File "/usr/lib/python2.7/dist-packages/toollabs/webservice/backends/kubernetesbackend.py", line 390, in shell
    'interactive'
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
tools.templatecheck@tools-bastion-03:~$ kubectl get pods
NAME                             READY     STATUS    RESTARTS   AGE
interactive                      1/1       Running   0          6s
templatecheck-3430673173-cycan   1/1       Running   0          11s
tools.templatecheck@tools-bastion-03:~$

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
scfc triaged this task as Medium priority.Jan 30 2017, 5:33 AM
scfc updated the task description. (Show Details)

(Copied and pasted verbatim from a new try to avoid any confusion.)

I got a somehow-more-verbose error:

tools.precise-tools@tools-bastion-02:~$ webservice --backend=kubernetes python2 shell
Traceback (most recent call last):
  File "/usr/local/bin/webservice", line 163, in <module>
    job.shell()
  File "/usr/lib/python2.7/dist-packages/toollabs/webservice/backends/kubernetesbackend.py", line 390, in shell
    'interactive'
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/apport_python_hook.py", line 145, in apport_excepthook
    os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o640), 'wb') as f:
OSError: [Errno 2] No such file or directory: '/var/crash/_usr_bin_webservice.53276.crash'

Original exception was:
Traceback (most recent call last):
  File "/usr/local/bin/webservice", line 163, in <module>
    job.shell()
  File "/usr/lib/python2.7/dist-packages/toollabs/webservice/backends/kubernetesbackend.py", line 390, in shell
    'interactive'
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Happens for older tools as well, eg. video2commons-test. CC @yuvipanda

Idk why the line numbers are different, but it's probably a subprocess.Popen call that does not seem to have anything special.

zhuyifei1999 raised the priority of this task from Medium to High.
zhuyifei1999 added a project: Tools-Kubernetes.

Testing Popen on /bin/ls works fine, but the original fails, so I suspected /usr/local/bin/kubectl is wrong. Indeed:

(venv)tools.video2commons-test@tools-bastion-02:~$ which kubectl
/usr/bin/kubectl
(venv)tools.video2commons-test@tools-bastion-02:~$ ls /usr/local/bin/kubectl
ls: cannot access /usr/local/bin/kubectl: No such file or directory

kubectl is /usr/bin/kubectl is across all 3 bastions, 02, 03 & 05.

Change 334978 had a related patch set uploaded (by Zhuyifei1999):
kubernetesbackend: change absolute kubectl path to '/usr/bin/kubectl'

https://gerrit.wikimedia.org/r/334978

Change 334978 merged by jenkins-bot:
kubernetesbackend: change absolute kubectl path to '/usr/bin/kubectl'

https://gerrit.wikimedia.org/r/334978

Mentioned in SAL (#wikimedia-labs) [2017-02-07T13:49:32Z] <scfc_de> Deployed toollabs-webservice_0.33_all.deb (T156605, T156626).