"webservice shell" fails with "No such file or directory" (with php5.6)
Closed, ResolvedPublic
Actions

Assigned To

zhuyifei1999

Authored By

	scfc
	Jan 30 2017, 5:33 AM

Description

tools.templatecheck@tools-bastion-03:~$ webservice --backend=kubernetes php5.6 start
Starting webservice..
tools.templatecheck@tools-bastion-03:~$ webservice --backend=kubernetes php5.6 shell
Traceback (most recent call last):
  File "/usr/local/bin/webservice", line 163, in <module>
    job.shell()
  File "/usr/lib/python2.7/dist-packages/toollabs/webservice/backends/kubernetesbackend.py", line 390, in shell
    'interactive'
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
tools.templatecheck@tools-bastion-03:~$ kubectl get pods
NAME                             READY     STATUS    RESTARTS   AGE
interactive                      1/1       Running   0          6s
templatecheck-3430673173-cycan   1/1       Running   0          11s
tools.templatecheck@tools-bastion-03:~$

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	kubernetesbackend: change absolute kubectl path to '/usr/bin/kubectl'	operations/software/tools-webservice	master	+7 -1

Customize query in gerrit

Related Objects

Mentioned In: T156626: k8s webservice restart failure with `ValueError: get() more than one object; use filter`
T156386: Create a wall for tools migration to trusty
Mentioned Here: T156626: k8s webservice restart failure with `ValueError: get() more than one object; use filter`

Event Timeline

scfc created this task.Jan 30 2017, 5:33 AM

Restricted Application added a project: Cloud-Services. · View Herald TranscriptJan 30 2017, 5:33 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

scfc triaged this task as Medium priority.Jan 30 2017, 5:33 AM

scfc updated the task description. (Show Details)

(Copied and pasted verbatim from a new try to avoid any confusion.)

zhuyifei1999 mentioned this in T156386: Create a wall for tools migration to trusty.Jan 30 2017, 8:21 AM

I got a somehow-more-verbose error:

tools.precise-tools@tools-bastion-02:~$ webservice --backend=kubernetes python2 shell
Traceback (most recent call last):
  File "/usr/local/bin/webservice", line 163, in <module>
    job.shell()
  File "/usr/lib/python2.7/dist-packages/toollabs/webservice/backends/kubernetesbackend.py", line 390, in shell
    'interactive'
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/apport_python_hook.py", line 145, in apport_excepthook
    os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o640), 'wb') as f:
OSError: [Errno 2] No such file or directory: '/var/crash/_usr_bin_webservice.53276.crash'

Original exception was:
Traceback (most recent call last):
  File "/usr/local/bin/webservice", line 163, in <module>
    job.shell()
  File "/usr/lib/python2.7/dist-packages/toollabs/webservice/backends/kubernetesbackend.py", line 390, in shell
    'interactive'
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Happens for older tools as well, eg. video2commons-test. CC @yuvipanda

Idk why the line numbers are different, but it's probably a subprocess.Popen call that does not seem to have anything special.

Testing Popen on /bin/ls works fine, but the original fails, so I suspected /usr/local/bin/kubectl is wrong. Indeed:

(venv)tools.video2commons-test@tools-bastion-02:~$ which kubectl
/usr/bin/kubectl
(venv)tools.video2commons-test@tools-bastion-02:~$ ls /usr/local/bin/kubectl
ls: cannot access /usr/local/bin/kubectl: No such file or directory

kubectl is /usr/bin/kubectl is across all 3 bastions, 02, 03 & 05.

Change 334978 had a related patch set uploaded (by Zhuyifei1999):
kubernetesbackend: change absolute kubectl path to '/usr/bin/kubectl'

https://gerrit.wikimedia.org/r/334978

gerritbot added a project: Patch-For-Review.Jan 30 2017, 9:03 AM

scfc moved this task from Ready to be worked on to Waiting for code review on the Toolforge board.Jan 30 2017, 6:13 PM

Change 334978 merged by jenkins-bot:
kubernetesbackend: change absolute kubectl path to '/usr/bin/kubectl'

https://gerrit.wikimedia.org/r/334978

tom29739 subscribed.Jan 30 2017, 10:22 PM

scfc removed a project: Patch-For-Review.Jan 31 2017, 8:14 PM

scfc moved this task from Waiting for code review to In Progress on the Toolforge board.

Mentioned in SAL (#wikimedia-labs) [2017-02-07T13:49:32Z] <scfc_de> Deployed toollabs-webservice_0.33_all.deb (T156605, T156626).

Stashbot mentioned this in T156626: k8s webservice restart failure with `ValueError: get() more than one object; use filter`.Feb 7 2017, 1:49 PM

scfc closed this task as Resolved.Feb 7 2017, 1:51 PM

• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:39 PM

"webservice shell" fails with "No such file or directory" (with php5.6)Closed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

"webservice shell" fails with "No such file or directory" (with php5.6)
Closed, ResolvedPublic
Actions