@Andrew do you have any spare time to look at them please ? :-}

This is probably because they are hosted on labvirt1007 which no longer has space to expand instance drives. I'm working on a fix for this and may be able to resolve it later today... if you need things working in the next few hours it's best to destroy and rebuild them.

At worse I will recreate tomorrow morning Europe time :-) Thanks Andrew!

Self note: manual steps are listed at https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration/Setup?redirect=no#Patches

integration-slave-trusty-1014 on boot reports:

/bin/sh: 1: exec: cloud-init: not found

integration-slave-trusty-1017 when running an unknown command I got:

/usr/bin/python: can't find '__main__' module in '/usr/share/command-not-found'

integration-slave-precise-1014 probably suffers similar issue.

I have unopposed all three instances and deleting them. We need to recreate them from scratch.

I have poked the Release-Engineering-Team team mailing list to pair the rebuild with someone.

Zeljko rebuild integration-slave-precise-1014 :-}

In T110052#1575146, @hashar wrote:

Zeljko rebuild integration-slave-precise-1014 :-}

I just marked https://integration.wikimedia.org/ci/computer/integration-slave-precise-1014/ offline. It seems to be having problems cloning git://zuul.eqiad.wmnet/mediawiki/core that are very repeatable (even with stock git from the cli). See https://integration.wikimedia.org/ci/job/mediawiki-core-phplint/9469/console and other failed mediawiki-core-phplint jobs.

In T110052#1575146, @hashar wrote:

Zeljko rebuild integration-slave-precise-1014 :-}

It's not working properly, @bd808 marked it as offline in jenkins for now. See https://integration.wikimedia.org/ci/job/mediawiki-core-phplint/9468/console and https://integration.wikimedia.org/ci/job/tox-flake8/8196/console for example.

Seems it takes more than 10 minutes to clone mediawiki/core from zuul.eqiad.wmnet :-/ Maybe the git-daemon serving them on gallium.wikimedia.org is overloaded or gallium itself as disk I/O troubles.

Would need to reproduce with GIT_TRACE=1 GIT_TRACE_PACKETS=1.

Gave it a try on integration-slave-precise1014:

$ git init .
$ GIT_TRACE=1 time git -c core.askpass=true fetch --tags --progress git://zuul.eqiad.wmnet/mediawiki/core +refs/heads/*:refs/remotes/origin/*
trace: built-in: git 'fetch' '--tags' '--progress' 'git://zuul.eqiad.wmnet/mediawiki/core' '+refs/heads/*:refs/remotes/origin/*'
trace: run_command: 'rev-list' '--verify-objects' '--stdin' '--not' '--all' '--quiet'
remote: Counting objects: 578493, done.
remote: Compressing objects: 100% (105780/105780), done.
trace: run_command: 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 23014 on integration-slave-precise-1014' '--pack_header=2,578493'
trace: exec: 'git' 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 23014 on integration-slave-precise-1014' '--pack_header=2,578493'
trace: built-in: git 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 23014 on integration-slave-precise-1014' '--pack_header=2,578493'
remote: Total 578493 (delta 478370), reused 570493 (delta 470938)
Receiving objects: 100% (578493/578493), 310.03 MiB | 23.57 MiB/s, done.
Resolving deltas: 100% (478370/478370), done.
trace: run_command: 'rev-list' '--verify-objects' '--stdin' '--not' '--all'
trace: exec: 'git' 'rev-list' '--verify-objects' '--stdin' '--not' '--all'
trace: built-in: git 'rev-list' '--verify-objects' '--stdin' '--not' '--all'

At this point it takes roughly 10 minutes to write informations in the pack file :-/

667.51user 9.14system 11:29.68elapsed 98%CPU (0avgtext+0avgdata 1603184maxresident)k
0inputs+672856outputs (0major+2249250minor)pagefaults 0swaps

So 667 seconds or 11 minutes :-/

Recreating integration-slave-trusty-1014 and integration-slave-trusty-1017

(npm / grunt-cli are up-to-date)