Page MenuHomePhabricator

mediawiki-core-phplint clone the whole repo from Zuul and times out
Closed, ResolvedPublic

Description

From T110052 on integration-slave-precise1014, the job https://integration.wikimedia.org/ci/job/mediawiki-core-phplint/ times out after 10 minutes.

$ git init .
$ GIT_TRACE=1 time git -c core.askpass=true fetch --tags --progress git://zuul.eqiad.wmnet/mediawiki/core +refs/heads/*:refs/remotes/origin/*
trace: built-in: git 'fetch' '--tags' '--progress' 'git://zuul.eqiad.wmnet/mediawiki/core' '+refs/heads/*:refs/remotes/origin/*'
trace: run_command: 'rev-list' '--verify-objects' '--stdin' '--not' '--all' '--quiet'
remote: Counting objects: 578493, done.
remote: Compressing objects: 100% (105780/105780), done.
trace: run_command: 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 23014 on integration-slave-precise-1014' '--pack_header=2,578493'
trace: exec: 'git' 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 23014 on integration-slave-precise-1014' '--pack_header=2,578493'
trace: built-in: git 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 23014 on integration-slave-precise-1014' '--pack_header=2,578493'
remote: Total 578493 (delta 478370), reused 570493 (delta 470938)
Receiving objects: 100% (578493/578493), 310.03 MiB | 23.57 MiB/s, done.
Resolving deltas: 100% (478370/478370), done.
trace: run_command: 'rev-list' '--verify-objects' '--stdin' '--not' '--all'
trace: exec: 'git' 'rev-list' '--verify-objects' '--stdin' '--not' '--all'
trace: built-in: git 'rev-list' '--verify-objects' '--stdin' '--not' '--all'
At this point it takes roughly 10 minutes to write informations in the pack file :-/

667.51user 9.14system 11:29.68elapsed 98%CPU (0avgtext+0avgdata 1603184maxresident)k
0inputs+672856outputs (0major+2249250minor)pagefaults 0swaps
So 667 seconds or 11 minutes :-/

Maybe a shallow clone would speed it up or even git-archive since zuul.eqiad.wmnet should support it (unlike Gerrit).

Event Timeline

hashar created this task.Aug 27 2015, 12:08 PM
hashar claimed this task.
hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)

We could try shallow clone with no submodules again, but we should just fix T97098: Update jobs to use zuul-cloner with git cache/T96687: Set up git replication on integration slaves IMO, since that is a long term solution that will help with *all* jobs, not just one.

hashar closed this task as Resolved.Sep 14 2015, 2:08 PM

I have manually cloned the mediawiki/core repo on that slave for that job. That has let me pool it back in.

The long term fixes are tasks mentioned by @Legoktm above.