Page MenuHomePhabricator

Jenkins job can't clone repos: git/objects/* is corrupt
Closed, DuplicatePublic

Description

Pretty consistently (but not 100%) https://gerrit.wikimedia.org/r/#/c/233660/ is failing with:

13:36:50 DEBUG:zuul.Repo:Updating repository src/extensions/VisualEditor
13:36:50 Traceback (most recent call last):
13:36:50   File "/usr/bin/zuul-cloner", line 10, in <module>
13:36:50     sys.exit(main())
13:36:50   File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/cmd/cloner.py", line 160, in main
13:36:50     cloner.main()
13:36:50   File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/cmd/cloner.py", line 155, in main
13:36:50     cloner.execute()
13:36:50   File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/lib/cloner.py", line 68, in execute
13:36:50     self.prepareRepo(project, dest)
13:36:50   File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/lib/cloner.py", line 133, in prepareRepo
13:36:50     repo.update()
13:36:50   File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 180, in update
13:36:50     origin.update()
13:36:50   File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/remote.py", line 536, in update
13:36:50     self.repo.git.remote(scmd, self.name, **kwargs)
13:36:50   File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/cmd.py", line 440, in <lambda>
13:36:50     return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
13:36:50   File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/cmd.py", line 834, in _call_process
13:36:50     return self.execute(make_call(), **_kwargs)
13:36:50   File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/cmd.py", line 627, in execute
13:36:50     raise GitCommandError(command, status, stderr_value)
13:36:50 git.exc.GitCommandError: 'git remote update origin' returned with exit code 1
13:36:50 stderr: 'error: object file .git/objects/a3/3b082c13c0f154fc498e51977f380423c4572f is empty
13:36:50 error: object file .git/objects/a3/3b082c13c0f154fc498e51977f380423c4572f is empty
13:36:50 fatal: loose object a33b082c13c0f154fc498e51977f380423c4572f (stored in .git/objects/a3/3b082c13c0f154fc498e51977f380423c4572f) is corrupt
13:36:50 fatal: The remote end hung up unexpectedly
13:36:50 error: Could not fetch origin'
13:36:50 Build step 'Execute shell' marked build as failure

E.g. https://integration.wikimedia.org/ci/job/mwext-qunit/4397/console and https://integration.wikimedia.org/ci/job/mwext-qunit/4399/console .

Maybe the source repository it's cloning from is corrupt and needs to be re-cloned from the canonical one?

Event Timeline

Krinkle renamed this task from Jenkins job can't clone VisualEditor, blocking merge to [Regression] Jenkins job can't clone repos: git/objects/* is corrupt.Aug 25 2015, 2:42 PM
Krinkle triaged this task as Unbreak Now! priority.
Krinkle set Security to None.

The same is happening for other extensions and mediawiki-core:

See https://integration.wikimedia.org/ci/job/mwext-testextension-zend/7104/console

14:23:02 INFO:zuul.Cloner:Creating repo mediawiki/core from upstream https://gerrit.wikimedia.org/r/p/mediawiki/core
14:23:02 DEBUG:zuul.Repo:Updating repository src
14:23:04 Traceback (most recent call last):
...
 fatal: loose object 25cda1d8a3c1c9e59ca901e12d0df8b7171e396e (stored in git/objects/25/cda1d8a3c1c9e59ca901e12d0df8b7171e396e) is corrupt
hashar renamed this task from [Regression] Jenkins job can't clone repos: git/objects/* is corrupt to Jenkins job can't clone repos: git/objects/* is corrupt.Aug 25 2015, 6:59 PM

Some instances have been corrupted ( T110052 ) :-(