Page MenuHomePhabricator
Create Task
Maniphest T309764

Gate pipeline build failure on mediawiki/core: Install of phpdocumentor/type-resolver or wmde/hamcrest-html-matchers fail
Closed, ResolvedPublicBUG REPORT
Actions

Description

Observed from recent changes that can't be merged:
https://gerrit.wikimedia.org/r/c/mediawiki/core/+/675291
https://gerrit.wikimedia.org/r/c/mediawiki/core/+/745246

https://integration.wikimedia.org/ci/job/mediawiki-quibble-selenium-vendor-mysql-php72-docker/40006/console
https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php72-docker/153883/console

example of logs:

14:30:09   - Installing wmde/hamcrest-html-matchers (v1.0.0): Extracting archive
14:30:09     Failed to extract phpdocumentor/type-resolver: (9) '/usr/bin/unzip' -qq '/workspace/src/vendor/./composer/tmp-8a489bbca9bf86e345fd9519adaa08cf' -d '/workspace/src/vendor/./composer/db96f08e'
14:30:09 
14:30:09 [/workspace/src/vendor/./composer/tmp-8a489bbca9bf86e345fd9519adaa08cf]
14:30:09   End-of-central-directory signature not found.  Either this file is not
14:30:09   a zipfile, or it constitutes one disk of a multi-part archive.  In the
14:30:09   latter case the central directory and zipfile comment will be found on
14:30:09   the last disk(s) of this archive.
14:30:09 unzip:  cannot find zipfile directory in one of /workspace/src/vendor/./composer/tmp-8a489bbca9bf86e345fd9519adaa08cf or
14:30:09         /workspace/src/vendor/./composer/tmp-8a489bbca9bf86e345fd9519adaa08cf.zip, and cannot find /workspace/src/vendor/./composer/tmp-8a489bbca9bf86e345fd9519adaa08cf.ZIP, period.
14:30:09 
14:30:09     The archive may contain identical file names with different capitalization (which fails on case insensitive filesystems)
14:30:09     Unzip with unzip command failed, falling back to ZipArchive class
14:30:09     Install of phpdocumentor/type-resolver failed
14:30:10 
14:30:10 In ZipDownloader.php line 209:
14:30:10                                                                                
14:30:10   [UnexpectedValueException (-1)]                                              
14:30:10   '/workspace/src/vendor/./composer/tmp-8a489bbca9bf86e345fd9519adaa08cf' is   
14:30:10   a corrupted zip archive (0 bytes), try again.                                
14:30:10                                                                                
14:30:10 
14:30:10 Exception trace:
14:30:10   at phar:///usr/bin/composer/src/Composer/Downloader/ZipDownloader.php:209
14:30:10  Composer\Downloader\ZipDownloader->extractWithZipArchive() at phar:///usr/bin/composer/src/Composer/Downloader/ZipDownloader.php:157
14:30:10  Composer\Downloader\ZipDownloader->Composer\Downloader\{closure}() at phar:///usr/bin/composer/src/Composer/Downloader/ZipDownloader.php:172
14:30:10  Composer\Downloader\ZipDownloader->Composer\Downloader\{closure}() at phar:///usr/bin/composer/vendor/react/promise/src/FulfilledPromise.php:28
14:30:10  React\Promise\FulfilledPromise->then() at phar:///usr/bin/composer/vendor/react/promise/src/Promise.php:134
14:30:10  React\Promise\Promise::React\Promise\{closure}() at phar:///usr/bin/composer/vendor/react/promise/src/Promise.php:168
14:30:10  React\Promise\Promise->settle() at phar:///usr/bin/composer/vendor/react/promise/src/Promise.php:231
14:30:10  React\Promise\Promise::React\Promise\{closure}() at phar:///usr/bin/composer/vendor/react/promise/src/FulfilledPromise.php:42
14:30:10  React\Promise\FulfilledPromise->done() at phar:///usr/bin/composer/vendor/react/promise/src/Promise.php:135
14:30:10  React\Promise\Promise::React\Promise\{closure}() at phar:///usr/bin/composer/vendor/react/promise/src/Promise.php:168
14:30:10  React\Promise\Promise->settle() at phar:///usr/bin/composer/vendor/react/promise/src/Promise.php:231
14:30:10  React\Promise\Promise::React\Promise\{closure}() at n/a:n/a
14:30:10  call_user_func() at phar:///usr/bin/composer/src/Composer/Util/ProcessExecutor.php:321
14:30:10  Composer\Util\ProcessExecutor->countActiveJobs() at phar:///usr/bin/composer/src/Composer/Util/Loop.php:98
14:30:10  Composer\Util\Loop->wait() at phar:///usr/bin/composer/src/Composer/Installer/InstallationManager.php:496
14:30:10  Composer\Installer\InstallationManager->waitOnPromises() at phar:///usr/bin/composer/src/Composer/Installer/InstallationManager.php:469
14:30:10  Composer\Installer\InstallationManager->executeBatch() at phar:///usr/bin/composer/src/Composer/Installer/InstallationManager.php:390
14:30:10  Composer\Installer\InstallationManager->downloadAndExecuteBatch() at phar:///usr/bin/composer/src/Composer/Installer/InstallationManager.php:282
14:30:10  Composer\Installer\InstallationManager->execute() at phar:///usr/bin/composer/src/Composer/Installer.php:759
14:30:10  Composer\Installer->doInstall() at phar:///usr/bin/composer/src/Composer/Installer.php:588
14:30:10  Composer\Installer->doUpdate() at phar:///usr/bin/composer/src/Composer/Installer.php:279
14:30:10  Composer\Installer->run() at phar:///usr/bin/composer/src/Composer/Command/RequireCommand.php:425
14:30:10  Composer\Command\RequireCommand->doUpdate() at phar:///usr/bin/composer/src/Composer/Command/RequireCommand.php:288
14:30:10  Composer\Command\RequireCommand->execute() at phar:///usr/bin/composer/vendor/symfony/console/Command/Command.php:298
14:30:10  Symfony\Component\Console\Command\Command->run() at phar:///usr/bin/composer/vendor/symfony/console/Application.php:1015
14:30:10  Symfony\Component\Console\Application->doRunCommand() at phar:///usr/bin/composer/vendor/symfony/console/Application.php:299
14:30:10  Symfony\Component\Console\Application->doRun() at phar:///usr/bin/composer/src/Composer/Console/Application.php:334
14:30:10  Composer\Console\Application->doRun() at phar:///usr/bin/composer/vendor/symfony/console/Application.php:171
14:30:10  Symfony\Component\Console\Application->run() at phar:///usr/bin/composer/src/Composer/Console/Application.php:130
14:30:10  Composer\Console\Application->run() at phar:///usr/bin/composer/bin/composer:83
14:30:10  require() at /usr/bin/composer:29

Event Timeline

Func created this task.Jun 2 2022, 7:09 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 2 2022, 7:09 AM
Winston_Sung subscribed.Jun 2 2022, 7:11 AM
Nikerabbit subscribed.Jun 2 2022, 7:26 AM
Urbanecm_WMF moved this task from Backlog to WMF-deployed Build Failure on the ci-test-error board.Jun 2 2022, 7:27 AM
Urbanecm_WMF edited projects, added ci-test-error (WMF-deployed Build Failure); removed ci-test-error.
Urbanecm_WMF subscribed.

Looks to affect mediawiki/*. Moving to WMF-deployed build failure.

kostajh triaged this task as Unbreak Now! priority.Jun 2 2022, 8:45 AM
kostajh added a project: Release-Engineering-Team.
kostajh subscribed.
In T309764#7975738, @Urbanecm_WMF wrote:

Looks to affect mediawiki/*. Moving to WMF-deployed build failure.

Marking as an unbreak now, then.

kostajh added a subscriber: hashar.Jun 2 2022, 8:55 AM
hashar added a comment.Jun 2 2022, 9:02 AM

The Zip file for the composer dependency wmde/hamcrest-html-matchers is corrupt:

is  a corrupted zip archive (0 bytes)

It comes from https://api.github.com/repos/wmde/hamcrest-html-matchers/zipball/f0f727f80bdb97831628329cfe508fce2a44028a

But on CI we have composer cache pointing to a directory set via the environment variable XDG_CACHE_HOME and that content is saved between builds. There must have been some race condition between the builds that caused an empty file to be stored in the cache and I am guessing composer is not smart enough to invalidate the cached file and redownload which lead to the failure.

Gotta clear the cache.

hashar added a comment.Jun 2 2022, 9:09 AM

The cache system is more or less described https://www.mediawiki.org/wiki/Continuous_integration/Architecture/Castor . The files are stored on integration-castor05.integration.eqiad1.wikimedia.cloud under /srv/castor/$CASTOR_NAMESPACE. The env variable is defined by CI and is shown at the top of the build:

$ docker run ... docker-registry.wikimedia.org/releng/castor:0.2.6 load
Defined: CASTOR_NAMESPACE="mediawiki-core/master/mediawiki-quibble-selenium-vendor-mysql-php72-docker"
Syncing...
Done

Then there is a file that has some content:

-rw-r--r-- 1 jenkins-deploy wikidev 23210 Feb  9 04:10 composer/files/wmde/hamcrest-html-matchers/0b022626079d35be564747215d951a7ced6333f6.zip
md5sum
8197d733b8bdf5b5e8e5a278473b8790
/srv/castor/mediawiki-core/master/mediawiki-quibble-selenium-vendor-mysql-php72-docker/composer/files/wmde/hamcrest-html-matchers/0b022626079d35be564747215d951a7ced6333f6.zip

Same for phpdocumentor/type-resolver:

07:00:26 Defined: CASTOR_NAMESPACE="mediawiki-core/master/wmf-quibble-selenium-php72-docker"
-rw-r--r-- 1 jenkins-deploy wikidev 46393 Feb  4 17:32 ./composer/files/phpdocumentor/type-resolver/835223c687c7406ac712579841d5f8a47d149778.zip
md5sum
3d717bc331ee0bd757601dab052feb95
/srv/castor/mediawiki-core/master/wmf-quibble-selenium-php72-docker/composer/files/phpdocumentor/type-resolver/835223c687c7406ac712579841d5f8a47d149778.zip

So looks like the cache is valid

Aklapper renamed this task from Gate pipeline build filure on mediawiki/core: Install of phpdocumentor/type-resolver failed to Gate pipeline build failure on mediawiki/core: Install of phpdocumentor/type-resolver failed.Jun 2 2022, 9:37 AM
hashar added a comment.Jun 2 2022, 9:44 AM

Kosta pointed out the dependencies are downloaded a build like https://integration.wikimedia.org/ci/job/mediawiki-quibble-vendor-mysql-php72-docker/69272/console has:

- Downloading phpdocumentor/type-resolver (1.6.1)"

Additionally the mwcore-phpunit-coverage-patch job does not have any cache (since it is never triggered in gate-and-submit nor in postmerge pipelines), thus composer always download from packagist/github or whatever. Example build https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-patch/44197/console which has:

10:37:48   - Locking phpdocumentor/type-resolver (1.6.1)
10:37:48   - Downloading phpdocumentor/type-resolver (1.6.1)
10:37:50   - Installing phpdocumentor/type-resolver (1.6.1): Extracting archive

10:37:50     Failed to extract phpdocumentor/type-resolver: (9) '/usr/bin/unzip' -qq '/workspace/src/vendor/./composer/tmp-eed06b4d987270f6d49dd5a222735b06' -d '/workspace/src/vendor/./composer/18537034'
10:37:50 
10:37:50 [/workspace/src/vendor/./composer/tmp-eed06b4d987270f6d49dd5a222735b06]
10:37:50   End-of-central-directory signature not found.  Either this file is not
10:37:50   a zipfile, or it constitutes one disk of a multi-part archive.  In the
10:37:50   latter case the central directory and zipfile comment will be found on
10:37:50   the last disk(s) of this archive.
10:37:50 unzip:  cannot find zipfile directory in one of /workspace/src/vendor/./composer/tmp-eed06b4d987270f6d49dd5a222735b06 or
10:37:50         /workspace/src/vendor/./composer/tmp-eed06b4d987270f6d49dd5a222735b06.zip, and cannot find /workspace/src/vendor/./composer/tmp-eed06b4d987270f6d49dd5a222735b06.ZIP, period.
10:37:50 
10:37:50     The archive may contain identical file names with different capitalization (which fails on case insensitive filesystems)
10:37:50     Unzip with unzip command failed, falling back to ZipArchive class
10:37:50     Install of phpdocumentor/type-resolver failed
10:37:51 
10:37:51 In ZipDownloader.php line 209:
10:37:51                                                                                
10:37:51   [UnexpectedValueException (-1)]                                              
10:37:51   '/workspace/src/vendor/./composer/tmp-eed06b4d987270f6d49dd5a222735b06' is   
10:37:51   a corrupted zip archive (0 bytes), try again.

The only dependencies I have seen failing are wmde/hamcrest-html-matchers:v1.0.0 and phpdocumentor/type-resolver:1.6.1 which are downloaded from GitHub at:

https://codeload.github.com/phpDocumentor/TypeResolver/legacy.zip/77a32518733312af16a44300404e945338981de3
https://codeload.github.com/wmde/hamcrest-html-matchers/legacy.zip/f0f727f80bdb97831628329cfe508fce2a44028a

GitHub doesn't mention anything on their status page https://www.githubstatus.com/history

hashar renamed this task from Gate pipeline build failure on mediawiki/core: Install of phpdocumentor/type-resolver failed to Gate pipeline build failure on mediawiki/core: Install of phpdocumentor/type-resolver or wmde/hamcrest-html-matchers fail.Jun 2 2022, 9:45 AM
Restricted Application added a subscriber: Masumrezarock100. · View Herald TranscriptJun 2 2022, 9:45 AM
hashar added a comment.Jun 2 2022, 9:59 AM

If I look at the build trend for the job https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php72-docker/buildTimeTrend

wmf-quibble-selenium-php72-docker.png (400×500 px, 46 KB)

Looks like that has been transient and finally resolved. I highly suspect https://codeload.github.com/ had an issue of some sort which caused some zip archives to be served with 0 bytes. Maybe one of their backend had a failure of some sort.

hashar closed this task as Resolved.Jun 2 2022, 10:41 AM
hashar claimed this task.
hashar added a project: Upstream.

I believe it was a transient issue with GitHub infrastructure which served some Zip files with 0 bytes. I don't see builds failing anymore.

I have send a quick message on wikitech-l https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/EUUBZJOCDNJLFR5H7ER5GHRFPMEBSHT7/

TheresNoTime added subscribers: Samwilson, TheresNoTime.Jun 2 2022, 10:42 AM
MoritzMuehlenhoff subscribed.Jun 2 2022, 11:06 AM
In T309764#7976064, @hashar wrote:

I believe it was a transient issue with GitHub infrastructure which served some Zip files with 0 bytes. I don't see builds failing anymore.

I have send a quick message on wikitech-l https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/EUUBZJOCDNJLFR5H7ER5GHRFPMEBSHT7/

FWIW, Github had an outage yesterday: https://www.githubstatus.com/incidents/8xxms3q3qzgp Maybe that caused it directyl (or via fallout of the outage)

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits