Page MenuHomePhabricator

Error "TransportException 404 Not Found" in Jenkins jobs using composer
Closed, ResolvedPublic

Description /

23:34:20 + composer install --ansi --no-progress --prefer-dist --profile -v
23:34:20 [2.0MB/0.04s] Loading composer repositories with package information
23:34:20 [2.0MB/0.49s] Updating dependencies (including require-dev)
23:34:25   [Composer\Downloader\TransportException]                                                                                                                                     
23:34:25   The "" file could not be downloaded (HTTP/1.1 404 Not Found)  
23:34:25 Exception trace:
23:34:25  () at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/Util/RemoteFilesystem.php:350
23:34:25  Composer\Util\RemoteFilesystem->get() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/Util/RemoteFilesystem.php:101
23:34:25  Composer\Util\RemoteFilesystem->getContents() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/Repository/ComposerRepository.php:645
23:34:25  Composer\Repository\ComposerRepository->fetchFile() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/Repository/ComposerRepository.php:558
23:34:25  Composer\Repository\ComposerRepository->loadProviderListings() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/Repository/ComposerRepository.php:289
23:34:25  Composer\Repository\ComposerRepository->whatProvides() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/DependencyResolver/Pool.php:204
23:34:25  Composer\DependencyResolver\Pool->computeWhatProvides() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/DependencyResolver/Pool.php:193
23:34:25  Composer\DependencyResolver\Pool->whatProvides() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/DependencyResolver/RuleSetGenerator.php:271
23:34:25  Composer\DependencyResolver\RuleSetGenerator->whitelistFromJobs() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/DependencyResolver/RuleSetGenerator.php:324
23:34:25  Composer\DependencyResolver\RuleSetGenerator->getRulesFor() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/DependencyResolver/Solver.php:214
23:34:25  Composer\DependencyResolver\Solver->solve() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/Installer.php:455
23:34:25  Composer\Installer->doInstall() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/Installer.php:215
23:34:25  Composer\Installer->run() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/Command/InstallCommand.php:133
23:34:25  Composer\Command\InstallCommand->execute() at /srv/deployment/integration/composer/vendor/symfony/console/Command/Command.php:264
23:34:25  Symfony\Component\Console\Command\Command->run() at /srv/deployment/integration/composer/vendor/symfony/console/Application.php:846
23:34:25  Symfony\Component\Console\Application->doRunCommand() at /srv/deployment/integration/composer/vendor/symfony/console/Application.php:191
23:34:25  Symfony\Component\Console\Application->doRun() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/Console/Application.php:227
23:34:25  Composer\Console\Application->doRun() at /srv/deployment/integration/composer/vendor/symfony/console/Application.php:122
23:34:25  Symfony\Component\Console\Application->run() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/Console/Application.php:102
23:34:25  Composer\Console\Application->run() at /srv/deployment/integration/composer/vendor/composer/composer/bin/composer:44

I can run composer install locally on a clean checkout with no issues.

Event Timeline

Tgr created this task.Dec 6 2017, 11:42 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 6 2017, 11:42 PM
Tgr updated the task description. (Show Details)Dec 6 2017, 11:43 PM
Tgr added a comment.Dec 6 2017, 11:50 PM

composer#2198 suggests this could be caused by an outdated Composer executable.

Tgr triaged this task as Unbreak Now! priority.Dec 7 2017, 12:07 AM

Seems to be breaking all MediaWiki extension tests.

Restricted Application added subscribers: Liuxinyu970226, Jay8g, TerraCodes. · View Herald TranscriptDec 7 2017, 12:07 AM
Tgr added a comment.EditedDec 7 2017, 12:29 AM is at 1.1.3. Can't reproduce the error with that version locally; installation seems to work fine:

1tgr@tgr-ThinkPad:~/Wikimedia/grep/extensions/ReadingLists$ wget
2--2017-12-06 16:27:37--
3Resolving (, 2001:41d0:a:7b19::2
4Connecting to (||:443... connected.
5HTTP request sent, awaiting response... 200 OK
6Length: 1641245 (1.6M) [application/octet-stream]
7Saving to: ‘composer.phar’
9composer.phar 100%[===================>] 1.56M 1.14MB/s in 1.4s
112017-12-06 16:27:40 (1.14 MB/s) - ‘composer.phar’ saved [1641245/1641245]
13tgr@tgr-ThinkPad:~/Wikimedia/grep/extensions/ReadingLists$ ./composer.phar clear-cache
14Clearing cache (cache-dir): /home/tgr/.composer/cache
15Clearing cache (cache-files-dir): /home/tgr/.composer/cache/files
16Clearing cache (cache-repo-dir): /home/tgr/.composer/cache/repo
17Clearing cache (cache-vcs-dir): /home/tgr/.composer/cache/vcs
18All caches cleared.
19tgr@tgr-ThinkPad:~/Wikimedia/grep/extensions/ReadingLists$ ./composer.phar install -vvv --ignore-platform-reqs
20Reading ./composer.json
21Loading config file ./composer.json
22Checked CA file /etc/ssl/certs/ca-certificates.crt: valid
23Executing command (/home/tgr/Wikimedia/grep/extensions/ReadingLists): git branch --no-color --no-abbrev -v
24Failed to initialize global composer: Composer could not find the config file: /home/tgr/.composer/composer.json
25To initialize a project, please create a composer.json file as described in the "Getting Started" section
26Running 1.1.3 (2016-06-26 15:42:08) with PHP 7.0.22-0ubuntu0.16.04.1 on Linux / 4.4.0-101-generic
27Loading composer repositories with package information
29Writing /home/tgr/.composer/cache/repo/ into cache
30Updating dependencies (including require-dev)
32Writing /home/tgr/.composer/cache/repo/ into cache
34Writing /home/tgr/.composer/cache/repo/ into cache
36Writing /home/tgr/.composer/cache/repo/ into cache
38Writing /home/tgr/.composer/cache/repo/ into cache
40Writing /home/tgr/.composer/cache/repo/ into cache
42Writing /home/tgr/.composer/cache/repo/ into cache
44Writing /home/tgr/.composer/cache/repo/ into cache
46Writing /home/tgr/.composer/cache/repo/ into cache
48Writing /home/tgr/.composer/cache/repo/ into cache
50Writing /home/tgr/.composer/cache/repo/ into cache
52Writing /home/tgr/.composer/cache/repo/$php-parallel-lint.json into cache
54Writing /home/tgr/.composer/cache/repo/$mediawiki-codesniffer.json into cache
56Writing /home/tgr/.composer/cache/repo/$php-codesniffer.json into cache
58Writing /home/tgr/.composer/cache/repo/$php-console-highlighter.json into cache
60Writing /home/tgr/.composer/cache/repo/$php-console-color.json into cache
62Writing /home/tgr/.composer/cache/repo/$minus-x.json into cache
64Writing /home/tgr/.composer/cache/repo/$console.json into cache
66Writing /home/tgr/.composer/cache/repo/$polyfill-mbstring.json into cache
68Writing /home/tgr/.composer/cache/repo/$debug.json into cache
70Writing /home/tgr/.composer/cache/repo/$log.json into cache
72Writing /home/tgr/.composer/cache/repo/$dependency-injection.json into cache
74Writing /home/tgr/.composer/cache/repo/$http-kernel.json into cache
75Resolving dependencies through SAT
76Dependency resolution completed in 0.007 seconds
77Analyzed 1375 packages to resolve dependencies
78Analyzed 3841 rules to resolve dependencies
79Resolving dependencies through SAT
80Dependency resolution completed in 0.000 seconds
81 - Installing jakub-onderka/php-parallel-lint (v0.9.2)
83 Downloading: 100%
84Writing /home/tgr/.composer/cache/files/jakub-onderka/php-parallel-lint/ into cache from /home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/jakub-onderka/php-parallel-lint/2acd828cf8f1474ea521abc952596fff
85 Extracting archive
86Executing command (CWD): unzip '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/jakub-onderka/php-parallel-lint/2acd828cf8f1474ea521abc952596fff' -d '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/fdd40091' && chmod -R u+w '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/fdd40091'
88 REASON: Required by the root package: Install command rule (install jakub-onderka/php-parallel-lint v0.9.2)
90 - Installing squizlabs/php_codesniffer (3.1.1)
92 Downloading: 100%
93Writing /home/tgr/.composer/cache/files/squizlabs/php_codesniffer/ into cache from /home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/squizlabs/php_codesniffer/c61fa8f2ac7513ef4279da872f17d280
94 Extracting archive
95Executing command (CWD): unzip '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/squizlabs/php_codesniffer/c61fa8f2ac7513ef4279da872f17d280' -d '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/0e504644' && chmod -R u+w '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/0e504644'
97 REASON: mediawiki/mediawiki-codesniffer v14.1.0 requires squizlabs/php_codesniffer 3.1.1 -> satisfiable by squizlabs/php_codesniffer[3.1.1].
99 - Installing mediawiki/mediawiki-codesniffer (v14.1.0)
101 Downloading: 100%
102Writing /home/tgr/.composer/cache/files/mediawiki/mediawiki-codesniffer/ into cache from /home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/mediawiki/mediawiki-codesniffer/79a3bc434c01c25cc753bd6784086e65
103 Extracting archive
104Executing command (CWD): unzip '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/mediawiki/mediawiki-codesniffer/79a3bc434c01c25cc753bd6784086e65' -d '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/e5bde2f1' && chmod -R u+w '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/e5bde2f1'
106 REASON: Required by the root package: Install command rule (install mediawiki/mediawiki-codesniffer v14.1.0)
108 - Installing jakub-onderka/php-console-color (0.1)
110 Downloading: 100%
111Writing /home/tgr/.composer/cache/files/jakub-onderka/php-console-color/ into cache from /home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/jakub-onderka/php-console-color/dea5abc5c7088f768a188f290f325403
112 Extracting archive
113Executing command (CWD): unzip '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/jakub-onderka/php-console-color/dea5abc5c7088f768a188f290f325403' -d '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/b06b24ac' && chmod -R u+w '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/b06b24ac'
115 REASON: jakub-onderka/php-console-highlighter v0.3.2 requires jakub-onderka/php-console-color ~0.1 -> satisfiable by jakub-onderka/php-console-color[0.1].
117 - Installing jakub-onderka/php-console-highlighter (v0.3.2)
119 Downloading: 100%
120Writing /home/tgr/.composer/cache/files/jakub-onderka/php-console-highlighter/ into cache from /home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/jakub-onderka/php-console-highlighter/7a9beb063535e330c839b84ccfb8ba52
121 Extracting archive
122Executing command (CWD): unzip '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/jakub-onderka/php-console-highlighter/7a9beb063535e330c839b84ccfb8ba52' -d '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/ddb0d0d4' && chmod -R u+w '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/ddb0d0d4'
124 REASON: Required by the root package: Install command rule (install jakub-onderka/php-console-highlighter v0.3.2)
126 - Installing psr/log (1.0.2)
128 Downloading: 100%
129Writing /home/tgr/.composer/cache/files/psr/log/ into cache from /home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/psr/log/34c4d0bac8590e7036931785cb9014a7
130 Extracting archive
131Executing command (CWD): unzip '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/psr/log/34c4d0bac8590e7036931785cb9014a7' -d '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/2c5d0d39' && chmod -R u+w '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/2c5d0d39'
133 REASON: symfony/debug v3.4.1 requires psr/log ~1.0 -> satisfiable by psr/log[1.0.0, 1.0.1, 1.0.2].
135 - Installing symfony/debug (v3.4.1)
137 Downloading: 100%
138Writing /home/tgr/.composer/cache/files/symfony/debug/ into cache from /home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/symfony/debug/1c20455faf60b221363372ec3fe12f00
139 Extracting archive
140Executing command (CWD): unzip '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/symfony/debug/1c20455faf60b221363372ec3fe12f00' -d '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/e64883a1' && chmod -R u+w '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/e64883a1'
142 REASON: symfony/console v3.3.5 requires symfony/debug ~2.8|~3.0 -> satisfiable by symfony/debug[v2.8.0, v2.8.1, v2.8.10, v2.8.11, v2.8.12, v2.8.13, v2.8.14, v2.8.15, v2.8.16, v2.8.17, v2.8.18, v2.8.19, v2.8.2, v2.8.20, v2.8.21, v2.8.22, v2.8.23, v2.8.24, v2.8.25, v2.8.26, v2.8.27, v2.8.28, v2.8.29, v2.8.3, v2.8.30, v2.8.31, v2.8.32, v2.8.4, v2.8.5, v2.8.6, v2.8.7, v2.8.8, v2.8.9, v3.0.0, v3.0.1, v3.0.2, v3.0.3, v3.0.4, v3.0.5, v3.0.6, v3.0.7, v3.0.8, v3.0.9, v3.1.0, v3.1.1, v3.1.10, v3.1.2, v3.1.3, v3.1.4, v3.1.5, v3.1.6, v3.1.7, v3.1.8, v3.1.9, v3.2.0, v3.2.1, v3.2.10, v3.2.11, v3.2.12, v3.2.13, v3.2.14, v3.2.2, v3.2.3, v3.2.4, v3.2.5, v3.2.6, v3.2.7, v3.2.8, v3.2.9, v3.3.0, v3.3.1, v3.3.10, v3.3.11, v3.3.12, v3.3.13, v3.3.14, v3.3.2, v3.3.3, v3.3.4, v3.3.5, v3.3.6, v3.3.7, v3.3.8, v3.3.9, v3.4.0, v3.4.1].
144 - Installing symfony/polyfill-mbstring (v1.6.0)
146 Downloading: 100%
147Writing /home/tgr/.composer/cache/files/symfony/polyfill-mbstring/ into cache from /home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/symfony/polyfill-mbstring/5aa812aa8f79f5e03717b7be593d6a4b
148 Extracting archive
149Executing command (CWD): unzip '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/symfony/polyfill-mbstring/5aa812aa8f79f5e03717b7be593d6a4b' -d '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/afe9485c' && chmod -R u+w '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/afe9485c'
151 REASON: symfony/console v3.3.5 requires symfony/polyfill-mbstring ~1.0 -> satisfiable by symfony/polyfill-mbstring[v1.0.0, v1.0.1, v1.1.0, v1.1.1, v1.2.0, v1.3.0, v1.3.1, v1.4.0, v1.5.0, v1.6.0].
153 - Installing symfony/console (v3.3.5)
155 Downloading: 100%
156Writing /home/tgr/.composer/cache/files/symfony/console/ into cache from /home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/symfony/console/f94003d2ee747342a2c0ad34d26b42f5
157 Extracting archive
158Executing command (CWD): unzip '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/symfony/console/f94003d2ee747342a2c0ad34d26b42f5' -d '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/7f32aad7' && chmod -R u+w '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/7f32aad7'
160 REASON: mediawiki/minus-x 0.2.0 requires symfony/console 3.3.5 -> satisfiable by symfony/console[v3.3.5].
162 - Installing mediawiki/minus-x (0.2.0)
164 Downloading: 100%
165Writing /home/tgr/.composer/cache/files/mediawiki/minus-x/ into cache from /home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/mediawiki/minus-x/07e8bcea62cc2caaa9fd576e47cd6438
166 Extracting archive
167Executing command (CWD): unzip '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/mediawiki/minus-x/07e8bcea62cc2caaa9fd576e47cd6438' -d '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/fe28386f' && chmod -R u+w '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/fe28386f'
169 REASON: Required by the root package: Install command rule (install mediawiki/minus-x 0.2.0)
171symfony/console suggests installing symfony/event-dispatcher ()
172symfony/console suggests installing symfony/filesystem ()
173symfony/console suggests installing symfony/process ()
174Reading /home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/installed.json
175Writing lock file
176Generating autoload files

Tgr added a comment.Dec 7 2017, 1:04 AM

Looking at the Composer code it seems like these URLs come from by replacing %hash% with the sha256 parameter (1.1.3, master - no difference). So not sure how outdated Composer code could cause this. An outdated cached version of packages.json maybe? But I'd expect those to get regenerated as CI containers are created and destroyed.

Arlolra added a subscriber: Arlolra.Dec 7 2017, 1:07 AM

I've regenerated the nodepool jessie snapshot using the instructions from:

Although the timeline of this problem doesn't match up with when the last snapshot was built (2017-12-05 14:14).

Composer diag on a nodepool instance shows with mw-config

jenkins@ci-jessie-wikimedia-910534:~/space$ composer diag                                                                                                           
PHP Deprecated:  Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/20-xhprof.ini on line 2 in Unknown on line 0                                          
You are running composer with xdebug enabled. This has a major impact on runtime performance. See                                         
Checking composer.json: FAIL                                                                                                                                             
No license specified, it is recommended to do so. For closed-source software you may use "proprietary" as license.                                                       
require.mongofill/mongofill : unbound version constraints (dev-master) should be avoided                                                                                 
Checking platform settings: FAIL                                                                                                                                         
The xdebug extension is loaded, this can slow down Composer a little.                                                                                                    
 Disabling it when using Composer is recommended.                                                                                                                        
Checking git settings: OK                                                                                                                                                
Checking http connectivity to packagist: Warning: Accessing over http which is an insecure protocol.                                                       
Checking https connectivity to packagist: OK                                                                                                                             
Checking rate limit: OK                                                                                                                                       
Checking disk free space: OK

so for some reason adding HTTPS_PROXY_REQUEST_FULLURI=false seems to fix this (found via

However, this didn't work:

node /srv/deployment/integration/slave-scripts/tools/composer-dev-args.js "$(pwd)/composer.json" | xargs --verbose composer require --dev --ansi --no-progress --ignore-platform-reqs --prefer-dist -v

but this did:

HTTP_PROXY_REQUEST_FULLURI=false HTTPS_PROXY_REQUEST_FULLURI=false composer require --dev --ansi --no-progress --ignore-platform-reqs --prefer-dist -v jakub-onderka/php-parallel-lint=0.9.2 mediawiki/mediawiki-codesniffer=14.1.0 phpunit/phpunit=4.8.36

Change 395913 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[integration/config@master] HTTPS_PROXY_REQUEST_FULLURI=false for composer

Change 395913 merged by jenkins-bot:
[integration/config@master] HTTPS_PROXY_REQUEST_FULLURI=false for composer

thcipriani closed this task as Resolved.Dec 7 2017, 2:53 AM
thcipriani claimed this task.

Alright, I've deployed the fix for 41 jobs and saw a few jobs that were previously failing succeeded. Calling this closed, please reopen if you notice otherwise.

Legoktm added a subscriber: Legoktm.Dec 7 2017, 4:07 AM

Do we know why this suddenly just happened?

Tgr added a comment.Dec 8 2017, 11:25 PM

Maybe changed, the old content got cached in some proxy somewhere, and HTTPS_PROXY_REQUEST_FULLURI broke the cache by changing the request URL? (In which case the root problem would be in the proxy we are using.)

Legoktm added a subscriber: hashar.Dec 9 2017, 8:42 PM

AFAIK there's no outbound HTTP(S) proxy - @hashar is that correct?

There should not be any proxy intercepting traffic between the labs instance and the internet. As I know it, the instances requests are just passed via port address translation and share some IP.

Seems composer tried to download:" file could not be downloaded (HTTP/1.1 404 Not Found)

My best guess is that .composer/cache/repo/ ended up being stall somehow. But the Nodepool instances have no composer cache:

composer clear-cache
PHP Deprecated:  Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/20-xhprof.ini on line 2 in Unknown on line 0
You are running composer with xdebug enabled. This has a major impact on runtime performance. See
Clearing cache (cache-dir): /home/jenkins/.composer/cache
Cache directory does not exist (cache-files-dir): 
Cache directory does not exist (cache-repo-dir): 
Cache directory does not exist (cache-vcs-dir): 
All caches cleared.

So maybe was stall somehow?

One sure thing, when trying on a Nodepool instance there is no issue happening. Also the images are rebuild at 14:14UTC which does not match with the start of the failure.

Doing a diff via:

$ composer clear-cache && rm -fR composer* vendor && composer clear-cache
$ composer require -vvv wikimedia/ip-set &>ONE.txt

$ composer clear-cache && rm -fR composer* vendor && composer clear-cache
$ (export HTTP_PROXY_REQUEST_FULLURI=false; HTTPS_PROXY_REQUEST_FULLURI=false; composer require -vvv wikimedia/ip-set &>TWO.txt)
- Downloading
+ Downloading

Then if I run the command twice the checksum keeps changing. Based on time maybe? Then when looping a composer require over and over, seems to yield different checksums for some of the provider-xxx.json, which result in composer redownloading the file due to a cache mismatch. Specially with provider-2017-10

I am tempted to say there is/was an issue on packagist proxy/cache.

Files repo-old/ and repo/ differ
Files repo-old/ and repo/ differ
Files repo-old/ and repo/ differ
Files repo-old/ and repo/ differ

Soo hmm is giving out different checksums for the provider-xxx.json files.

I could not reproduce. Guess we want to upgrade composer to a more recent version, and maybe later we can drop those env variables. didnt touch the docker images so perhaps we also need to apply the fix there.

thcipriani lowered the priority of this task from Unbreak Now! to Normal.Jan 2 2018, 7:44 PM

It looks like this change eventually merged and that this test eventually passed ( is a run of the same change). Lowering the priority from UBN accordingly. didnt touch the docker images so perhaps we also need to apply the fix there.

We could apply that fix to the docker images. It was late and it fixed the problem over the short term when we came up with that solution, would be good to find the root cause if we could reproduce.

Tgr added a comment.Jan 4 2018, 11:54 PM

I just got this error on a local vagrant box (so definitely no firewall or proxy or other network interference). Couldn't reproduce it though, the second provisioning run worked fine.

==> default: Notice: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]/returns: Do not run Composer as root/super user! See for details
==> default: Notice: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]/returns: > ComposerHookHandler::onPreUpdate
==> default: Notice: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]/returns: Loading composer repositories with package information
==> default: Notice: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]/returns: Updating dependencies (including require-dev)
==> default: Notice: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]/returns: 
==> default: Notice: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]/returns:                                                                                                                                                                             
==> default: Notice: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]/returns:   [Composer\Downloader\TransportException]                                                                                                                                  
==> default: Notice: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]/returns:   The "" file could not be downloaded (HTTP/1.1 404 Not Found)  
==> default: Notice: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]/returns:                                                                                                                                                                             
==> default: Notice: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]/returns: 
==> default: Notice: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]/returns: update [--prefer-source] [--prefer-dist] [--dry-run] [--dev] [--no-dev] [--lock] [--no-custom-installers] [--no-autoloader] [--no-scripts] [--no-progress] [--no-suggest] [--with-dependencies] [--with-all-dependencies] [-v|vv|vvv|--verbose] [-o|--optimize-autoloader] [-a|--classmap-authoritative] [--apcu-autoloader] [--ignore-platform-reqs] [--prefer-stable] [--prefer-lowest] [-i|--interactive] [--root-reqs] [--] [<packages>]...
==> default: Error: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]: Failed to call refresh: /usr/local/bin/composer update --no-progress returned 255 instead of one of [0]
==> default: Error: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]: /usr/local/bin/composer update --no-progress returned 255 instead of one of [0]
thcipriani removed thcipriani as the assignee of this task.Feb 5 2018, 5:28 PM

This is no longer happening for mediawiki-core-phpcs-docker AFAICT. There's definitely something deeper at work here than HTTPS_PROXY_REQUEST_FULLURI=false that requires a bit of a deeper dive.

Krinkle renamed this task from Composer\Downloader\TransportException in mwext-testextension-hhvm-jessie to Composer\Downloader\TransportException in jobs with mediawiki or composer.
Krinkle added a subscriber: Krinkle.Jun 5 2018, 8:25 PM

This happens both for HHVM and PHP7, and for both MediaWiki core/extension jobs as well as standalone/php-library jobs. Which makes sense, given it is a problem with composer install and not influenced by anything outside of that afaik.

Got one on <> just now:

# exec docker run --rm --env-file /dev/fd/63 --volume /srv/jenkins-workspace/workspace/composer-package-php70-docker/log:/log --volume /srv/jenkins-workspace/workspace/composer-package-php70-docker/cache:/cache --volume /srv/jenkins-workspace/workspace/composer-package-php70-docker/src:/src
# composer install --no-progress --prefer-dist
Loading composer repositories with package information
Updating dependencies (including require-dev)

  The "" file could not be downloaded (HTTP/1.1 404 Not Found)
Finished: FAILURE

Last time I checked ( T182266#3830378 ) was yielding different content. That would indicate an issue on their cache layer. Potentially an old version that did not get purged.

Krinkle renamed this task from Composer\Downloader\TransportException in jobs with mediawiki or composer to Error "TransportException 404 Not Found" in Jenkins jobs using composer.Aug 6 2018, 8:44 PM

composer/packagist#470 seems the same issue.

As it was closed, I've created a new report at

Tgr added a comment.Aug 29 2018, 1:40 PM

I have not seen this error for months and now see it multiple times a day. I doubt it's just bad luck with timing. Something occasionally gets messed either in some cache we use (but last time we did not find any caching) or with the Composer server, and stays messed up for a while, and during that time the error is much more frequent.

Legoktm raised the priority of this task from Normal to Unbreak Now!.Sep 4 2018, 7:09 AM

The frequency of this has drastically increased over the past week. If this were a normal PHPUnit test or something, it would have been disabled by now for extreme flakiness.

I also e-mailed support yesterday, with a pointer to

Krinkle lowered the priority of this task from Unbreak Now! to Normal.Sep 4 2018, 6:49 PM

This should not happen anymore now, it was a race condition between mirrors but I put a fallback in place now so they should go fetch the missing file if a race happens. Please do tell if you see it again.

Legoktm closed this task as Resolved.Sep 7 2018, 2:12 AM
Legoktm assigned this task to Krinkle.

I haven't seen this occur since Tuesday. (npm on the other hand...)