Page MenuHomePhabricator

Error "TransportException 404 Not Found" in Jenkins jobs using composer
Closed, ResolvedPublic

Description

https://gerrit.wikimedia.org/r/#/c/395885/ / https://integration.wikimedia.org/ci/job/mwext-testextension-hhvm-jessie/25185/console

23:34:20 + composer install --ansi --no-progress --prefer-dist --profile -v
23:34:20 [2.0MB/0.04s] Loading composer repositories with package information
23:34:20 [2.0MB/0.49s] Updating dependencies (including require-dev)
23:34:25 
23:34:25                                                                                                                                                                                
23:34:25   [Composer\Downloader\TransportException]                                                                                                                                     
23:34:25   The "http://packagist.org/p/provider-2017-04%2479cabf34c8e4339cec9a3dc680e605399113dccc173bf26455a458d48adf82a2.json" file could not be downloaded (HTTP/1.1 404 Not Found)  
23:34:25                                                                                                                                                                                
23:34:25 
23:34:25 Exception trace:
23:34:25  () at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/Util/RemoteFilesystem.php:350
23:34:25  Composer\Util\RemoteFilesystem->get() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/Util/RemoteFilesystem.php:101
23:34:25  Composer\Util\RemoteFilesystem->getContents() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/Repository/ComposerRepository.php:645
23:34:25  Composer\Repository\ComposerRepository->fetchFile() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/Repository/ComposerRepository.php:558
23:34:25  Composer\Repository\ComposerRepository->loadProviderListings() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/Repository/ComposerRepository.php:289
23:34:25  Composer\Repository\ComposerRepository->whatProvides() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/DependencyResolver/Pool.php:204
23:34:25  Composer\DependencyResolver\Pool->computeWhatProvides() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/DependencyResolver/Pool.php:193
23:34:25  Composer\DependencyResolver\Pool->whatProvides() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/DependencyResolver/RuleSetGenerator.php:271
23:34:25  Composer\DependencyResolver\RuleSetGenerator->whitelistFromJobs() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/DependencyResolver/RuleSetGenerator.php:324
23:34:25  Composer\DependencyResolver\RuleSetGenerator->getRulesFor() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/DependencyResolver/Solver.php:214
23:34:25  Composer\DependencyResolver\Solver->solve() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/Installer.php:455
23:34:25  Composer\Installer->doInstall() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/Installer.php:215
23:34:25  Composer\Installer->run() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/Command/InstallCommand.php:133
23:34:25  Composer\Command\InstallCommand->execute() at /srv/deployment/integration/composer/vendor/symfony/console/Command/Command.php:264
23:34:25  Symfony\Component\Console\Command\Command->run() at /srv/deployment/integration/composer/vendor/symfony/console/Application.php:846
23:34:25  Symfony\Component\Console\Application->doRunCommand() at /srv/deployment/integration/composer/vendor/symfony/console/Application.php:191
23:34:25  Symfony\Component\Console\Application->doRun() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/Console/Application.php:227
23:34:25  Composer\Console\Application->doRun() at /srv/deployment/integration/composer/vendor/symfony/console/Application.php:122
23:34:25  Symfony\Component\Console\Application->run() at /srv/deployment/integration/composer/vendor/composer/composer/src/Composer/Console/Application.php:102
23:34:25  Composer\Console\Application->run() at /srv/deployment/integration/composer/vendor/composer/composer/bin/composer:44

I can run composer install locally on a clean checkout with no issues.

Event Timeline

Tgr created this task.Dec 6 2017, 11:42 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 6 2017, 11:42 PM
Tgr updated the task description. (Show Details)Dec 6 2017, 11:43 PM
Tgr added a comment.Dec 6 2017, 11:50 PM

composer#2198 suggests this could be caused by an outdated Composer executable.

Tgr triaged this task as Unbreak Now! priority.Dec 7 2017, 12:07 AM

Seems to be breaking all MediaWiki extension tests.

Restricted Application added subscribers: Liuxinyu970226, Jay8g, TerraCodes. · View Herald TranscriptDec 7 2017, 12:07 AM
Tgr added a comment.EditedDec 7 2017, 12:29 AM

https://github.com/wikimedia/integration-composer is at 1.1.3. Can't reproduce the error with that version locally; installation seems to work fine:

1tgr@tgr-ThinkPad:~/Wikimedia/grep/extensions/ReadingLists$ wget https://getcomposer.org/download/1.1.3/composer.phar
2--2017-12-06 16:27:37-- https://getcomposer.org/download/1.1.3/composer.phar
3Resolving getcomposer.org (getcomposer.org)... 87.98.253.108, 2001:41d0:a:7b19::2
4Connecting to getcomposer.org (getcomposer.org)|87.98.253.108|:443... connected.
5HTTP request sent, awaiting response... 200 OK
6Length: 1641245 (1.6M) [application/octet-stream]
7Saving to: ‘composer.phar’
8
9composer.phar 100%[===================>] 1.56M 1.14MB/s in 1.4s
10
112017-12-06 16:27:40 (1.14 MB/s) - ‘composer.phar’ saved [1641245/1641245]
12
13tgr@tgr-ThinkPad:~/Wikimedia/grep/extensions/ReadingLists$ ./composer.phar clear-cache
14Clearing cache (cache-dir): /home/tgr/.composer/cache
15Clearing cache (cache-files-dir): /home/tgr/.composer/cache/files
16Clearing cache (cache-repo-dir): /home/tgr/.composer/cache/repo
17Clearing cache (cache-vcs-dir): /home/tgr/.composer/cache/vcs
18All caches cleared.
19tgr@tgr-ThinkPad:~/Wikimedia/grep/extensions/ReadingLists$ ./composer.phar install -vvv --ignore-platform-reqs
20Reading ./composer.json
21Loading config file ./composer.json
22Checked CA file /etc/ssl/certs/ca-certificates.crt: valid
23Executing command (/home/tgr/Wikimedia/grep/extensions/ReadingLists): git branch --no-color --no-abbrev -v
24Failed to initialize global composer: Composer could not find the config file: /home/tgr/.composer/composer.json
25To initialize a project, please create a composer.json file as described in the https://getcomposer.org/ "Getting Started" section
26Running 1.1.3 (2016-06-26 15:42:08) with PHP 7.0.22-0ubuntu0.16.04.1 on Linux / 4.4.0-101-generic
27Loading composer repositories with package information
28Downloading https://packagist.org/packages.json
29Writing /home/tgr/.composer/cache/repo/https---packagist.org/packages.json into cache
30Updating dependencies (including require-dev)
31Downloading http://packagist.org/p/provider-2013%24082ae17536d6bbcc5149ec2635eae612373d7adfa1068e8179da007957831c7e.json
32Writing /home/tgr/.composer/cache/repo/https---packagist.org/p-provider-2013.json into cache
33Downloading http://packagist.org/p/provider-2014%2467ac94409dcd4da75fc57158057f09f00619a11a17511cac4b7b3bd3ceff49d4.json
34Writing /home/tgr/.composer/cache/repo/https---packagist.org/p-provider-2014.json into cache
35Downloading http://packagist.org/p/provider-2015%2434ad80fc77a418239d70f27e503496fa83d137afadc9cb6d0829cbc962b2a717.json
36Writing /home/tgr/.composer/cache/repo/https---packagist.org/p-provider-2015.json into cache
37Downloading http://packagist.org/p/provider-2016%24511db3a77b1d848249f82879927d480608ad55851f9633dd3ea1d9d19cf2ed45.json
38Writing /home/tgr/.composer/cache/repo/https---packagist.org/p-provider-2016.json into cache
39Downloading http://packagist.org/p/provider-2017-01%24a269f860b1da8e96129b32c7e76b6171182cd26aa581d5b9ccbe7ddf2cc2ba98.json
40Writing /home/tgr/.composer/cache/repo/https---packagist.org/p-provider-2017-01.json into cache
41Downloading http://packagist.org/p/provider-2017-04%2447568496b5ec7309a3b1a6a172620bc4729dda2acb59c85648bd6d25b6a99ffc.json
42Writing /home/tgr/.composer/cache/repo/https---packagist.org/p-provider-2017-04.json into cache
43Downloading http://packagist.org/p/provider-2017-07%24137c9da42c3aec6b5d7f999d56a285376bd3ecf71c9a79e6c35bc0ea8e6350f8.json
44Writing /home/tgr/.composer/cache/repo/https---packagist.org/p-provider-2017-07.json into cache
45Downloading http://packagist.org/p/provider-2017-10%244e596f3871cf290b620c2384d01a56a2dd5b7e92a0d9486775c5d149166fe683.json
46Writing /home/tgr/.composer/cache/repo/https---packagist.org/p-provider-2017-10.json into cache
47Downloading http://packagist.org/p/provider-archived%247a808bf19f27e2a2116fba474877ad621855e971aaf72f7523a2aa0535378b20.json
48Writing /home/tgr/.composer/cache/repo/https---packagist.org/p-provider-archived.json into cache
49Downloading http://packagist.org/p/provider-latest%242c29a1504a883209540655248ad5d39d3d8291e7d9636c87663cc7472466f188.json
50Writing /home/tgr/.composer/cache/repo/https---packagist.org/p-provider-latest.json into cache
51Downloading http://packagist.org/p/jakub-onderka/php-parallel-lint%24a2f7ebb22ccdc526cb1a8d978c49fd94b233e75546d3681f7d8c139b3147e744.json
52Writing /home/tgr/.composer/cache/repo/https---packagist.org/provider-jakub-onderka$php-parallel-lint.json into cache
53Downloading http://packagist.org/p/mediawiki/mediawiki-codesniffer%24ff7aa8bae9cb074b5882b341f76d895b7a8e211a7a92320a686a555749de6308.json
54Writing /home/tgr/.composer/cache/repo/https---packagist.org/provider-mediawiki$mediawiki-codesniffer.json into cache
55Downloading http://packagist.org/p/squizlabs/php_codesniffer%243521b11cf8159d9fe7dc66dc986d5990995e2bbe370ecee282c13bb54725e189.json
56Writing /home/tgr/.composer/cache/repo/https---packagist.org/provider-squizlabs$php-codesniffer.json into cache
57Downloading http://packagist.org/p/jakub-onderka/php-console-highlighter%24c012b0a5b32c95d37e219e944a53ffcf9beaee5e16f2eec4d1980872fd4b7762.json
58Writing /home/tgr/.composer/cache/repo/https---packagist.org/provider-jakub-onderka$php-console-highlighter.json into cache
59Downloading http://packagist.org/p/jakub-onderka/php-console-color%240c572771d4a400e1924fe4c0f86a66dbd82a800bcdff4ba775edda09365aaf4b.json
60Writing /home/tgr/.composer/cache/repo/https---packagist.org/provider-jakub-onderka$php-console-color.json into cache
61Downloading http://packagist.org/p/mediawiki/minus-x%241a4043cdd1a55d057733a5b8830ec9ae30c70a3219945828aa03123e025a9e80.json
62Writing /home/tgr/.composer/cache/repo/https---packagist.org/provider-mediawiki$minus-x.json into cache
63Downloading http://packagist.org/p/symfony/console%249e393534a970979cb45345d87a61e3a4e75fdbb0e643d9fa7e0a711b0eac03ba.json
64Writing /home/tgr/.composer/cache/repo/https---packagist.org/provider-symfony$console.json into cache
65Downloading http://packagist.org/p/symfony/polyfill-mbstring%2431f103756f8c66e65b4abffa8759e8d5427b741fb9342190095e67df13e36cf1.json
66Writing /home/tgr/.composer/cache/repo/https---packagist.org/provider-symfony$polyfill-mbstring.json into cache
67Downloading http://packagist.org/p/symfony/debug%240fa6b797833c65937d7b3ff32ab822e3523c3ef61f4cde5417e962545bd3027c.json
68Writing /home/tgr/.composer/cache/repo/https---packagist.org/provider-symfony$debug.json into cache
69Downloading http://packagist.org/p/psr/log%243160233eb27d18b62531dc5b72ae6a26e277e41a1d7f5f0bf78735635c3717fc.json
70Writing /home/tgr/.composer/cache/repo/https---packagist.org/provider-psr$log.json into cache
71Downloading http://packagist.org/p/symfony/dependency-injection%24e58ac93f561dec2e485ce97391447381f6fb1aca79386961f38785511ca78517.json
72Writing /home/tgr/.composer/cache/repo/https---packagist.org/provider-symfony$dependency-injection.json into cache
73Downloading http://packagist.org/p/symfony/http-kernel%245a5d00b085d74df8f8f0c1612657c3b8cef9f19b5491e1b272b0d2b7aaff0b3f.json
74Writing /home/tgr/.composer/cache/repo/https---packagist.org/provider-symfony$http-kernel.json into cache
75Resolving dependencies through SAT
76Dependency resolution completed in 0.007 seconds
77Analyzed 1375 packages to resolve dependencies
78Analyzed 3841 rules to resolve dependencies
79Resolving dependencies through SAT
80Dependency resolution completed in 0.000 seconds
81 - Installing jakub-onderka/php-parallel-lint (v0.9.2)
82Downloading https://api.github.com/repos/JakubOnderka/PHP-Parallel-Lint/zipball/2ead2e4043ab125bee9554f356e0a86742c2d4fa
83 Downloading: 100%
84Writing /home/tgr/.composer/cache/files/jakub-onderka/php-parallel-lint/f8258503dfb598cb0722ac1165ce32d683a40473.zip into cache from /home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/jakub-onderka/php-parallel-lint/2acd828cf8f1474ea521abc952596fff
85 Extracting archive
86Executing command (CWD): unzip '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/jakub-onderka/php-parallel-lint/2acd828cf8f1474ea521abc952596fff' -d '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/fdd40091' && chmod -R u+w '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/fdd40091'
87
88 REASON: Required by the root package: Install command rule (install jakub-onderka/php-parallel-lint v0.9.2)
89
90 - Installing squizlabs/php_codesniffer (3.1.1)
91Downloading https://api.github.com/repos/squizlabs/PHP_CodeSniffer/zipball/d667e245d5dcd4d7bf80f26f2c947d476b66213e
92 Downloading: 100%
93Writing /home/tgr/.composer/cache/files/squizlabs/php_codesniffer/ed88e1fdc0ef8619f54c3205e01909156407e6a7.zip into cache from /home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/squizlabs/php_codesniffer/c61fa8f2ac7513ef4279da872f17d280
94 Extracting archive
95Executing command (CWD): unzip '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/squizlabs/php_codesniffer/c61fa8f2ac7513ef4279da872f17d280' -d '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/0e504644' && chmod -R u+w '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/0e504644'
96
97 REASON: mediawiki/mediawiki-codesniffer v14.1.0 requires squizlabs/php_codesniffer 3.1.1 -> satisfiable by squizlabs/php_codesniffer[3.1.1].
98
99 - Installing mediawiki/mediawiki-codesniffer (v14.1.0)
100Downloading https://api.github.com/repos/wikimedia/mediawiki-tools-codesniffer/zipball/092ce78529556a3b774c0920a2a5d40cefb4207a
101 Downloading: 100%
102Writing /home/tgr/.composer/cache/files/mediawiki/mediawiki-codesniffer/15aff3cbb32fe1c86438613dc9cf35d9b5132cd9.zip into cache from /home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/mediawiki/mediawiki-codesniffer/79a3bc434c01c25cc753bd6784086e65
103 Extracting archive
104Executing command (CWD): unzip '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/mediawiki/mediawiki-codesniffer/79a3bc434c01c25cc753bd6784086e65' -d '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/e5bde2f1' && chmod -R u+w '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/e5bde2f1'
105
106 REASON: Required by the root package: Install command rule (install mediawiki/mediawiki-codesniffer v14.1.0)
107
108 - Installing jakub-onderka/php-console-color (0.1)
109Downloading https://api.github.com/repos/JakubOnderka/PHP-Console-Color/zipball/e0b393dacf7703fc36a4efc3df1435485197e6c1
110 Downloading: 100%
111Writing /home/tgr/.composer/cache/files/jakub-onderka/php-console-color/49d780f73590c804c696360b8a7850232c99d48c.zip into cache from /home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/jakub-onderka/php-console-color/dea5abc5c7088f768a188f290f325403
112 Extracting archive
113Executing command (CWD): unzip '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/jakub-onderka/php-console-color/dea5abc5c7088f768a188f290f325403' -d '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/b06b24ac' && chmod -R u+w '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/b06b24ac'
114
115 REASON: jakub-onderka/php-console-highlighter v0.3.2 requires jakub-onderka/php-console-color ~0.1 -> satisfiable by jakub-onderka/php-console-color[0.1].
116
117 - Installing jakub-onderka/php-console-highlighter (v0.3.2)
118Downloading https://api.github.com/repos/JakubOnderka/PHP-Console-Highlighter/zipball/7daa75df45242c8d5b75a22c00a201e7954e4fb5
119 Downloading: 100%
120Writing /home/tgr/.composer/cache/files/jakub-onderka/php-console-highlighter/70007c6ea8a15cefdba185100cd9ee87e3f912bd.zip into cache from /home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/jakub-onderka/php-console-highlighter/7a9beb063535e330c839b84ccfb8ba52
121 Extracting archive
122Executing command (CWD): unzip '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/jakub-onderka/php-console-highlighter/7a9beb063535e330c839b84ccfb8ba52' -d '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/ddb0d0d4' && chmod -R u+w '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/ddb0d0d4'
123
124 REASON: Required by the root package: Install command rule (install jakub-onderka/php-console-highlighter v0.3.2)
125
126 - Installing psr/log (1.0.2)
127Downloading https://api.github.com/repos/php-fig/log/zipball/4ebe3a8bf773a19edfe0a84b6585ba3d401b724d
128 Downloading: 100%
129Writing /home/tgr/.composer/cache/files/psr/log/216fe71da49480fa51edd2c2fe0f7f93d5e2714a.zip into cache from /home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/psr/log/34c4d0bac8590e7036931785cb9014a7
130 Extracting archive
131Executing command (CWD): unzip '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/psr/log/34c4d0bac8590e7036931785cb9014a7' -d '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/2c5d0d39' && chmod -R u+w '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/2c5d0d39'
132
133 REASON: symfony/debug v3.4.1 requires psr/log ~1.0 -> satisfiable by psr/log[1.0.0, 1.0.1, 1.0.2].
134
135 - Installing symfony/debug (v3.4.1)
136Downloading https://api.github.com/repos/symfony/debug/zipball/fb2001e5d85f95d8b6ab94ae3be5d2672df128fd
137 Downloading: 100%
138Writing /home/tgr/.composer/cache/files/symfony/debug/bcc7d7aa7c8fb262a6d116be45354fd2442182a6.zip into cache from /home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/symfony/debug/1c20455faf60b221363372ec3fe12f00
139 Extracting archive
140Executing command (CWD): unzip '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/symfony/debug/1c20455faf60b221363372ec3fe12f00' -d '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/e64883a1' && chmod -R u+w '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/e64883a1'
141
142 REASON: symfony/console v3.3.5 requires symfony/debug ~2.8|~3.0 -> satisfiable by symfony/debug[v2.8.0, v2.8.1, v2.8.10, v2.8.11, v2.8.12, v2.8.13, v2.8.14, v2.8.15, v2.8.16, v2.8.17, v2.8.18, v2.8.19, v2.8.2, v2.8.20, v2.8.21, v2.8.22, v2.8.23, v2.8.24, v2.8.25, v2.8.26, v2.8.27, v2.8.28, v2.8.29, v2.8.3, v2.8.30, v2.8.31, v2.8.32, v2.8.4, v2.8.5, v2.8.6, v2.8.7, v2.8.8, v2.8.9, v3.0.0, v3.0.1, v3.0.2, v3.0.3, v3.0.4, v3.0.5, v3.0.6, v3.0.7, v3.0.8, v3.0.9, v3.1.0, v3.1.1, v3.1.10, v3.1.2, v3.1.3, v3.1.4, v3.1.5, v3.1.6, v3.1.7, v3.1.8, v3.1.9, v3.2.0, v3.2.1, v3.2.10, v3.2.11, v3.2.12, v3.2.13, v3.2.14, v3.2.2, v3.2.3, v3.2.4, v3.2.5, v3.2.6, v3.2.7, v3.2.8, v3.2.9, v3.3.0, v3.3.1, v3.3.10, v3.3.11, v3.3.12, v3.3.13, v3.3.14, v3.3.2, v3.3.3, v3.3.4, v3.3.5, v3.3.6, v3.3.7, v3.3.8, v3.3.9, v3.4.0, v3.4.1].
143
144 - Installing symfony/polyfill-mbstring (v1.6.0)
145Downloading https://api.github.com/repos/symfony/polyfill-mbstring/zipball/2ec8b39c38cb16674bbf3fea2b6ce5bf117e1296
146 Downloading: 100%
147Writing /home/tgr/.composer/cache/files/symfony/polyfill-mbstring/9ed0687870fa5a494ce6327719b4b295969ed49e.zip into cache from /home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/symfony/polyfill-mbstring/5aa812aa8f79f5e03717b7be593d6a4b
148 Extracting archive
149Executing command (CWD): unzip '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/symfony/polyfill-mbstring/5aa812aa8f79f5e03717b7be593d6a4b' -d '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/afe9485c' && chmod -R u+w '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/afe9485c'
150
151 REASON: symfony/console v3.3.5 requires symfony/polyfill-mbstring ~1.0 -> satisfiable by symfony/polyfill-mbstring[v1.0.0, v1.0.1, v1.1.0, v1.1.1, v1.2.0, v1.3.0, v1.3.1, v1.4.0, v1.5.0, v1.6.0].
152
153 - Installing symfony/console (v3.3.5)
154Downloading https://api.github.com/repos/symfony/console/zipball/a97e45d98c59510f085fa05225a1acb74dfe0546
155 Downloading: 100%
156Writing /home/tgr/.composer/cache/files/symfony/console/c6c3a14ce3796482f5be6ff504bf2326e14e5e9f.zip into cache from /home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/symfony/console/f94003d2ee747342a2c0ad34d26b42f5
157 Extracting archive
158Executing command (CWD): unzip '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/symfony/console/f94003d2ee747342a2c0ad34d26b42f5' -d '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/7f32aad7' && chmod -R u+w '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/7f32aad7'
159
160 REASON: mediawiki/minus-x 0.2.0 requires symfony/console 3.3.5 -> satisfiable by symfony/console[v3.3.5].
161
162 - Installing mediawiki/minus-x (0.2.0)
163Downloading https://api.github.com/repos/wikimedia/mediawiki-tools-minus-x/zipball/5655eedf572728c4210cecd064d9146fba58983c
164 Downloading: 100%
165Writing /home/tgr/.composer/cache/files/mediawiki/minus-x/6d9b080862224f93c179627b86ec5484581fe6d6.zip into cache from /home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/mediawiki/minus-x/07e8bcea62cc2caaa9fd576e47cd6438
166 Extracting archive
167Executing command (CWD): unzip '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/mediawiki/minus-x/07e8bcea62cc2caaa9fd576e47cd6438' -d '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/fe28386f' && chmod -R u+w '/home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/fe28386f'
168
169 REASON: Required by the root package: Install command rule (install mediawiki/minus-x 0.2.0)
170
171symfony/console suggests installing symfony/event-dispatcher ()
172symfony/console suggests installing symfony/filesystem ()
173symfony/console suggests installing symfony/process ()
174Reading /home/tgr/Wikimedia/grep/extensions/ReadingLists/vendor/composer/installed.json
175Writing lock file
176Generating autoload files

Tgr added a comment.Dec 7 2017, 1:04 AM

Looking at the Composer code it seems like these URLs come from https://packagist.org/packages.json by replacing %hash% with the sha256 parameter (1.1.3, master - no difference). So not sure how outdated Composer code could cause this. An outdated cached version of packages.json maybe? But I'd expect those to get regenerated as CI containers are created and destroyed.

Arlolra added a subscriber: Arlolra.Dec 7 2017, 1:07 AM

I've regenerated the nodepool jessie snapshot using the instructions from: https://wikitech.wikimedia.org/wiki/Nodepool#Manually_generate_a_new_snapshot

Although the timeline of this problem doesn't match up with when the last snapshot was built (2017-12-05 14:14).

Composer diag on a nodepool instance shows with mw-config

jenkins@ci-jessie-wikimedia-910534:~/space$ composer diag                                                                                                           
PHP Deprecated:  Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/20-xhprof.ini on line 2 in Unknown on line 0                                          
You are running composer with xdebug enabled. This has a major impact on runtime performance. See https://getcomposer.org/xdebug                                         
Checking composer.json: FAIL                                                                                                                                             
No license specified, it is recommended to do so. For closed-source software you may use "proprietary" as license.                                                       
require.mongofill/mongofill : unbound version constraints (dev-master) should be avoided                                                                                 
Checking platform settings: FAIL                                                                                                                                         
The xdebug extension is loaded, this can slow down Composer a little.                                                                                                    
 Disabling it when using Composer is recommended.                                                                                                                        
                                                                                                                                                                         
Checking git settings: OK                                                                                                                                                
Checking http connectivity to packagist: Warning: Accessing packagist.org over http which is an insecure protocol.                                                       
OK                                                                                                                                                                       
Checking https connectivity to packagist: OK                                                                                                                             
Checking github.com rate limit: OK                                                                                                                                       
Checking disk free space: OK

so for some reason adding HTTPS_PROXY_REQUEST_FULLURI=false seems to fix this https://integration.wikimedia.org/ci/job/operations-mw-config-composer-hhvm-jessie/9083/console (found via https://github.com/composer/composer/issues/1839)

However, this didn't work:

export HTTP_PROXY_REQUEST_FULLURI=false
export HTTPS_PROXY_REQUEST_FULLURI=false 
node /srv/deployment/integration/slave-scripts/tools/composer-dev-args.js "$(pwd)/composer.json" | xargs --verbose composer require --dev --ansi --no-progress --ignore-platform-reqs --prefer-dist -v

but this did:

HTTP_PROXY_REQUEST_FULLURI=false HTTPS_PROXY_REQUEST_FULLURI=false composer require --dev --ansi --no-progress --ignore-platform-reqs --prefer-dist -v jakub-onderka/php-parallel-lint=0.9.2 mediawiki/mediawiki-codesniffer=14.1.0 phpunit/phpunit=4.8.36

Change 395913 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[integration/config@master] HTTPS_PROXY_REQUEST_FULLURI=false for composer

https://gerrit.wikimedia.org/r/395913

Change 395913 merged by jenkins-bot:
[integration/config@master] HTTPS_PROXY_REQUEST_FULLURI=false for composer

https://gerrit.wikimedia.org/r/395913

thcipriani closed this task as Resolved.Dec 7 2017, 2:53 AM
thcipriani claimed this task.

Alright, I've deployed the fix for 41 jobs and saw a few jobs that were previously failing succeeded. Calling this closed, please reopen if you notice otherwise.

Legoktm added a subscriber: Legoktm.Dec 7 2017, 4:07 AM

Do we know why this suddenly just happened?

Tgr added a comment.Dec 8 2017, 11:25 PM

Maybe https://packagist.org/packages.json changed, the old content got cached in some proxy somewhere, and HTTPS_PROXY_REQUEST_FULLURI broke the cache by changing the request URL? (In which case the root problem would be in the proxy we are using.)

Legoktm added a subscriber: hashar.Dec 9 2017, 8:42 PM

AFAIK there's no outbound HTTP(S) proxy - @hashar is that correct?

There should not be any proxy intercepting traffic between the labs instance and the internet. As I know it, the instances requests are just passed via port address translation and share some IP.

Seems composer tried to download:

http://packagist.org/p/provider-2017-04%2437a59cae92e9cbe81dc61e72331b18726d63bd97aca2784a0886b7e937f30724.json" file could not be downloaded (HTTP/1.1 404 Not Found)

My best guess is that .composer/cache/repo/https---packagist.org/packages.json ended up being stall somehow. But the Nodepool instances have no composer cache:

composer clear-cache
PHP Deprecated:  Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/20-xhprof.ini on line 2 in Unknown on line 0
You are running composer with xdebug enabled. This has a major impact on runtime performance. See https://getcomposer.org/xdebug
Clearing cache (cache-dir): /home/jenkins/.composer/cache
Cache directory does not exist (cache-files-dir): 
Cache directory does not exist (cache-repo-dir): 
Cache directory does not exist (cache-vcs-dir): 
All caches cleared.

So maybe https://packagist.org/packages.json was stall somehow?

One sure thing, when trying on a Nodepool instance there is no issue happening. Also the images are rebuild at 14:14UTC which does not match with the start of the failure.

Doing a diff via:

$ composer clear-cache && rm -fR composer* vendor && composer clear-cache
$ composer require -vvv wikimedia/ip-set &>ONE.txt

$ composer clear-cache && rm -fR composer* vendor && composer clear-cache
$ (export HTTP_PROXY_REQUEST_FULLURI=false; HTTPS_PROXY_REQUEST_FULLURI=false; composer require -vvv wikimedia/ip-set &>TWO.txt)
- Downloading http://packagist.org/p/provider-2017-10%245814f55cb7472b6ae705b4d3bc61c57036d5197bdaf041667460bd77bef89dd4.json
+ Downloading http://packagist.org/p/provider-2017-10%24ccfcfb70028b0584528cdef771e0105be59d0603e6b5c8880a1364099587fbd8.json

Then if I run the command twice the checksum keeps changing. Based on time maybe? Then when looping a composer require over and over, https://packagist.org/packages.json seems to yield different checksums for some of the provider-xxx.json, which result in composer redownloading the file due to a cache mismatch. Specially with provider-2017-10

I am tempted to say there is/was an issue on packagist proxy/cache.

Files repo-old/https---packagist.org/packages.json and repo/https---packagist.org/packages.json differ
Files repo-old/https---packagist.org/p-provider-2017-04.json and repo/https---packagist.org/p-provider-2017-04.json differ
Files repo-old/https---packagist.org/p-provider-2017-10.json and repo/https---packagist.org/p-provider-2017-10.json differ
Files repo-old/https---packagist.org/p-provider-latest.json and repo/https---packagist.org/p-provider-latest.json differ

Soo hmm https://packagist.org/packages.json is giving out different checksums for the provider-xxx.json files.

I could not reproduce. Guess we want to upgrade composer to a more recent version, and maybe later we can drop those env variables.

https://gerrit.wikimedia.org/r/#/c/395913/ didnt touch the docker images so perhaps we also need to apply the fix there.

thcipriani lowered the priority of this task from Unbreak Now! to Normal.Jan 2 2018, 7:44 PM

It looks like this change eventually merged and that this test eventually passed (https://integration.wikimedia.org/ci/job/mediawiki-core-phpcs-docker/3352/console is a run of the same change). Lowering the priority from UBN accordingly.

https://gerrit.wikimedia.org/r/#/c/395913/ didnt touch the docker images so perhaps we also need to apply the fix there.

We could apply that fix to the docker images. It was late and it fixed the problem over the short term when we came up with that solution, would be good to find the root cause if we could reproduce.

Tgr added a comment.Jan 4 2018, 11:54 PM

I just got this error on a local vagrant box (so definitely no firewall or proxy or other network interference). Couldn't reproduce it though, the second provisioning run worked fine.

==> default: Notice: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]/returns: Do not run Composer as root/super user! See https://getcomposer.org/root for details
==> default: Notice: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]/returns: > ComposerHookHandler::onPreUpdate
==> default: Notice: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]/returns: Loading composer repositories with package information
==> default: Notice: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]/returns: Updating dependencies (including require-dev)
==> default: Notice: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]/returns: 
==> default: Notice: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]/returns:                                                                                                                                                                             
==> default: Notice: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]/returns:   [Composer\Downloader\TransportException]                                                                                                                                  
==> default: Notice: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]/returns:   The "http://packagist.org/p/provider-2017%242ffaf177611f8b7aa72cb02383b422b463c4f82790151ab9dac12ebcf9b8995c.json" file could not be downloaded (HTTP/1.1 404 Not Found)  
==> default: Notice: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]/returns:                                                                                                                                                                             
==> default: Notice: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]/returns: 
==> default: Notice: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]/returns: update [--prefer-source] [--prefer-dist] [--dry-run] [--dev] [--no-dev] [--lock] [--no-custom-installers] [--no-autoloader] [--no-scripts] [--no-progress] [--no-suggest] [--with-dependencies] [--with-all-dependencies] [-v|vv|vvv|--verbose] [-o|--optimize-autoloader] [-a|--classmap-authoritative] [--apcu-autoloader] [--ignore-platform-reqs] [--prefer-stable] [--prefer-lowest] [-i|--interactive] [--root-reqs] [--] [<packages>]...
==> default: Error: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]: Failed to call refresh: /usr/local/bin/composer update --no-progress returned 255 instead of one of [0]
==> default: Error: /Stage[main]/Mediawiki/Exec[composer update /vagrant/mediawiki]: /usr/local/bin/composer update --no-progress returned 255 instead of one of [0]
thcipriani removed thcipriani as the assignee of this task.Feb 5 2018, 5:28 PM

This is no longer happening for mediawiki-core-phpcs-docker AFAICT. There's definitely something deeper at work here than HTTPS_PROXY_REQUEST_FULLURI=false that requires a bit of a deeper dive.

Krinkle renamed this task from Composer\Downloader\TransportException in mwext-testextension-hhvm-jessie to Composer\Downloader\TransportException in jobs with mediawiki or composer.
Krinkle added a subscriber: Krinkle.Jun 5 2018, 8:25 PM

This happens both for HHVM and PHP7, and for both MediaWiki core/extension jobs as well as standalone/php-library jobs. Which makes sense, given it is a problem with composer install and not influenced by anything outside of that afaik.

Got one on <> just now:

https://integration.wikimedia.org/ci/job/composer-package-php70-docker/2335/console

# exec docker run --rm --env-file /dev/fd/63 --volume /srv/jenkins-workspace/workspace/composer-package-php70-docker/log:/log --volume /srv/jenkins-workspace/workspace/composer-package-php70-docker/cache:/cache --volume /srv/jenkins-workspace/workspace/composer-package-php70-docker/src:/src docker-registry.wikimedia.org/releng/composer-package:0.1.3
..
# composer install --no-progress --prefer-dist
Loading composer repositories with package information
Updating dependencies (including require-dev)

  [Composer\Downloader\TransportException]                                                                                                                                     
  The "http://packagist.org/p/provider-2018-01%2436b3015ac80446b5833d620f8bf99bee407e5da061c33faa5c2f67f9f326facd.json" file could not be downloaded (HTTP/1.1 404 Not Found)
...
Finished: FAILURE

Last time I checked ( T182266#3830378 ) https://packagist.org/packages.json was yielding different content. That would indicate an issue on their cache layer. Potentially an old version that did not get purged.

Krinkle renamed this task from Composer\Downloader\TransportException in jobs with mediawiki or composer to Error "TransportException 404 Not Found" in Jenkins jobs using composer.Aug 6 2018, 8:44 PM

composer/packagist#470 seems the same issue.

As it was closed, I've created a new report at https://github.com/composer/packagist/issues/940.

Tgr added a comment.Aug 29 2018, 1:40 PM

I have not seen this error for months and now see it multiple times a day. I doubt it's just bad luck with timing. Something occasionally gets messed either in some cache we use (but last time we did not find any caching) or with the Composer server, and stays messed up for a while, and during that time the error is much more frequent.

Legoktm raised the priority of this task from Normal to Unbreak Now!.Sep 4 2018, 7:09 AM

The frequency of this has drastically increased over the past week. If this were a normal PHPUnit test or something, it would have been disabled by now for extreme flakiness.

I also e-mailed packagist.org support yesterday, with a pointer to https://github.com/composer/packagist/issues/940.

Krinkle lowered the priority of this task from Unbreak Now! to Normal.Sep 4 2018, 6:49 PM

This should not happen anymore now, it was a race condition between mirrors but I put a fallback in place now so they should go fetch the missing file if a race happens. Please do tell if you see it again.

Legoktm closed this task as Resolved.Sep 7 2018, 2:12 AM
Legoktm assigned this task to Krinkle.

I haven't seen this occur since Tuesday. (npm on the other hand...)