Page MenuHomePhabricator

Vagrant: Composer runs out of memory during vagrant git-update
Open, Needs TriagePublic

Description

When running vagrant git-update I get

Fatal error: Out of memory (allocated 141557760) (tried to allocate 2167538 bytes) in phar:///usr/local/bin/composer/src/Composer/Util/RemoteFilesystem.php on line 461

> ComposerHookHandler::onPreUpdate
Loading composer repositories with package information
Updating dependencies (including require-dev)

mmap() failed: [12] Cannot allocate memory
PHP Fatal error:  Out of memory (allocated 141557760) (tried to allocate 2167538 bytes) in phar:///usr/local/bin/composer/src/Composer/Util/RemoteFilesystem.php on line 461

Same for a lot of other repos, not just MediaWiki. The error is stochastic but happens frequently.

It's a standard box, so 1.5G memory.

run-git-update uses GNU Parallel with as many parallel jobs as the number of cores (which by default Vagrant inherits from the real machine), so how often this happens depends on the Vagrant memory and CPU settings, the number of cores in the host, and the number of extensions installed (not sure why that is, but empirically boxes with lots of roles seem to fail more often). In some setups, it happens pretty much deterministically.

In theory, git updates (also run in parallel) could also produce a similar error but I haven't seen that happen yet.

Event Timeline

Seems to always happen in the 140-180M range, even when the machine has more RAM. Too many composer threads run in parallel?

Change 570488 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/vagrant@master] Try to fix vagrant git-update memory issues

https://gerrit.wikimedia.org/r/570488

Change 570488 merged by jenkins-bot:
[mediawiki/vagrant@master] Try to fix vagrant git-update memory issues

https://gerrit.wikimedia.org/r/570488

Tgr renamed this task from Vagrant: Composer runs out of memory to Vagrant: Composer runs out of memory during vagrant git-update.May 2 2021, 3:54 PM
Tgr updated the task description. (Show Details)
Tgr added a subscriber: brion.
Tgr added subscribers: Krinkle, Jrbranaa, bd808, Zppix.

Change 684106 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/vagrant@master] Make number of parallel composer jobs configurable for git-update

https://gerrit.wikimedia.org/r/684106

Change 684106 merged by jenkins-bot:

[mediawiki/vagrant@master] Make number of parallel composer jobs configurable for git-update

https://gerrit.wikimedia.org/r/684106