I have tried some terrible hack along the lines of MODULE=foo; while [[ -n $MODULE ]]; do npm install $MODULE; MODULE=$(npm run selenium |& tee /dev/tty | grep 'Error: Cannot find module' | sed "s/^Error: Cannot find module '//" | sed "s/'$//"); done which installs several modules but at some point it starts complaining about ../core-js/is-iterable module missing and I have no idea what that is or how it could be installed.

The output of npm install often but not always ends with Killed. Given the amount of crap npm outputs, I'm not sure if that means something or not.

I tried to clone the working setup via npm shrinkwrap but no difference - wdio is missing.

@Tgr Not sure I can help much here but curious what the roles are you have enabled on the full instance and what the package.json file looks like? Have you tried a rm -rf node_modules before trying the npm install?

In T196082#4246547, @bearND wrote:

@Tgr Not sure I can help much here but curious what the roles are you have enabled on the full instance

T192306#4244492 has the list. I also enabled a test role:

class role::browsertests_core {
    require ::mediawiki

    require_package('chromedriver')

    npm::install { 'core browser tests':
        directory => $::mediawiki::dir
    }
}

which basically just runs npm install. That might have broken things somehow but I deleted node_modules and recreated by hand a couple times and other than that directory I don't think npm install leaves anything permanent.

and what the package.json file looks like?

https://github.com/wikimedia/mediawiki/blob/master/package.json

Killed sounds like a memory issue. As an experiment, maybe try spinning up a medium-sized instance with the same setup as the one with the failing npm installs and see if you have better luck? I notice that you're currently using all small instances.

I don't have a great explanation for why it would consistently succeed in the other case. Maybe a difference in the underlying hardware between the two instances?

OOM killer. Check $ dmesg, $ journalctl -k, or /var/log/syslog.

Thanks, that's probably the explanation. After restarting the host VM, npm install works fine. I guess something got stuck in memory during the provisioning process?

mcr-full has a bunch of services running in the background (restbase etc) but that in itself does not cause much of a difference:

vagrant@mcr-base:~$ free -mh
              total        used        free      shared  buff/cache   available
Mem:           2.0G        520M        239M        112M        1.2G        1.2G
Swap:          510M          0B        510M

vagrant@mcr-full:/vagrant/mediawiki$ free -mh
              total        used        free      shared  buff/cache   available
Mem:           2.0G        857M        268M         38M        878M        951M
Swap:          510M        103M        407M

I'm not sure what the issue is. On my machine with Node v10.0.0 after a successful install, npm rebuild executes:

> uws@9.14.0 install /home/stephen/Desktop/foo/mediawiki/node_modules/uws
> node-gyp rebuild > build_log.txt 2>&1 || exit 0

> sauce-connect-launcher@1.2.4 postinstall /home/stephen/Desktop/foo/mediawiki/node_modules/sauce-connect-launcher
> node scripts/install.js || nodejs scripts/install.js

> fibers@2.0.2 install /home/stephen/Desktop/foo/mediawiki/node_modules/fibers
> node build.js || nodejs build.js

These are the install and postinstall scripts that are normally executed after installing each package.json's dependencies and devDependencies. Based on testing, npm rebuild alone won't install packages specified in package.json dependencies and devDependencies. The rebuild may be succeeding but it sounds like fetching the dependencies themselves are not.

I don't have a solution but the following may help with debugging:

• Does the simple case work? mkdir foo && cd foo && npm init -y && npm i webdriverio && find -name wdio. Does the simple case work under vagrant ssh?
• Are node and npm executables located in the same place on each machine? Is NODE_ENV or NODE_PATH set on either machine? In general, it may be useful to compare the outputs of export -p.
• Have you tried removing the node_modules directory on the working mcr-base and reinstalling to hopefully make it fail like mcr-full?
• Binary search the failed dependencies by progressively halving the install.
• Share the NPM debug logs from a fresh npm install on each machine.

I also tried on the node:5 Docker image and a debian-9.3-stretch Labs instance with role::labs::mediawiki_vagrant and Node v6.0.0 but both worked.