Page MenuHomePhabricator

npm install for selenium tests not working on labs-vagrant
Closed, ResolvedPublic

Description

This is the weirdest thing. We have two identical boxes, mcr-base and mcr-full. Both use labs-vagrant, one as a default setup, the other one with a bunch of roles enabled. I followed the steps at Selenium/Node.js/Inside_MediaWiki-Vagrant to set up browser tests (which basically consist of npm install). In mcr-base it works, in mcr-full the dependencies of the webdriver package are not installed.

The two machines use the same base image with the same amount of resources, same OS version, same node version, same vagrant version. The MediaWiki checkout has the same package.json file in both. npm config ls -l gives the same result. And yet, in mcr-full, npm install consistently fails (without giving any error message) - npm run selenium complains about the wdio binary missing. npm rebuild fixes that but still does not install any dependencies of the webdriver module.

Related Objects

View Standalone Graph
This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.

Event Timeline

Tgr created this task.May 31 2018, 3:44 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 31 2018, 3:44 PM
Tgr updated the task description. (Show Details)May 31 2018, 3:44 PM

I have tried some terrible hack along the lines of MODULE=foo; while [[ -n $MODULE ]]; do npm install $MODULE; MODULE=$(npm run selenium |& tee /dev/tty | grep 'Error: Cannot find module' | sed "s/^Error: Cannot find module '//" | sed "s/'$//"); done which installs several modules but at some point it starts complaining about ../core-js/is-iterable module missing and I have no idea what that is or how it could be installed.

Tgr added a comment.May 31 2018, 4:03 PM

The output of npm install often but not always ends with Killed. Given the amount of crap npm outputs, I'm not sure if that means something or not.

I tried to clone the working setup via npm shrinkwrap but no difference - wdio is missing.

bearND added a subscriber: bearND.EditedMay 31 2018, 5:04 PM

@Tgr Not sure I can help much here but curious what the roles are you have enabled on the full instance and what the package.json file looks like? Have you tried a rm -rf node_modules before trying the npm install?

Tgr added a comment.May 31 2018, 5:12 PM

@Tgr Not sure I can help much here but curious what the roles are you have enabled on the full instance

T192306#4244492 has the list. I also enabled a test role:

class role::browsertests_core {
    require ::mediawiki

    require_package('chromedriver')

    npm::install { 'core browser tests':
        directory => $::mediawiki::dir
    }
}

which basically just runs npm install. That might have broken things somehow but I deleted node_modules and recreated by hand a couple times and other than that directory I don't think npm install leaves anything permanent.

and what the package.json file looks like?

https://github.com/wikimedia/mediawiki/blob/master/package.json

Mholloway added a subscriber: Mholloway.EditedMay 31 2018, 5:52 PM

Killed sounds like a memory issue. As an experiment, maybe try spinning up a medium-sized instance with the same setup as the one with the failing npm installs and see if you have better luck? I notice that you're currently using all small instances.

I don't have a great explanation for why it would consistently succeed in the other case. Maybe a difference in the underlying hardware between the two instances?

OOM killer. Check $ dmesg, $ journalctl -k, or /var/log/syslog.

Tgr added a comment.May 31 2018, 6:47 PM

Thanks, that's probably the explanation. After restarting the host VM, npm install works fine. I guess something got stuck in memory during the provisioning process?

mcr-full has a bunch of services running in the background (restbase etc) but that in itself does not cause much of a difference:

vagrant@mcr-base:~$ free -mh
              total        used        free      shared  buff/cache   available
Mem:           2.0G        520M        239M        112M        1.2G        1.2G
Swap:          510M          0B        510M
vagrant@mcr-full:/vagrant/mediawiki$ free -mh
              total        used        free      shared  buff/cache   available
Mem:           2.0G        857M        268M         38M        878M        951M
Swap:          510M        103M        407M

I'm not sure what the issue is. On my machine with Node v10.0.0 after a successful install, npm rebuild executes:

> uws@9.14.0 install /home/stephen/Desktop/foo/mediawiki/node_modules/uws
> node-gyp rebuild > build_log.txt 2>&1 || exit 0

> sauce-connect-launcher@1.2.4 postinstall /home/stephen/Desktop/foo/mediawiki/node_modules/sauce-connect-launcher
> node scripts/install.js || nodejs scripts/install.js

> fibers@2.0.2 install /home/stephen/Desktop/foo/mediawiki/node_modules/fibers
> node build.js || nodejs build.js

These are the install and postinstall scripts that are normally executed after installing each package.json's dependencies and devDependencies. Based on testing, npm rebuild alone won't install packages specified in package.json dependencies and devDependencies. The rebuild may be succeeding but it sounds like fetching the dependencies themselves are not.

I don't have a solution but the following may help with debugging:

  • Does the simple case work? mkdir foo && cd foo && npm init -y && npm i webdriverio && find -name wdio. Does the simple case work under vagrant ssh?
  • Are node and npm executables located in the same place on each machine? Is NODE_ENV or NODE_PATH set on either machine? In general, it may be useful to compare the outputs of export -p.
  • Have you tried removing the node_modules directory on the working mcr-base and reinstalling to hopefully make it fail like mcr-full?
  • Binary search the failed dependencies by progressively halving the install.
  • Share the NPM debug logs from a fresh npm install on each machine.

I also tried on the node:5 Docker image and a debian-9.3-stretch Labs instance with role::labs::mediawiki_vagrant and Node v6.0.0 but both worked.

CCicalese_WMF closed this task as Resolved.Jun 7 2018, 4:41 PM
CCicalese_WMF claimed this task.
Vvjjkkii renamed this task from npm install for selenium tests not working on labs-vagrant to lwbaaaaaaa.Jul 1 2018, 1:06 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii triaged this task as High priority.
Vvjjkkii removed CCicalese_WMF as the assignee of this task.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii edited subscribers, added: CCicalese_WMF; removed: Aklapper.
CCicalese_WMF renamed this task from lwbaaaaaaa to npm install for selenium tests not working on labs-vagrant.Jul 2 2018, 9:52 PM
CCicalese_WMF closed this task as Resolved.