@MaxSem would you have any idea what's going on there?

The source repo used is at https://gerrit.wikimedia.org/r/#/admin/projects/maps/kartotherian/package the deploy repo is https://gerrit.wikimedia.org/r/#/admin/projects/maps/kartotherian/deploy (those are not yet synced with diffusion / github, see T182848).

This seems to be a permissions problem, oddly enough. I did manage to build it. The current work-around is to run the build script as root. We should look into this problem more carefully to see why that is the case.

The current work-around is to run the build script as root.

Within docker? What files do I need to do to change that?

It's also worth noting this used to work.

I've successfully built the deploy repo: https://gerrit.wikimedia.org/r/398315

Most of the time these kind of FS errors are related to how docker mounts the volumes. Which version of OS/Docker do you have?

I'm on Ubuntu 17.10, with docker 1.5 (the one provided by Ubuntu, fairly old). I can try to install latest. @mobrovac seemed to have reproduced the issue and I expect him to be running something less ancient than me. Which leads me to think that upgrading might not be sufficient.

Yep, same issue with docker 17.12.0-ce

I've finally managed to reproduce the issue with an Ubuntu VM and the latest version of docker-ce (v17.12). However, I still have no idea what's happening here.

Previously we've seen similar issues with vagrant with certain mounting drivers. I'll check if we're seeing a similar issue here.

My initial theory that this is somehow related to the docker storage driver was incorrect. I've checked overlay, overlay2 and vfs drivers with no results - all end up the same result - build fails.

Another piece of info: after unsuccessfully poking around different ideas I've tried to build it with node 8 and it did build successfully. That indicates that the bug is in a particular version of npm we're using.

Ok, I've come up with a dirty workaround. We need to downgrade npm to version 2 in order to make it work.

To downgrade npm, apply the following patch to service-runner inside the node_modules of the sources repo

diff --git a/lib/docker.js b/lib/docker.js
index fe442c2..495ef4d 100644
--- a/lib/docker.js
+++ b/lib/docker.js
@@ -217,7 +217,7 @@ function createDockerFile() {
     contents += `${envCommand}\n`;
 
     if (opts.deploy) {
-        let beforeInstall = '';
+        let beforeInstall = `${npmCommand} i npm@2 && `;
         let afterInstall = '';
         if (opts.reshrinkwrap) {
             beforeInstall = 'rm -f npm-shrinkwrap.json && ';
@@ -227,7 +227,7 @@ function createDockerFile() {
         if (pkg.deploy.install_opts) {
             installOpts += `${pkg.deploy.install_opts.join(' ')} `;
         }
-        contents += `CMD ${beforeInstall}${npmCommand} install${installOpts}${afterInstall} && npm install heapdump gc-stats`;
+        contents += `CMD ${beforeInstall} ./node_modules/.bin/npm install${installOpts}${afterInstall} && npm install heapdump gc-stats`;
         if (!opts.reshrinkwrap) {
             contents += '&& if ! [ -e npm-shrinkwrap.json ] ; then npm dedupe ; fi';
         }

If that works I will formalize it a bit to allow clients select npm version along with the node version.

I just tested the patch, and it works for me as well. My guess is something was failing in the build and leaving a non empty directory, which could then not be removed. But honestly, I don't know what I'm talking about.

@Pchelolo thanks a lot for the help!

My guess is something was failing in the build and leaving a nonempty directory, which could then not be removed. But honestly, I don't know what I'm talking about.

I think that's correct, but that's not a build fault, it's npm bug. We've seen issues like this before.

I will add a more generic support for the workaround in service-runner that will allow us to specify npm version in the deploy section of the package.json

I've created a PR for service-runner that will allow specifying the npm version in the deploy section of the package.json file. At least with that, a hack looks a little bit less hacky. Also, it removes the custom installed npm so that we don't really deploy npm to production.

@Pchelolo: thanks for the fix! The PR looks good to me (but again, what do I know...). Ping me when it is merged, and I can verify that it works for maps. In the meantime, we are now truly unblocked on that task and can resume deploying kartotherian (as soon as we have a version to deploy).

Thanks!