npm 1.4.21 can't use a http proxy
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	hashar
	Dec 22 2017, 4:43 PM

Description

When building the Docker image releng/npm on contint1001, the npm upgrade fails:

"/usr/bin/nodejs" "/usr/bin/npm" "install" "-g" "npm@3.8.3"
     TypeError: Request path contains unescaped characters
         at new ClientRequest (_http_client.js:52:11)
         at TunnelingAgent.exports.request (http.js:31:10)
         at TunnelingAgent.createSocket (/usr/lib/nodejs/tunnel-agent/index.js:117:25)

We use the npm 1.4.21 package from Jessie which uses a broken node module tunnelling-agent. That fails when using a http proxy, apparently due to a change in NodeJS API.

Reference:
https://bugs.debian.org/801897 (has the nodejs and module patches)

The issue got fixed in April 2014 via: https://github.com/request/tunnel-agent/pull/8

Probably we could try patching https://packages.debian.org/jessie/node-tunnel-agent and see whether that solves the issue. The deb package is only used to upgrade to npm@3.8.3

How to reproduce

On a Jessie instance, install npm and a http proxy then try to install a single module via npm:

$ sudo apt install npm polipo
$ http_proxy=127.0.0.1:8123 npm install exit
npm ERR! TypeError: Request path contains unescaped characters
...

Details

Subject	Repo	Branch	Lines +/-
Convert npm, npm-test to docker-pkg (2)	integration/config	master	+59 -34
operations/debs/node-tunnel-agent -> voting debian-glue	integration/config	master	+5 -1
debian-glkue for operations/debs/node-tunnel-agent	integration/config	master	+1 -0
Package 0.4.3	operations/debs/node-tunnel-agent	master	+9 -0
Merge tag 'upstream/0.4.3'	operations/debs/node-tunnel-agent	master	+0 -0
pristine-tar data for node-tunnel-agent_0.4.3.orig.tar.gz	operations/debs/node-tunnel-agent	pristine-tar	+1 -0
New upstream version 0.4.3	operations/debs/node-tunnel-agent	upstream	+35 -10

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		hashar	T177276 Unify production and CI docker image build process
		Resolved		hashar	T183569 npm 1.4.21 can't use a http proxy

Event Timeline

hashar created this task.Dec 22 2017, 4:43 PM

Paladox subscribed.Dec 22 2017, 5:07 PM

This comment was removed by Paladox.

Resetting assignee, came from the parent task.

Potentially we could rebuild the Jessie package node-tunnel-agent with patch https://github.com/request/tunnel-agent/pull/8 and that would fix the bootstrap.

Ideally we would want npm 3 or later to be packaged for Stretch but that is a whole can of worms.

hashar added a project: Release-Engineering-Team (Kanban).Jan 8 2018, 5:54 PM

hashar updated the task description. (Show Details)Jan 9 2018, 9:31 AM

Came back to this the tldr is we have to upgrade to 0.4.3

The Debian bug report is https://bugs.debian.org/801897 it mentions an API change in NodeJS as well as a connection issue. The upstream commits are:

3d979c3 Fix various bugs with connection pool handling.
7abb63b agent.addRequest parameters have changed

Both included in 0.4.3, the other changes are meta changes (jshint/package.json). 0.5.x bring in a new dependency safe-buffer which would require us to package it for Debian.

0.4.3 would also add:
a2c2c2a set the defaultPort to 443 for https agents
And I remember that one has been hitting me when I tried to setup a https proxy.

I build the package and gave it a try on a Jessie labs instance, that indeed fix the issue \o/

Change 403124 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/debs/node-tunnel-agent@upstream] New upstream version 0.4.3

https://gerrit.wikimedia.org/r/403124

gerritbot added a project: Patch-For-Review.Jan 9 2018, 10:53 AM

Mentioned in SAL (#wikimedia-releng) [2018-01-09T10:53:58Z] <hashar> gerrit: created operations/debs/node-tunnel-agent a fork of git://anonscm.debian.org/collab-maint/node-tunnel-agent.git | T183569

Change 403125 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/debs/node-tunnel-agent@pristine-tar] pristine-tar data for node-tunnel-agent_0.4.3.orig.tar.gz

https://gerrit.wikimedia.org/r/403125

Change 403126 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/debs/node-tunnel-agent@master] Merge tag 'upstream/0.4.3'

https://gerrit.wikimedia.org/r/403126

Change 403127 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/debs/node-tunnel-agent@master] Package 0.4.3

https://gerrit.wikimedia.org/r/403127

Change 403124 merged by Hashar:
[operations/debs/node-tunnel-agent@upstream] New upstream version 0.4.3

https://gerrit.wikimedia.org/r/403124

Change 403125 merged by Hashar:
[operations/debs/node-tunnel-agent@pristine-tar] pristine-tar data for node-tunnel-agent_0.4.3.orig.tar.gz

https://gerrit.wikimedia.org/r/403125

Change 403126 merged by Hashar:
[operations/debs/node-tunnel-agent@master] Merge tag 'upstream/0.4.3'

https://gerrit.wikimedia.org/r/403126

Change 403127 merged by Hashar:
[operations/debs/node-tunnel-agent@master] Package 0.4.3

https://gerrit.wikimedia.org/r/403127

I have forked the Debian repository and bumped the package to 0.4.3. The packaging work is in Gerrit repository operations/debs/node-tunnel-agent.git

The result build for Jessie is https://people.wikimedia.org/~hashar/debs/node-tunnel-agent_0.4.3/ which would need to be imported to apt.wikimedia.org probably under jessie-wikimedia/backports

This way npm 1.4.21 will be able to use a proxy and then docker-pkg (which uses a proxy) will be able to use it to upgrade to npm 3.8.x ( T177276 ).

hashar moved this task from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.Jan 9 2018, 11:20 AM

hashar moved this task from Backlog to In progress on the Continuous-Integration-Infrastructure (shipyard) board.Jan 9 2018, 11:22 PM

Change 403362 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] debian-glkue for operations/debs/node-tunnel-agent

https://gerrit.wikimedia.org/r/403362

Change 403362 merged by jenkins-bot:
[integration/config@master] debian-glkue for operations/debs/node-tunnel-agent

https://gerrit.wikimedia.org/r/403362

hashar moved this task from In-progress to Blocked (externally) on the Release-Engineering-Team (Kanban) board.Jan 10 2018, 8:53 AM

Change 403365 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] operations/debs/node-tunnel-agent -> voting debian-glue

https://gerrit.wikimedia.org/r/403365

Change 403365 merged by jenkins-bot:
[integration/config@master] operations/debs/node-tunnel-agent -> voting debian-glue

https://gerrit.wikimedia.org/r/403365

I uploaded the files from https://people.wikimedia.org/~hashar/debs/node-tunnel-agent_0.4.3/ to install1002 /srv/wikimedia/incoming

reprepro -C backports include jessie-wikimedia node-tunnel-agent_0.4.3-1_amd64.changes 
.changes put in a distribution not listed within it!
To ignore use --ignore=wrongdistribution.
There have been errors!

the Distribution line said: "UNRELEASED" i edited that to jessie-wikimedia, then succesfully imported.

reprepro ls node-tunnel-agent
node-tunnel-agent | 0.4.3-1 | jessie-wikimedia | amd64, i386, source

Change 403896 had a related patch set uploaded (by Hashar; owner: Giuseppe Lavagetto):
[integration/config@master] Convert npm, npm-test to docker-pkg (2)

https://gerrit.wikimedia.org/r/403896

Change 403896 merged by jenkins-bot:
[integration/config@master] Convert npm, npm-test to docker-pkg (2)

https://gerrit.wikimedia.org/r/403896

Awesome. That fixed the issue I had locally as well as when running docker-pkg on contint1001. Thank you @Dzahn

hashar mentioned this in T208803: Migrate the Integration cloud project to eqiad1-r.Dec 11 2018, 10:03 AM

npm 1.4.21 can't use a http proxyClosed, ResolvedPublicActions