Page MenuHomePhabricator
Create Task
Maniphest T215562

npm 6 consistently fails with "Z_DATA_ERROR: invalid distance too far back" on some repos
Closed, ResolvedPublic
Actions

Description

This is not currently causing build failures because things are working fine on npm 3 (node 6), and the projects already switched to npm 6 (node 10) are only those not affected by this bug.

But, in trying out node10 on a number of projects, some of them consistently fail in this way.

Repos on which Node 10 / npm 6 fails (but Node 6 / npm 3 passes)

  • node-rdkafka-statsd (change; build)
  • labs/tools/heritage (change; build)
  • pywikibot/i18n (change)
  • wikibase/javascript-api (change)
  • performance/fresnel (when a node_modules dir exists).
  • wikipeg (change, build)
  • mediawiki/extensions/Flow
  • mediawiki/extensions/TemplateData
  • mediawiki/extensions/Wikibase
+ rm -rf node_modules
+ npm install --no-progress

...

npm ERR! code Z_DATA_ERROR
npm ERR! errno -3
npm ERR! invalid distance too far back

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Follow-up 8c5d25631: Revert Timo's inclusion of changes to VEintegration/configmaster+7 -1
Revert Flow js documentation back from node10 to jsduckintegration/configmaster+3 -1
jjb: update npm jobs to use node10:0.5.0 docker imageintegration/configmaster+8 -8
dockerfiles: Create node10:0.5.0 imagesintegration/configmaster+21 -0
Revert use of npm-run-doc on node10 for TemplateData and Wikibaseintegration/configmaster+4 -2
jjb: update node10 jobs to docker image 0.4.0integration/configmaster+8 -8
dockerfiles: Update node10 images with npm 6.5.0 tarballintegration/configmaster+21 -0
wmf: Rewrite update.sh based on npmjs.org tarballintegration/npmmaster+28 -13
build: Un-commit package-lockmediawiki/extensions/VisualEditormaster+1 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolveddancyT302086 Set scap minimum python version to 3.7
 ResolvedNoneT247045 Migrate all of production metal and VMs to Buster or later
 ResolvedakosiarisT249724 Track and remove jessie based container images from production
 ResolvedJdforrester-WMFT224908 Drop jessie testing support
 ResolvedJdforrester-WMFT224907 Drop php55 testing support
 ResolvedReedyT205039 Release MediaWiki 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedReedyT205041 Tracking bug for 1.27.6/1.30.2/1.31.2/1.32.2 security release
 ResolvedBawolffT197279 Direct POST to Special:ChangeEmail will bypass reauth check
 ResolvedAnomieT204729 Potential enwiki DOS due to slow WatchedItemStore::countVisitingWatchersMultiple
 ResolvedBawolffT207603 CVE-2019-12471: Loading JS from user space where the username is not a registered account is dangerous and should be banned
 ResolvedBawolffT208881 CSS using var() to create exponential sized calc() on wiki page will crash visitor's browser
 ResolvedLegoktmT199540 Forbid blocking IP ranges as big as /1 and /2, as done on ruwikiquote using the API
 ResolvedLucas_Werkmeister_WMDET212118 API responses for unpatrolled or (not) autopatrolled recent changes require privileges but may be cached publicly
 ResolvedBawolffT209794 Need to make a limit of count of attempts to change email address
 ResolvedsbassettT222036 Exposed suppressed username or log in Special:EditTags
 ResolvedsbassettT222038 Exposed suppressed log in RevisionDelete page
 ResolvedJdforrester-WMFT221739 Patch jQuery due to CVE-2019-11358
 ResolvedsbassettT25227 Use token when logging out
 ResolvedsbassettT221868 Send out wikitech-l post for T25227 ("Use token when logging out")
 ResolvedReedyT205042 Write and send release announcements for 1.27.6/1.30.2/1.31.2/1.32.2 security releases
 ResolvedReedyT205043 Write and send pre-release Announcements for MediaWiki 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedJdforrester-WMFT205044 Update onwiki release notes for 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedReedyT205046 Update HISTORY in master after 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedReedyT205047 Tag MW 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedReedyT224499 RELEASE-NOTES for 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedJdforrester-WMFT224912 Update MediaWiki.org links and versions
 ResolvedJdforrester-WMFT224913 EOL REL1_27 and REL1_30 onwiki
 ResolvedReedyT225201 Formally announce EOL of MW 1.27 and 1.30
 ResolvedReedyT225149 Update CVEs and publish them
 ResolvedReedyT205048 Obtain CVEs for 1.27.6/1.30.2/1.31.2/1.32.2 security releases
 ResolvedJdforrester-WMFT224906 Drop php56 testing support
 ResolvedhasharT223348 Run wikimedia/fundraising/crm CI jobs on PHP7x, not PHP5x
 ResolvedhasharT210287 Migrate wikimedia-fundraising-civicrm to a Docker container
 ResolvedJdforrester-WMFT224905 Move wikiba.se tests from php56 to php72
 ResolvedhasharT224591 Migrate contint* hosts to Buster
 DeclinedMoritzMuehlenhoffT226236 Upload docker-ce 18.06.3 upstream package for Stretch
 InvalidthciprianiT239880 Replacement hardware for buster/stretch upgrade of contint1001 and contint2001
 Resolved• mmodellT215458 Convert zuul to use scap
 ResolvedhasharT240551 Remove Zuul Debian package from WMCS instances
 ResolvedJoeT249110 Build and publish a python2 based container to build wheels
 ResolvedJMeybohmT249812 Rebuild helm/helm-diff for buster-wikimedia
 ResolvedJMeybohmT250479 Rebuild helmfile for buster-wikimedia
 DuplicateDzahnT210008 upgrade krypton (webserver_misc_apps) to stretch
 DeclinedDzahnT224194 switch webserver_misc_apps to PHP 7.2 (7.1)
 ResolvedJdforrester-WMFT211784 Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects
 ResolvedKrinkleT215562 npm 6 consistently fails with "Z_DATA_ERROR: invalid distance too far back" on some repos

Event Timeline

Krinkle created this task.Feb 7 2019, 10:22 PM
Krinkle triaged this task as Medium priority.
Krinkle renamed this task from Fix "Z_DATA_ERROR: invalid distance too far back to npm 6 consistently fails with "Z_DATA_ERROR: invalid distance too far back" on some repos.Feb 7 2019, 10:26 PM
Krinkle updated the task description. (Show Details)
Krinkle added a comment.Feb 21 2019, 4:58 PM

Reported upstream at https://npm.community/t/zlib-invalid-distance-too-far-back/5577.

Krinkle updated the task description. (Show Details)Feb 23 2019, 2:01 AM
Krinkle updated the task description. (Show Details)Feb 23 2019, 2:33 AM
Krinkle updated the task description. (Show Details)Feb 27 2019, 2:19 PM
Krinkle updated the task description. (Show Details)Mar 11 2019, 2:18 AM
Jdforrester-WMF subscribed.Mar 21 2019, 11:41 PM
gerritbot added a comment.Mar 21 2019, 11:48 PM

Change 498267 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[mediawiki/extensions/VisualEditor@master] build: Un-commit package-lock

https://gerrit.wikimedia.org/r/498267

gerritbot added a project: Patch-For-Review.Mar 21 2019, 11:48 PM
gerritbot added a comment.Mar 22 2019, 12:22 AM

Change 498276 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] Follow-up 8c5d25631: Revert Timo's inclusion of changes to VE

https://gerrit.wikimedia.org/r/498276

gerritbot added a comment.Mar 22 2019, 12:31 AM

Change 498276 merged by jenkins-bot:
[integration/config@master] Follow-up 8c5d25631: Revert Timo's inclusion of changes to VE

https://gerrit.wikimedia.org/r/498276

Stashbot added a comment.Mar 22 2019, 12:36 AM

Mentioned in SAL (#wikimedia-releng) [2019-03-22T00:36:19Z] <Krinkle> Reloading Zuul to deploy https://gerrit.wikimedia.org/r/498276 / T215562)

gerritbot added a comment.Mar 22 2019, 12:41 AM

Change 498267 abandoned by Jforrester:
build: Un-commit package-lock

Reason:
Testing artefact, no longer needed.

https://gerrit.wikimedia.org/r/498267

Jdforrester-WMF edited projects, added Upstream; removed Patch-For-Review.EditedMar 22 2019, 12:56 AM

Reported upstream by Timo.

MarcoAurelio subscribed.Mar 22 2019, 9:35 AM

I guess this is the same as T218978 but in that case it is causing builds to fail, thus I filed a different task. Feel free to merge, though, if you think it is best.

gerritbot added a comment.Mar 23 2019, 4:06 AM

Change 498524 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/npm@master] wmf: Rewrite update.sh based on npmjs.org tarball

https://gerrit.wikimedia.org/r/498524

gerritbot added a project: Patch-For-Review.Mar 23 2019, 4:06 AM
gerritbot added a comment.Mar 23 2019, 6:41 AM

Change 498524 merged by Legoktm:
[integration/npm@master] wmf: Rewrite update.sh based on npmjs.org tarball

https://gerrit.wikimedia.org/r/498524

gerritbot added a comment.Mar 23 2019, 7:58 PM

Change 498664 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] dockerfiles: Update node10 images with npm 6.5.0 tarball

https://gerrit.wikimedia.org/r/498664

gerritbot added a comment.Mar 23 2019, 8:45 PM

Change 498664 merged by jenkins-bot:
[integration/config@master] dockerfiles: Update node10 images with npm 6.5.0 tarball

https://gerrit.wikimedia.org/r/498664

Stashbot added a comment.Mar 23 2019, 8:52 PM

Mentioned in SAL (#wikimedia-releng) [2019-03-23T20:52:40Z] <Krinkle> Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/498664 / T215562

gerritbot added a comment.Mar 23 2019, 8:54 PM

Change 498667 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] jjb: update node10 jobs to docker image 0.4.0

https://gerrit.wikimedia.org/r/498667

gerritbot added a comment.Mar 23 2019, 9:36 PM

Change 498667 merged by jenkins-bot:
[integration/config@master] jjb: update node10 jobs to docker image 0.4.0

https://gerrit.wikimedia.org/r/498667

Krinkle added a comment.Mar 23 2019, 9:39 PM
Status: Downloaded newer image for docker-registry.wikimedia.org/releng/node10-test:0.4.0
..
+ node --version
v10.4.0
+ npm --version
6.5.0
..
+ rm -rf node_modules
+ npm install --no-progress
..
npm ERR! code Z_DATA_ERROR
npm ERR! errno -3
npm ERR! invalid distance too far back
..
npm ERR! Callback called more than once.

Same as before. At least we've confirmed that the issue isn't related to how we install npm.

gerritbot added a comment.Mar 26 2019, 5:25 PM

Change 499262 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] Revert Flow js documentation back from node10 to jsduck

https://gerrit.wikimedia.org/r/499262

gerritbot added a comment.Mar 26 2019, 5:40 PM

Change 499262 merged by jenkins-bot:
[integration/config@master] Revert Flow js documentation back from node10 to jsduck

https://gerrit.wikimedia.org/r/499262

Krinkle updated the task description. (Show Details)Mar 26 2019, 7:54 PM
Krinkle updated the task description. (Show Details)
gerritbot added a comment.Mar 26 2019, 7:57 PM

Change 499312 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] Revert use of npm-run-doc on node10 for TemplateData and Wikibase

https://gerrit.wikimedia.org/r/499312

gerritbot added a comment.Mar 26 2019, 7:59 PM

Change 499312 merged by jenkins-bot:
[integration/config@master] Revert use of npm-run-doc on node10 for TemplateData and Wikibase

https://gerrit.wikimedia.org/r/499312

zeljkofilipin mentioned this in T210506: `npm install` fails for mediawiki/core with EPEERINVALID when running on Node 11.Mar 28 2019, 3:13 PM
zeljkofilipin subscribed.
zeljkofilipin awarded a token.Mar 28 2019, 3:17 PM
zeljkofilipin added a project: User-zeljkofilipin.
zeljkofilipin moved this task from Backlog 🪒 to Watching 📺 on the User-zeljkofilipin board.
Krinkle assigned this task to MoritzMuehlenhoff.Mar 28 2019, 5:01 PM
Krinkle added a project: SRE.
Krinkle removed a project: Upstream.EditedMar 28 2019, 5:15 PM

It seems we've found the culprit. The problem is indeed the zlib1g library. This was apparent from the error, and also what the support forum pointed to.

However, what wasn't clear until today is how it is a problem.

  • Has Node 10 regressed since Node 6 in its support for zlib1g 1.2.8 and (by extent) Debian Linux?

This was my main question, given that Debian 8 and 9 have the same version of zlib1g, and Node 6 on Debian 8 worked fine for us, whereas Node 10 on Debian 9 (with the same zlib1g version) is broken for us.

It is likely true that Node.js's C++ code has changed between Node 6 and Node 10 in a way that no longer works with zlib1g 1.2.8.

@MoritzMuehlenhoff recalled that Node.js C++ source code contains a local copy of zlib1g that it is always compatible.

  • In Node 4 and Node 6, this was zlib 1.2.8.
  • As of Node 7 (and still in Node 8 and Node 10) this is zlib 1.2.11.

The default Makefile for Node.js, and the distribution by Nodesource compiles against this local copy, which is what Node.js supports.

The distribution by Debian, however, patched Node.js to link against the version of zlib1g that the current OS distribution prefers. With Node 6 that happened to match the version of Debian 8 and Debian 9. With Node 10 this is no longer the case.

As such, it is effectively our fault for packaging it this way. We need to either:

  1. make upstream Node.js support both (which they don't currently);
  2. or, backport a newer version of zlib1g to (our distribution of) Debian 9;
  3. or, re-compile our Node.js to use the local version.

Moritz says the easiest of these would be to re-compile our Node.js 10 package against the local copy of zlib1g from Node's source code, instead of dynamically linking the system-wide one from Debian.

zeljkofilipin added a comment.Mar 28 2019, 6:37 PM

Node 6 end of life is next month (2019-04-30).

MoritzMuehlenhoff added a comment.Mar 29 2019, 8:28 AM
In T215562#5066711, @Krinkle wrote:

As such, it is effectively our fault for packaging it this way. We need to either:

Not all all :-) This is entirely the fault of the node ecosystem, who designed their system of deploying binary modules after what they do for Javascript. Distributing binaries reliably is a very hard problem, people have been working in this since for decades, they should rather have investigated the lessons learned by others...

Moritz says the easiest of these would be to re-compile our Node.js 10 package against the local copy of zlib1g from Node's source code, instead of dynamically linking the system-wide one from Debian.

Ack, I'll take care of that.

MoritzMuehlenhoff added a comment.Apr 1 2019, 1:20 PM

@Krinkle I've prepared a new build and uploaded it to https://people.wikimedia.org/~jmm/node/

Could you verify that it fixes the Z_DATA_ERROR issue, if so I'll upload it to the node10 component on apt.wikimedia.org

Krinkle claimed this task.Apr 1 2019, 2:57 PM
Krinkle added a subscriber: MoritzMuehlenhoff.
bd808 subscribed.Apr 1 2019, 4:39 PM
xSavitar subscribed.Apr 1 2019, 10:36 PM
Krinkle added a comment.Apr 1 2019, 10:53 PM
In T215562#5074013, @MoritzMuehlenhoff wrote:

@Krinkle I've prepared a new build and uploaded it to https://people.wikimedia.org/~jmm/node/

Thanks! I've used dpkg -i sometimes to install a .deb file, but I may need some help with this directory.

I'm looking to install it instead of the apt-get command at node10/Dockerfile. I would then build it, and the node10-test child image locally and then start the container locally to see if the issue can still be reproduced.

Krinkle reassigned this task from Krinkle to MoritzMuehlenhoff.Apr 3 2019, 4:39 PM

OK. Looks like the image will already be tested as part of another service deployment. Assigning back to Moritz to notify once it's up on apt-wikimedia so that I can rebuild the relevant CI images after that.

Stashbot added a comment.Apr 4 2019, 11:35 AM

Mentioned in SAL (#wikimedia-operations) [2019-04-04T11:35:58Z] <moritzm> uploaded nodejs 10.15.2~dfsg-1+wmf1 to the component/node10 component of apt.wikimedia.org/stretch-wikimedia (updated to latest 10.x release and a change to ensure zlib binary compat with NodeSource) (T215562)

MoritzMuehlenhoff reassigned this task from MoritzMuehlenhoff to Krinkle.Apr 4 2019, 11:41 AM
In T215562#5082327, @Krinkle wrote:

OK. Looks like the image will already be tested as part of another service deployment. Assigning back to Moritz to notify once it's up on apt-wikimedia so that I can rebuild the relevant CI images after that.

I've updated the component/node10 component now, reassigning back for further tests/rebuild of CI images.

gerritbot added a comment.Apr 4 2019, 8:44 PM

Change 501407 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] dockerfiles: Create node10:0.5.0 images

https://gerrit.wikimedia.org/r/501407

gerritbot added a comment.Apr 4 2019, 8:46 PM

Change 501408 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] jjb: update npm jobs to use node10:0.5.0 docker image

https://gerrit.wikimedia.org/r/501408

gerritbot added a comment.Apr 4 2019, 8:46 PM

Change 501407 merged by jenkins-bot:
[integration/config@master] dockerfiles: Create node10:0.5.0 images

https://gerrit.wikimedia.org/r/501407

Krinkle updated the task description. (Show Details)Apr 4 2019, 8:48 PM
gerritbot added a comment.Apr 4 2019, 8:57 PM

Change 501408 merged by jenkins-bot:
[integration/config@master] jjb: update npm jobs to use node10:0.5.0 docker image

https://gerrit.wikimedia.org/r/501408

Krinkle closed this task as Resolved.Apr 4 2019, 9:32 PM
Jdforrester-WMF awarded a token.Apr 4 2019, 9:41 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits