Page MenuHomePhabricator

wikimedia/portals repo might be using outdated and or deprecated tests in jenkins
Closed, ResolvedPublic

Description

Repository: rWPOR Wikimedia Portals

I recently submitted a change for review and had the curiosity to look at the console output for the continuous integration job. It throws warnings such as:

18:33:05 [wikimedia-portals-npm-node-4-jessie] $ /bin/bash -xe /tmp/hudson6752207644662258065.sh
18:33:05 + mkdir -p /home/jenkins/workspace/wikimedia-portals-npm-node-4-jessie/log
18:33:05 + node --version
18:33:05 v4.6.0
18:33:05 + npm --version
18:33:06 2.15.2
18:33:06 + npm install
18:33:08 npm WARN deprecated cssnext@1.8.4: cssnext is now postcss-cssnext. cssnext is deprecated. See postcss-cssnext migration guide http://cssnext.io/postcss/
18:33:08 npm WARN deprecated gulp-minify-css@1.2.4: Please use gulp-clean-css
18:33:11 npm WARN peerDependencies The peer dependency stylelint@^7.0.2 included from stylelint-config-wikimedia will no
18:33:11 npm WARN peerDependencies longer be automatically installed to fulfill the peerDependency 
18:33:11 npm WARN peerDependencies in npm 3+. Your application will need to depend on it explicitly.
18:33:45 npm WARN deprecated node-uuid@1.4.7: use uuid module instead
18:33:53 npm WARN deprecated minimatch@2.0.10: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
18:35:32 npm WARN deprecated minimatch@0.2.14: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
18:35:39 npm WARN deprecated lodash@1.0.2: lodash@<3.0.0 is no longer maintained. Upgrade to lodash@^4.0.0.
18:36:49 npm WARN deprecated graceful-fs@1.2.3: graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.
18:37:37 npm WARN optional dep failed, continuing fsevents@1.0.15
18:37:45 npm WARN deprecated css-list@0.1.3: Deprecated.
18:38:08 npm WARN engine img-stats@0.4.2: wanted: {"node":"~0.10"} (current: {"node":"4.6.0","npm":"2.15.2"})

asking to update deprecated stuff. Maybe it should be updated so it does run with the latests code.

On the other hand, a jenkins job on each patch submitted takes 11 minutes in average to be fully tested. I don't think that's reasonable and maybe ways to reduce time of such tests can be attempted.

Thank you.

Event Timeline

Restricted Application added a project: Discovery. · View Herald TranscriptDec 4 2016, 6:44 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
MarcoAurelio renamed this task from Repo might be using outdated and or deprecated tests in jenkins to wikimedia/portals repo might be using outdated and or deprecated tests in jenkins.Dec 4 2016, 6:45 PM
MarcoAurelio updated the task description. (Show Details)
MarcoAurelio added subscribers: mxn, JGirault.

@Jdrewniak Does T150190 addresses the whole lot of the issues reported above or can this be a separate task?

hashar added a subscriber: hashar.EditedDec 4 2016, 10:19 PM

Regarding the long build time (11 minutes), that is due to the job compiling a lot of material. The cause is we only cache the tarballs downloaded from npmjs.org, I guess we should also cache node_modules and expire old modules by running npm prune. I think we tried at some point and that had side effects due to npm/node version changing. I guess we can try again. Some history from two years ago on T76304#795614

Worth forking to its own task :]

MarcoAurelio triaged this task as High priority.Dec 5 2016, 11:58 AM

@hashar: Fork done at T152386

After merging some code, it is now:

17:21:46 [wikimedia-portals-npm-node-4-jessie] $ /bin/bash -xe /tmp/hudson3322079734029663977.sh
17:21:46 + mkdir -p /home/jenkins/workspace/wikimedia-portals-npm-node-4-jessie/log
17:21:46 + node --version
17:21:46 v4.6.0
17:21:46 + npm --version
17:21:47 2.15.2
17:21:47 + npm install
17:21:54 npm WARN peerDependencies The peer dependency stylelint@^7.0.2 included from stylelint-config-wikimedia will no
17:21:54 npm WARN peerDependencies longer be automatically installed to fulfill the peerDependency 
17:21:54 npm WARN peerDependencies in npm 3+. Your application will need to depend on it explicitly.
17:22:35 npm WARN deprecated node-uuid@1.4.7: use uuid module instead
17:22:42 npm WARN deprecated minimatch@2.0.10: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
17:24:45 npm WARN deprecated minimatch@0.2.14: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
17:24:55 npm WARN deprecated lodash@1.0.2: lodash@<3.0.0 is no longer maintained. Upgrade to lodash@^4.0.0.
17:26:36 npm WARN deprecated graceful-fs@1.2.3: graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.

I did git clone wikimedia/portals to my machine and something weird happens. When running npm install it says:

$ npm install
npm WARN deprecated minimatch@2.0.10: Please update to minimatch 3.0.2 or higher                                                                                                                                   to avoid a RegExp DoS issue
npm WARN deprecated minimatch@0.2.14: Please update to minimatch 3.0.2 or higher                                                                                                                                   to avoid a RegExp DoS issue
npm WARN deprecated graceful-fs@1.2.3: graceful-fs v3.0.0 and before will fail o                                                                                                                                  n node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible                                                                                                                                  . Use 'npm ls graceful-fs' to find it in the tree.
npm WARN deprecated node-uuid@1.4.7: use uuid module instead
npm WARN deprecated lodash-node@2.4.1: This package is discontinued. Use lodash@^4.0.0.
npm WARN prefer global jsonlint@1.6.2 should be installed with -g
npm WARN prefer global colorguard@1.2.0 should be installed with -g
[...]
F:\MWDEV\portals\node_modules\lwip>if not defined npm_config_node_gyp (node "C:\                                                                                                                                  Users\MA\AppData\Roaming\npm\node_modules\npm\bin\node-gyp-bin\\..\                                                                                                                                  ..\node_modules\node-gyp\bin\node-gyp.js" rebuild )  else (node "" rebuild )
gyp ERR! configure error
gyp ERR! stack Error: Python executable "C:\Users\MA\AppData\Local\                                                                                                                                  Programs\Python\Python35-32\python.EXE" is v3.5.2, which is not supported by gyp                                                                                                                                  .
gyp ERR! stack You can pass the --python switch to point to Python >= v2.5.0 & <                                                                                                                                   3.0.0.
gyp ERR! stack     at failPythonVersion (C:\Users\MA\AppData\Roamin                                                                                                                                  g\npm\node_modules\npm\node_modules\node-gyp\lib\configure.js:454:14)
gyp ERR! stack     at C:\Users\MA\AppData\Roaming\npm\node_modules\                                                                                                                                  npm\node_modules\node-gyp\lib\configure.js:443:9
gyp ERR! stack     at ChildProcess.exithandler (child_process.js:197:7)
gyp ERR! stack     at emitTwo (events.js:106:13)
gyp ERR! stack     at ChildProcess.emit (events.js:191:7)
gyp ERR! stack     at maybeClose (internal/child_process.js:877:16)
gyp ERR! stack     at Socket.<anonymous> (internal/child_process.js:334:11)
gyp ERR! stack     at emitOne (events.js:96:13)
gyp ERR! stack     at Socket.emit (events.js:188:7)
gyp ERR! stack     at Pipe._handle.close [as _onclose] (net.js:498:12)
gyp ERR! System Windows_NT 10.0.14393
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\MA
\\AppData\\Roaming\\npm\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-                                                                                                                                  gyp.js" "rebuild"
gyp ERR! cwd F:\MWDEV\portals\node_modules\lwip
gyp ERR! node -v v6.9.1
gyp ERR! node-gyp -v v3.4.0
gyp ERR! not ok

Also, running any gulp command results in:

$ gulp update-stats
bash: gulp: command not found

@mxn and @Jdrewniak Any idea on what's going on?

hashar added a comment.Dec 5 2016, 6:09 PM
Python executable "C:\Users\MA\AppData\Local\                                                                                                                                  Programs\Python\Python35-32\python.EXE" is v3.5.2, which is not supported by gyp                                                                                                                                  .
gyp ERR! stack You can pass the --python switch to point to Python >= v2.5.0 & <                                                                                                                                   3.0.0.

Looks like you need a python 2.7 installation? If you already have one, maybe gyp accepts an environment variable to indicate the python path

@MarcoAurelio this looks like an error compiling lwip - the node image processor & optimizer we use to optimize the images (this thing has been a pain with Node 6 - and it cause CI to run for 15 min :/ ). The error your getting seems to be around node-gyp and the version of Python. Node-gyp documentation says to install it globally:
https://github.com/nodejs/node-gyp#installation
so first npm install --global --production windows-build-tools
and then you need to install Python 2.7 on your system, it looks like you have version 3.5.2 (I don't know much about Python version management, so I hope managing multiple python versions on one system isn't a nightmare).
After that, npm install -g node-gyp should work, as well as the local npm install

I've tried to get that done but then my bot stopped working because it doesn't know on which version to run and some scripts I use are for Python 3 so it is not worth for me to sacrifice those tools. Besides, Python 2 is legacy software and is being deprecated so maybe the tools we use in our repo should also be updated. I'll simply give up for now as I feel it is not worth the effort for me. I sincerely appreciate the help offered though.

Notwithstanding my comment above, we should focus on updating all instances where possible and I've opened a separate task to avoid CI tasks running for 15 minutes.

Regards.

hashar removed a subscriber: hashar.

Does not seem to be related to our CI.

Restricted Application added a project: User-MarcoAurelio. · View Herald TranscriptAug 1 2017, 7:53 PM

It's been some time since I worked on this repo so I'm not sure if this is still a problem. Maybe those from the Discovery team could check? Thanks!

debt added a subscriber: debt.Aug 1 2017, 11:09 PM

Thanks for the ping, @Jdrewniak will take a look next week when he's back from his vacation.

Change 384563 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[wikimedia/portals@master] Replacing portal image workflow with SVGs

https://gerrit.wikimedia.org/r/384563

debt moved this task from In Progress to Needs code review on the Discovery-Portal-Sprint board.

Change 387194 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[wikimedia/portals@master] SVG Sprites - Removing old gulp tasks & npm deps

https://gerrit.wikimedia.org/r/387194

Change 384563 abandoned by Jdrewniak:
Replacing portal image workflow with SVGs

Reason:
As per @Volker's suggestion. I've split this patch into multiple commits on the following topic thread

https://gerrit.wikimedia.org/r/#/q/topic:new-svg-sprites (status:open OR status:merged)

https://gerrit.wikimedia.org/r/384563

Change 387194 merged by jenkins-bot:
[wikimedia/portals@master] SVG Sprites - Remove old gulp tasks & npm deps

https://gerrit.wikimedia.org/r/387194

debt closed this task as Resolved.Nov 21 2017, 6:26 PM
debt claimed this task.