Page MenuHomePhabricator
Create Task
Maniphest T276477

mediawiki/core eslint:all is slow and does way too many stat() calls
Open, Needs TriagePublic
Actions

Description

On mediawiki/core after a npm ci I ran: ./node_modules/.bin/grunt eslint:all which takes several minutes for apparently no good reasons. Running it under strace -f -etrace=file there are very suspicious stat() calls:

[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/api/i18n/udm.json", {st_mode=S_IFREG|0644, st_size=258, ...}) = 0
[pid 19194] openat(AT_FDCWD, "/home/hashar/projects/mediawiki/core/includes/api/i18n/udm.json", O_RDONLY|O_CLOEXEC) = 20
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/api/i18n/udm.json/babel.config.json", 0x7ffcace38290) = -1 ENOTDIR (Not a directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/api/i18n/udm.json/babel.config.js", 0x7ffcace38290) = -1 ENOTDIR (Not a directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/api/i18n/udm.json/.babelrc", 0x7ffcace38290) = -1 ENOTDIR (Not a directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/api/i18n/udm.json/.babelrc.json", 0x7ffcace38290) = -1 ENOTDIR (Not a directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/api/i18n/udm.json/.babelrc.js", 0x7ffcace38290) = -1 ENOTDIR (Not a directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/api/i18n/udm.json/tsconfig.json", 0x7ffcace38290) = -1 ENOTDIR (Not a directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/api/i18n/babel.config.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/api/i18n/babel.config.js", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/api/i18n/.babelrc", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/api/i18n/.babelrc.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/api/i18n/.babelrc.js", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/api/i18n/tsconfig.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/api/babel.config.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/api/babel.config.js", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/api/.babelrc", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/api/.babelrc.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/api/.babelrc.js", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/api/tsconfig.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/babel.config.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/babel.config.js", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/.babelrc", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/.babelrc.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/.babelrc.js", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/tsconfig.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/babel.config.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/babel.config.js", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/.babelrc", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/.babelrc.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/.babelrc.js", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/tsconfig.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/babel.config.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/babel.config.js", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/.babelrc", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/.babelrc.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/.babelrc.js", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/tsconfig.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/babel.config.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/babel.config.js", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/.babelrc", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/.babelrc.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/.babelrc.js", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/tsconfig.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/babel.config.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/babel.config.js", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/.babelrc", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/.babelrc.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/.babelrc.js", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/tsconfig.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/babel.config.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/babel.config.js", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/.babelrc", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/.babelrc.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/.babelrc.js", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/tsconfig.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/babel.config.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/babel.config.js", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/.babelrc", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/.babelrc.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/.babelrc.js", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/tsconfig.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/api/i18n/udm.json/package.json", 0x7ffcace38290) = -1 ENOTDIR (Not a directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/api/i18n/package.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/api/package.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/package.json", 0x7ffcace38290) = -1 ENOENT (No such file or directory)
[pid 19194] stat("/home/hashar/projects/mediawiki/core/package.json", {st_mode=S_IFREG|0644, st_size=1471, ...}) = 0
[pid 19194] openat(AT_FDCWD, "/home/hashar/projects/mediawiki/core/package.json", O_RDONLY|O_CLOEXEC) = 20
[pid 19194] access("/home/hashar/projects/mediawiki/core/includes/api/i18n/udm.json", F_OK) = 0
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/api/i18n/udm.json", {st_mode=S_IFREG|0644, st_size=258, ...}) = 0
[pid 19194] stat("/home/hashar/projects/mediawiki/core/includes/api/i18n/udm.json", {st_mode=S_IFREG|0644, st_size=258, ...}) = 0

That pils up quickly.

grunt-eslint23.0.0
eslint-config-wikimedia0.18.1
eslint7.19.0

Details

SubjectRepoBranchLines +/-
build: Only run eslint on locally-mastered i18n filesmediawiki/coremaster+4 -0
Customize query in gerrit

Related Objects

Event Timeline

hashar created this task.Mar 4 2021, 4:48 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 4 2021, 4:48 PM
hashar added a comment.Mar 4 2021, 4:52 PM

If I revert 5a622b6a2ec2856d1952da87a86f7d1540a010b5 build: Upgrade eslint-config-wikimedia from 0.17.0 to 0.18.1 the extra stat() calls vanish and I have the following dependencies installed:

grunt-eslint23.0.0
eslint-config-wikimedia0.17.0
eslint7.8.1
hashar added a comment.Mar 4 2021, 5:06 PM

With eslint-config-wikimedia 0.17.0:

real	0m31,086s
user	0m45,998s
sys	0m0,781s

With eslint-config-wikimedia 0.18.1

real	3m4,794s
user	4m10,203s
sys	0m3,999s

So in short went from 31 seconds to 3 minutes :-\

Jdforrester-WMF added a comment.Mar 4 2021, 5:32 PM

Yes. We switched to an eslint plugin for JSON files that actually read them. This significantly increased run time, but we decided it was worth it given it now was actually testing the files. Ed's been in touch upstream about improving performance; mostly we've been customising for core the files it's run on.

Jdforrester-WMF added a comment.Mar 4 2021, 5:35 PM

(Specifically, we switched from eslint-plugin-json to eslint-plugin-json-es.)

Esanders added a comment.Mar 4 2021, 5:36 PM

We fixed some performance issues upstream. My suggestion here was to exclude all i18n files except for en.json & qqq.json, as the rest are generated. That should cut the time significantly.

Running it under strace -f -etrace=file there are very suspicious stat() calls:

What do you think is suspicious about these? Do you not get a similar list with 0.17.0?

gerritbot added a comment.Mar 4 2021, 5:51 PM

Change 668528 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[mediawiki/core@master] build: Only run eslint on locally-mastered i18n files

https://gerrit.wikimedia.org/r/668528

gerritbot added a project: Patch-For-Review.Mar 4 2021, 5:51 PM
gerritbot added a comment.Mar 4 2021, 6:22 PM

Change 668528 merged by jenkins-bot:
[mediawiki/core@master] build: Only run eslint on locally-mastered i18n files

https://gerrit.wikimedia.org/r/668528

ReleaseTaggerBot added a project: MW-1.36-notes (1.36.0-wmf.34; 2021-03-09).Mar 4 2021, 7:00 PM
Jdforrester-WMF added a comment.Mar 4 2021, 7:02 PM

@hashar, is that now fast enough for you?

Maintenance_bot removed a project: Patch-For-Review.Mar 4 2021, 7:10 PM
hashar added a comment.Mar 5 2021, 7:37 AM
In T276477#6883761, @Esanders wrote:

We fixed some performance issues upstream. My suggestion here was to exclude all i18n files except for en.json & qqq.json, as the rest are generated. That should cut the time significantly.

The exclusion rule does make the eslint:all faster:

real	0m58,691s
user	1m20,714s
sys	0m1,026s

Though it is still twice slower than it used to be at least there is some improvement :]

Running it under strace -f -etrace=file there are very suspicious stat() calls:

What do you think is suspicious about these? Do you not get a similar list with 0.17.0?

What is written on the tin!

With 0.17.0, it just open the file it wants to lint:

stat("/home/hashar/projects/mediawiki/core/includes/installer/i18n/cs.json", {st_mode=S_IFREG|0644, st_size=41829, ...}) = 0
openat(AT_FDCWD, "/home/hashar/projects/mediawiki/core/includes/installer/i18n/cs.json", O_RDONLY|O_CLOEXEC) = 20
stat("/home/hashar/projects/mediawiki/core/includes/installer/i18n/cs.json", {st_mode=S_IFREG|0644, st_size=41829, ...}) = 0

With 0.18.1, for each file eslint looks up for six files (babel.config.json, babel.config.js, .babelrc,.babelrc.json,.babelrc.js,tsconfig.json) in the current directory, in every single parent directory down to / and in an imaginary directory named the same as the file (example: includes/api/i18n/udm.json/babel.config.js). Although it is a strong change which can surely be optimized (stop crawling down too far or trying to find the files in a directory named the same as the file), the stat calls seem to only take ~ 50 miliseconds (as measured via strace -f -c -e trace=file. So surely it is a waste, but that does not seem to be the culprit.

I guess some Javascript profling will give more details, but I don't know how to do that with NodeJS :-\

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL