Page MenuHomePhabricator

RESTBase linting API is very slow (not cached)
Closed, ResolvedPublic

Event Timeline

Esanders created this task.May 27 2020, 7:30 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 27 2020, 7:30 PM

This longer page (village pump technical) takes ~3.7s: https://en.wikipedia.org/w/api.php?action=visualeditor&format=json&paction=metadata&page=Wikipedia%3AVillage_pump_(technical)&uselang=en-gb&lint=true&formatversion=2&oldid=959179040
A nearly empty page took 400ms.

That would suggest that the time is spent doing the linting, and the result is not being cached.

CC @cscott

I get similar timings talking to RESTBase directly: https://en.wikipedia.org/api/rest_v1/page/lint/Wikipedia%3AVillage_pump_(technical)/959179040

So this definitely looks like a caching issue on the RESTBase side.

Esanders renamed this task from Fetching linting data through API is very slow to RESTBase linting API is very slow (not cached).May 27 2020, 7:40 PM

@ssastry please correct me if I'm wrong, but Linter extension stores Parsoid lints in MySQL. If I understand it well, the flow is the following: page edit -> parsoid rerender -> parsoid calling MW API to record lint errors -> API is scheduling a job -> errors are stored in the database. So, VE doesn't need to call RESTBase to re-render the lint errors, it can just execute an action API query if working from the client side, or depend on Linter extension and get the errors from linter table directly.

Quick test on beta reveals the results are equivalent:

curl 'https://en.wikipedia.beta.wmflabs.org/w/api.php?action=query&list=linterrors&lntpageid=1&format=json'

{"batchcomplete":"","query":{"linterrors":[{"pageid":1,"ns":0,"title":"Main Page","lintId":16114,"category":"bogus-image-options","location":[3862,3877],"templateInfo":{"name":"Template:In_the_news"},"params":{"items":[""]}}]}}

curl -XPOST -H 'Content-Type: application/json' -H 'Host: en.wikipedia.beta.wmflabs.org' --data '{}' http://parsoid-external-ci-access.beta.wmflabs.org/w/rest.php/en.wikipedia.beta.wmflabs.org/v3/transform/wikitext/to/lint/Main_Page/416077

[{"type":"bogus-image-options","dsr":[3862,3877,null,null],"templateInfo":{"name":"Template:In_the_news"},"params":{"items":[""]}}]

In general, we're trying to replace RESTBase caching with other means of caching for Parsoid since it's moving to core, so I would like to avoid adding more caching into RESTBase

Change 599418 had a related patch set uploaded (by Esanders; owner: Esanders):
[mediawiki/extensions/VisualEditor@master] Fetch lint errors from MW API, not RESTBase

https://gerrit.wikimedia.org/r/599418

Change 599420 had a related patch set uploaded (by Esanders; owner: Esanders):
[mediawiki/extensions/DiscussionTools@master] Use new lint API

https://gerrit.wikimedia.org/r/599420

Change 602182 had a related patch set uploaded (by Esanders; owner: Esanders):
[mediawiki/extensions/DiscussionTools@master] Call the linter API directly if installed

https://gerrit.wikimedia.org/r/602182

Change 599420 abandoned by Bartosz Dziewoński:
Use new lint API

Reason:
Doing https://gerrit.wikimedia.org/r/602182 instead

https://gerrit.wikimedia.org/r/599420

Esanders claimed this task.Jun 4 2020, 2:50 PM

Change 602182 merged by jenkins-bot:
[mediawiki/extensions/DiscussionTools@master] Call the linter API directly if installed

https://gerrit.wikimedia.org/r/602182

Change 599418 merged by jenkins-bot:
[mediawiki/extensions/VisualEditor@master] Drop linting from the VE API

https://gerrit.wikimedia.org/r/599418

ppelberg closed this task as Resolved.Jun 5 2020, 1:14 AM
Restricted Application added a project: User-Ryasmeen. · View Herald TranscriptJun 5 2020, 1:14 AM