Page MenuHomePhabricator

Temporarily disable wikitext linting updates to help resolve some cluster overload scenarios
Open, HighPublic

Description

The API cluster is going through some load issues and SRE is trying to shed unnecessary traffic.

As it turns out, while we've enabled Parsoid/PHP everywhere, we still have ongoing traffic mirroring to Parsoid/JS for the reparse requests for two reasons. (a) in the offchance we had to fall back to Parsoid/JS (b) linting updates are still happening via Parsoid/JS.

At this time, (a) is highly unlikely to transpire. Parsoid/PHP looks stable and there have been no reports of any serious bugs. So, that leaves us with the linter. While Parsoid/PHP has code to turn on those linter updates, given this week's mediawiki train instability and incident, the consensus is that we don't turn on any new things unless absolutely necessary.

Given that wikitext linting is not a heavily used feature and disabling it temporarily isn't going to be disruptive, we are taking the decision to temporarily turn off updates. What does this mean? All the existing lints are accessible and you can query the API, etc. But when pages are edited (that introduce new lints or to fix existing lints), those changes will not be reflected in the linter database while we have this disabled.

Details

Related Gerrit Patches:
operations/mediawiki-config : masterMake Parsoid/PHP cluster read-write to record lints

Event Timeline

ssastry created this task.Dec 6 2019, 10:44 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 6 2019, 10:44 PM
ssastry triaged this task as High priority.Dec 6 2019, 11:22 PM
ssastry edited projects, added Parsoid, MediaWiki-extensions-Linter; removed Security.
Restricted Application added a project: Security. · View Herald TranscriptDec 6 2019, 11:22 PM
ssastry edited projects, added User-notice; removed Security.Dec 6 2019, 11:25 PM
ssastry added a subscriber: Daimona.

About 20 minutes back, there was a RESTBase deploy that stopped parses from Parsoid/JS and so this temporary disabling of linter updates is now in place. We hope to re-enable linting updates next week.

@Whatamidoing-WMF @Izno @Daimona @Jonesey95 FYI. I'll send out some emails and post on a few village pumps as well.

ssastry changed the visibility from "Custom Policy" to "Public (No Login Required)".Dec 7 2019, 1:11 PM

When wikitext linting updates are re-enabled, what will happen to everything that was edited while wikitext linting updates was disabled? How long will it take to get caught up?

When wikitext linting updates are re-enabled, what will happen to everything that was edited while wikitext linting updates was disabled? How long will it take to get caught up?

We will have to figure out the specifics how to handle that, but we will update this ticket when we iron out those details.

Change 556001 had a related patch set uploaded (by Arlolra; owner: Arlolra):
[operations/mediawiki-config@master] Make Parsoid/PHP cluster read-write to record lints

https://gerrit.wikimedia.org/r/556001

Johan added a subscriber: Johan.Dec 12 2019, 7:09 AM

Something like this for Tech News going out on Monday?

The [[<tvar|linter>mw:Extension:Linter</>|Linter extension]] helps you find technical errors in articles. It has not showed new changes for more than a week. This is because of an [[:w:en:Application programming interface|API]] problem. It will soon work again. [https://phabricator.wikimedia.org/T240057]

Or will it already be re-enabled then?

In that case:

The [[<tvar|linter>mw:Extension:Linter</>|Linter extension]] helps you find technical errors in articles. It did not show new changes last week. This was because of an [[:w:en:Application programming interface|API]] problem. It is now working again. [https://phabricator.wikimedia.org/T240057]

Something like this for Tech News going out on Monday?
...
Or will it already be re-enabled then?

Yes, it very likely will be.

In that case:
The [[<tvar|linter>mw:Extension:Linter</>|Linter extension]] helps you find technical errors in articles. It did not show new changes last week. This was because of an [[:w:en:Application programming interface|API]] problem. It is now working again. [https://phabricator.wikimedia.org/T240057]

Sounds good.

Change 556001 merged by jenkins-bot:
[operations/mediawiki-config@master] Make Parsoid/PHP cluster read-write to record lints

https://gerrit.wikimedia.org/r/556001

With T237326#5737277, linting is now re-enabled.

When wikitext linting updates are re-enabled, what will happen to everything that was edited while wikitext linting updates was disabled? How long will it take to get caught up?

To elaborate on @ssastry's response a bit -- what we expect we will do is start a low-priority background job that will pull every page in every wikimedia project through the linter to ensure that all missing lints are regenerated. Last time we did this it took about a week for that job to run. We'll update here w/ more details once that gets started.

Purging or editing a page will also regenerate the lints for that particular page. (But not null-editing a page -- as I understand it there's an optimization somewhere which skips the re-parse if the new revision of a page is identical to the old one.)

@cscott wrote "what we expect we will do is start a low-priority background job that will pull every page in every wikimedia project through the linter to ensure that all missing lints are regenerated. Last time we did this it took about a week for that job to run."
Well, if you're going to do that, why not wipe the linter database and start over, to clear out the nonexistent 1 Paragraph wrapping bug workaround, the nonexistent 66 Self-closed tags, the nonexistent 77 Unclosed quote in heading, and the nonexistent 14 Multi colon escape, and other miscounted lint errors.

That may be a good idea.

If we are going to clear the database, we should wait till Jan 2020 to embark on this. Let us not go into the holiday break with major changes like that.

Is this problem related to the "error -- Error: Unsupported Media Type" message that I receive on every page in en.WP when I have LintHint enabled? LintHint works if I edit the page, but not in viewing mode.

Also, the down arrow that jumps to the error and highlights the erroneous block of text started highlighting text below the actual error. Is this related to the parsing change? Maybe a separate bug?