Page MenuHomePhabricator

Enable transform/wikitext/to/lint endpoint in production
Closed, ResolvedPublic3 Estimated Story Points

Description

Description

To unblock the RESTbase sunsetting work, we need to begin rerouting the lint endpoints. This capability seems to have already been implemented in Core, but was never enabled in production. The scope of this work is to enable that endpoint and perform testing to ensure we have feature parity with the existing public endpoint exposed through RESTbase, so that we may begin rerouting.

Conditions of acceptance

  • Add /wikitext/to/lint endpoints to coreRoutes.json to enable functionality in production:
    • /transform/wikitext/to/lint
    • /transform/wikitext/to/lint/{title}
    • /transform/wikitext/to/lint/{title}/{revision}
  • Test functionality to identify gaps in behavior between RESTbase and Core implementations
    • Include tests for recent header parameter issues, such as eTag
  • Document any gaps in functionality on this ticket.
  • Fix any small issues that are identified; if a larger fix is required, create a subtask for the team to review.

OPEN QUESTION: Do we need to have a compatibility mode flag? Assume not since this feature was not previously enabled in production.

Event Timeline

HCoplin-WMF set the point value for this task to 5.

Change #1151716 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[mediawiki/core@master] REST: Enable wikitext to lint transformations

https://gerrit.wikimedia.org/r/1151716

aaron raised the priority of this task from Medium to High.Jun 17 2025, 3:21 PM

Change #1151716 merged by jenkins-bot:

[mediawiki/core@master] REST: Enable wikitext to lint transformations

https://gerrit.wikimedia.org/r/1151716

Change #1164962 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] Revert "REST: Enable wikitext to lint transformations"

https://gerrit.wikimedia.org/r/1164962

Change #1164962 merged by jenkins-bot:

[mediawiki/core@master] Revert "REST: Enable wikitext to lint transformations"

https://gerrit.wikimedia.org/r/1164962

The patch was reverted because it caused T398175. Looks like AbuseFilter tests are unhappy with $wgParsoidSettings['linting'] = true;

Change #1164988 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[mediawiki/core@master] REST: Enable wikitext to lint transformations (2nd try)

https://gerrit.wikimedia.org/r/1164988

HCoplin-WMF changed the point value for this task from 5 to 1.Jul 1 2025, 2:39 PM
HCoplin-WMF set Final Story Points to 5.
HCoplin-WMF changed the point value for this task from 1 to 3.Jul 1 2025, 3:28 PM

Change #1164988 merged by jenkins-bot:

[mediawiki/core@master] REST: Enable wikitext to lint transformations (2nd try)

https://gerrit.wikimedia.org/r/1164988

Change #1169794 had a related patch set uploaded (by Aaron Schulz; author: Aaron Schulz):

[mediawiki/core@master] Tweak and enable the lint/{title}/{revision} mocha tests

https://gerrit.wikimedia.org/r/1169794

Change #1169794 merged by jenkins-bot:

[mediawiki/core@master] Tweak and enable the lint/{title}/{revision} mocha tests

https://gerrit.wikimedia.org/r/1169794

It's interesting that

POST https://en.wikipedia.org/api/rest_v1/transform/wikitext/to/lint/NASA
POST https://en.wikipedia.org/api/rest_v1/transform/wikitext/to/lint/NASA/1301322295

...require the wikitext field in the body whereas

POST https://en.wikipedia.org/w/rest.php/v1/transform/wikitext/to/lint/NASA
POST https://en.wikipedia.org/w/rest.php/v1/transform/wikitext/to/lint/NASA/1301322295

...do not. The later do accept the wikitext field, so it rest.php isn't allowing less stuff than restbase, so I suppose that's fine.

It's also a bit scary how many parameters get pulled in via ParsoidHandler::getRequestAttributes(). Since it lets various parameters get provided via request body or query parameters (e.g. "bodyOnly"/"offsetType") or request body alone (/"wikitext"/"original", ect...), we have to maintain a POST route for these paths to support the title-based POST transform/wikitext/to/lint/ restbase routes (it cannot be GET only) . To support Envoy mapping of title-based GET page/lint/ restbase routes, we need rest.php GET counterparts or compatible GET wikitext/to/lint routes. Since all this rest.php transform stuff goes to the same TransformHandler, the later option is for coreRoutes.json to have:

{
		"method": [ "GET", "POST" ]
		"path": "/v1/transform/wikitext/to/lint/{title}",
		...
}
...
{
		"method": [ "GET", "POST" ],
		"path": "/v1/transform/wikitext/to/lint/{title}/{revision}",
		...
}

The getRequestAttributes() method also sets "stash". The restbase title-based GET page/html/ and GET /wikitext/to/html/ routes have a "stash" parameter, which is kind of fishy for the GET case from a "safe request" perspective (even if hidden behind If-Match). In any case, this doesn't matter for the lint route, since it's not respected for restbase GET page/lint nor POST /wikitext/to/lint nor POST wikitext/to/lint/title(/revision). Likewise, rest.php lint routes correctly only apply "stash" for wikitext/to/html, per wt2html(), but not "wikitext/to/lint", per w2lint().

One ramification of clients choosing to use POST when GET would have sufficed is that the request will always go to the primary datacenter, even if it's farther away. If a body parameter is needed, I guess they could use POST with the header Promise-Api-Nonwrite as a work-around.

Still, I prefer "one way of doing things", so perhaps rest.php title-based POST wikitext/to/lint/ requests with no body params (e.g. {}) could be considered deprecated. This is exactly what restbase is doing in getLintErrors() for the title-based page/lint and parsoid/lint routes: making a POST request with a body of {}, a case were GET makes more sense. We don't have warning headers AFAIK, but we can log such things. In the future, TransformHandler would eventually be broken up or wrapped to make reasonably concrete Handler classes. This would make it way easier to reason about parameters and do api spec generation.

Change #1172712 had a related patch set uploaded (by Aaron Schulz; author: Aaron Schulz):

[mediawiki/core@master] Support GET requests for title-based wikitext/to/lint/ routes

https://gerrit.wikimedia.org/r/1172712

It's also a bit scary

I find these endpoints generally scary and hard to work with. That's not a criticism of the initial implementers, who were trying to solve a complex problem at a time when the MW REST API framework was much less mature than it is today. But it is an indication that improvements would be helpful.

My recollection is that some of the existing use cases may be retired once this rerouting/migration is complete, reducing the different contexts in which these endpoints are called. If I'm right about that, then it probably makes sense to wait until that has occurred to make large changes. But maybe our future selves could set aside some time to refactor these endpoints, bringing them up to current patterns. Otherwise, they're going to continue to be challenging to maintain.

This would be a good effort to do sometime after we make already-planned improvements to our monitoring and alerting systems, as there's a pretty high chance of accidental breakage, Better monitoring/alerting would help us limit any impact.

I find these endpoints generally scary and hard to work with. That's not a criticism of the initial implementers, who were trying to solve a complex problem at a time when the MW REST API framework was much less mature than it is today.

Afaik this started as a port of JS code from the original Node.js Parsoid. It has undercone a couple of refactorings since, to make it more MediaWiki-like, but we kept the original behavior intact. @cscott probably has the full history.

Change #1172712 merged by jenkins-bot:

[mediawiki/core@master] Support GET requests for title-based wikitext/to/lint/ routes

https://gerrit.wikimedia.org/r/1172712