Page MenuHomePhabricator
Create Task
Maniphest T375616

Block RESTBase cxserver v1 endpoints in favor of the new endpoints
Closed, ResolvedPublic
Actions

Description

Background Information

After discussions on T372753: Decommission cxserver endpoints from RESTBase and investigations on the RESTBase codebase, we realised that cxserver v1 endpoints are not supported anymore and they should be removed from our infrastructure.

What

Like we did for MCS in T328036: MCS decommission (2023) we should block the requests before removing the code.

How

Block the following endpoints:

  • POST <domain>/api/rest_v1/transform/html/from/{from}
  • POST <domain>/api/rest_v1/transform/html/from/{from}/{provider}
  • GET <domain>/api/rest_v1/transform/word/from/{from}/{word}
  • GET <domain>/api/rest_v1/transform/word/from/{from}/{word}/{provider}
  • GET <domain>/api/rest_v1/page/segments/{title}
  • GET <domain>/api/rest_v1/page/segments/{title}/{revision}
  • POST <domain>/api/rest_v1/transform/html/from/{from_lang}/to/{to_lang}
  • POST <domain>/api/rest_v1/transform/html/from/{from_lang}/to/{to_lang}/{provider}
  • GET <domain>/api/rest_v1/transform/word/from/{from_lang}/to/{to_lang}/{word}
  • GET <domain>/api/rest_v1/transform/word/from/{from_lang}/to/{to_lang}/{word}/{provider}
  • GET <domain>/api/rest_v1/list/pair/{from}/{to}/
  • GET <domain>/api/rest_v1/list/tool/{tool}
  • GET <domain>/api/rest_v1/list/tool/{tool}/{from}/{to}
  • GET <domain>/api/rest_v1/list/languagepairs/

Why is safe to block requests

It's safe to block the requests for the following reasons:

  • Looking at the requests data, example here, no relevant clients consume it and the majority of the requests are attempts to exploit the endpoint.
  • Clients already had time to migrate cxserver to v2, and there's no appetite to make further communications.
  • The endpoints are either not supported or broken

Related Objects

Event Timeline

MSantos created this task.Sep 25 2024, 12:04 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 25 2024, 12:04 PM
MSantos renamed this task from Block to [DRAFT] Block RESTBase cxserver v1 endpoints in favor of the new endpoints.Sep 25 2024, 12:04 PM
Reedy added projects: RESTBase Sunsetting, CX-cxserver.Sep 25 2024, 3:05 PM
abi_ subscribed.Oct 22 2024, 12:50 PM

The cxserver API v2 end-points were introduced back in 2017 (383788: cxserver v2 apis | https://gerrit.wikimedia.org/r/c/mediawiki/services/cxserver/+/383788) but we never formally deprecated the v1 API's.

We were looking at the clients of cxserver v1 endpoint, and noticed that Translate uses it, but not via RESTBase.

I think it should be safe to remove the cxserver RESTBase endpoint.

MSantos renamed this task from [DRAFT] Block RESTBase cxserver v1 endpoints in favor of the new endpoints to Block RESTBase cxserver v1 endpoints in favor of the new endpoints.Oct 29 2024, 1:33 PM
MSantos updated the task description. (Show Details)
MSantos added a subscriber: Jgiannelos.
Jgiannelos added a comment.Oct 29 2024, 2:35 PM

Looks OK to me.

MSantos added a project: Traffic.Oct 29 2024, 4:36 PM
santhosh subscribed.Oct 30 2024, 5:52 AM

Looks good to me

MSantos triaged this task as High priority.Oct 30 2024, 1:27 PM
MSantos added a project: Essential-Work.
MSantos mentioned this in T372753: Decommission cxserver endpoints from RESTBase.
MSantos added a subscriber: akosiaris.
Nikerabbit moved this task from Backlog to Maintenance and operational issues on the CX-cxserver board.Nov 11 2024, 11:57 AM
akosiaris claimed this task.Nov 19 2024, 5:22 PM

I gather that we disable this at the CDN layer at any time, so I 'll be doing it this week. Let me know if the timing is bad (or any other blockers)

MSantos added a comment.Nov 19 2024, 6:56 PM
In T375616#10337009, @akosiaris wrote:

I gather that we disable this at the CDN layer at any time, so I 'll be doing it this week. Let me know if the timing is bad (or any other blockers)

That's right, you can proceed as there are no blockers.

Stashbot added a comment.Nov 20 2024, 9:38 AM

Mentioned in SAL (#wikimedia-operations) [2024-11-20T09:38:15Z] <akosiaris> decommission cxserver endpoints /api/rest_v1/list/(pair|tool|languagepairs) from RESTBase T375616

akosiaris updated the task description. (Show Details)Nov 20 2024, 9:41 AM
akosiaris updated the task description. (Show Details)
akosiaris added a comment.Nov 20 2024, 10:47 AM

I 've gone ahead and blocked

GET <domain>/api/rest_v1/list/pair/{from}/{to}/
GET <domain>/api/rest_v1/list/tool/{tool}
GET <domain>/api/rest_v1/list/tool/{tool}/{from}/{to}
GET <domain>/api/rest_v1/list/languagepairs/

As a first easy step. HTTP status code is a 403 with a message CXServer decommissioned. See https://phabricator.wikimedia.org/T372753

I am going through the list and I think I 've spotted a couple of errrors. Namely the following lack a page part in the URL. Correct me if I am wrong.

currentshould be
GET <domain>/api/rest_v1/segments/{title}GET <domain>/api/rest_v1/page/segments/{title}
GET <domain>/api/rest_v1/segments/{title}/{revision}GET <domain>/api/rest_v1/page/segments/{title}/{revision}

Furthermore the following (from transform-global.yaml) do not apply to any domain, but only to wikimedia.org. So we have the following table.

currentshould be
POST <domain>/api/rest_v1/transform/html/from/{from_lang}/to/{to_lang}POST wikimedia.org/api/rest_v1/transform/html/from/{from_lang}/to/{to_lang}
POST <domain>/api/rest_v1/transform/html/from/{from_lang}/to/{to_lang}/{provider}POST wikimedia.org/api/rest_v1/transform/html/from/{from_lang}/to/{to_lang}/{provider}
GET <domain>/api/rest_v1/transform/word/from/{from_lang}/to/{to_lang}/{word}GET wikimedia.org/api/rest_v1/transform/word/from/{from_lang}/to/{to_lang}/{word}
GET <domain>/api/rest_v1/transform/word/from/{from_lang}/to/{to_lang}/{word}/{provider}GET wikimedia.org /api/rest_v1/transform/word/from/{from_lang}/to/{to_lang}/{word}/{provider}

@MSantos does the above look correct to you?

akosiaris added a comment.Nov 20 2024, 11:17 AM

I just noticed that the wikimedia.org argument applies to

GET <domain>/api/rest_v1/list/pair/{from}/{to}/
GET <domain>/api/rest_v1/list/tool/{tool}
GET <domain>/api/rest_v1/list/tool/{tool}/{from}/{to}
GET <domain>/api/rest_v1/list/languagepairs/

as well. They aren't exposed in anything else, as far as I can tell, than wikimedia.org domain.

akosiaris updated the task description. (Show Details)Nov 20 2024, 11:21 AM
Stashbot added a comment.Nov 20 2024, 11:22 AM

Mentioned in SAL (#wikimedia-operations) [2024-11-20T11:22:39Z] <akosiaris> decommission cxserver endpoints /api/rest_v1/transform/html/from, /api/rest_v1/transform/word/from from RESTBase T375616

MSantos added a comment.Nov 20 2024, 11:55 AM
In T375616#10339426, @akosiaris wrote:

[...]
@MSantos does the above look correct to you?

Yes, it does. Thanks for raising it.

akosiaris updated the task description. (Show Details)Nov 20 2024, 11:56 AM
akosiaris closed this task as Resolved.Nov 20 2024, 12:12 PM
akosiaris updated the task description. (Show Details)

All done. Resolving, hopefully we won't have to reopen.

akosiaris added a comment.Nov 20 2024, 12:14 PM

Courtesy of requestctl, we have the following superset view we can monitor to see if someone tries to use those endpoints in the future.

MSantos awarded a token.Wed, Nov 27, 12:47 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits