Decom legacy ex-parsoidcache cxserver, citoid, and restbase service hostnames
Open, NormalPublic0 Story Points

Description

Related: T110472

This is about eventually decommissioning the alternate hostname entrypoints for these services. Going forward, these should all be using /api/rest_v1/ on the wiki domains themselves.

Current Status

Original Explanation
Quoting from the wikimedia.org DNS zonefile, the current list of these entrypoint hostnames is:

;;;; cxserver/citoid/rest(base) below are LEGACY, to be removed eventually
cxserver.eqiad      600 IN DYNA geoip!text-addrs
cxserver            600 IN DYNA geoip!text-addrs
citoid.eqiad        600 IN DYNA geoip!text-addrs
citoid              600 IN DYNA geoip!text-addrs
restbase.eqiad      600 IN DYNA geoip!text-addrs
restbase            600 IN DYNA geoip!text-addrs
rest                600 IN DYNA geoip!text-addrs

There's also some varnish code in support of these on the text cluster.

I'd like to start with pulling the .eqiad. subdomain entries - these should never have been used publicly and probably aren't now, and we can try to confirm that with traffic logging and code searches (aside: perhaps we should do better messaging within the org that any hostname in a datacenter-specific public subdomain is purely for infrastructure tracking/debugging, not for use as an official service hostname).

Related Objects

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a project: Operations. · View Herald TranscriptApr 19 2016, 1:47 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
BBlack triaged this task as Normal priority.Apr 19 2016, 1:47 AM
BBlack updated the task description. (Show Details)

citoid.wm.org and cxserver.wm.org are still being actively and officially used, as we haven't put them behind RESTBase yet. We are officially dropping support for rest*.wm.org next Monday.

I'd like to start with pulling the .eqiad. subdomain entries

+1

Restricted Application added a project: VisualEditor. · View Herald TranscriptApr 20 2016, 7:32 AM
Jdforrester-WMF set the point value for this task to 0.Apr 28 2016, 1:26 AM
BBlack added a comment.May 9 2016, 6:53 PM

I saw the deprecation email about RESTBase itself. Are we clear to pull the DNS for rest(base)?.wm.o? I'm going to do some Host: and/or DNS logging about the other .eqiad names to make sure we don't have significant usage volume before pulling them.

@BBlack, if there is no specific reason to pull the DNS soon, I would propose to keep the current RB error / redirect notice up for a little longer. We definitely don't serve any content from rest.wikimedia.org any more, but there are still links from old announcements out there that some people might still follow.

BBlack added a comment.May 9 2016, 7:00 PM

And actually, on the issue of the .eqiad. hostnames: they've been broken for a while due to out TLS redirects and our certs work, so no point leaving them around.

BBlack added a comment.May 9 2016, 7:03 PM

@BBlack, if there is no specific reason to pull the DNS soon, I would propose to keep the current RB error / redirect notice up for a little longer. We definitely don't serve any content from rest.wikimedia.org any more, but there are still links from old announcements out there that some people might still follow.

Ok maybe I'm confused on this point. Testing requests like https://restbase.wikimedia.org/en.wikipedia.org/v1/page/mobile-text/Occitan_language, they seem to still work fine.

@BBlack, restbase.wikimedia.org was never published or in any way supposed to work. All public references were to rest.wikimedia.org, which now returns errors with a hint on the move: https://rest.wikimedia.org/en.wikipedia.org/v1/page/mobile-text/Occitan_language

So yes, it would be great to remove *restbase*.wikimedia.org from DNS, but leave the old official entry point *rest*.wikimedia.org around for a little longer.

Change 287682 had a related patch set uploaded (by BBlack):
Remove (citoid|cxserver|restbase).eqiad.wm.o hostnames

https://gerrit.wikimedia.org/r/287682

BBlack added a comment.May 9 2016, 7:20 PM

Ok got it, thanks!

Change 287682 merged by BBlack:
Remove deprecated/dysfunctional wm.o service hostnames

https://gerrit.wikimedia.org/r/287682

Krinkle added a subscriber: Krinkle.EditedMay 9 2016, 8:16 PM

restbase 600 IN DYNA geoip!text-addrs

Despite it being undocumented, beware that it was commonly used (albeit by accident) by several developers, including staff:

BBlack added a comment.May 9 2016, 8:19 PM

If something important breaks, we can put that name back in. But in that case, we should also update RB to deny service there with update info, like it does for rest.wikimedia.org requests.

Krinkle removed a subscriber: Krinkle.May 16 2016, 1:12 PM

Re the rest.wikimedia.org: we effectively shut off service back in April ( https://lists.wikimedia.org/pipermail/wikitech-l/2016-April/085309.html ) but the hostname is still live and connected to the RB backends through the caches, just to serve the notice that it moved. Has it been long enough now that we can go ahead and remove it? If not, do we have some kind of plan or timeline for when?

Hm, I can see 40k log entries for this domain in the logs for requests coming from clients from all over the world. Note that the are logged with a 10% probability, so the actual number of requests is much larger. However, since 2016-04-14 we've been returning the 410 Gone status, so I'd be +1 on sending yet another announcement and turning it off in a month or so.

Sounds good to me, you want to do it, or me, or @GWicke since he sent the first one?

Sounds good to me, you want to do it, or me, or @GWicke since he sent the first one?

I'd let @GWicke do it for consistency :P (but also because for some reason I can't post to api-announce).

Okay, I'll send out an announcement today. Lets say we'll switch it off by September 1st?

Okay, I'll send out an announcement today.

Awesome, thnx!

Lets say we'll switch it off by September 1st?

Yup, sounds good.

BBlack updated the task description. (Show Details)Jul 22 2016, 4:17 PM

thanks!

Mvolz moved this task from Backlog to Production on the Citoid board.Jul 29 2016, 3:03 PM

Mentioned in SAL [2016-08-17T21:47:24Z] <cscott> updated OCG to version e3e0fd015ad8fdbf9da1838c830fe4b075c59a29 (T133001, T142226)

Change 308723 had a related patch set uploaded (by BBlack):
remove rest.wikimedia.org from DNS

https://gerrit.wikimedia.org/r/308723

Change 308724 had a related patch set uploaded (by BBlack):
text VCL: remove support for rest.wm.o

https://gerrit.wikimedia.org/r/308724

Change 308723 merged by BBlack:
remove rest.wikimedia.org from DNS

https://gerrit.wikimedia.org/r/308723

Change 308724 merged by BBlack:
text VCL: remove support for rest.wm.o

https://gerrit.wikimedia.org/r/308724

BBlack added a comment.Sep 6 2016, 4:53 PM

rest.wikimedia.org is gone. We've still got citoid.wikimedia.org and cxserver.wikimedia.org for eventual removal when they eventually move to a Restbased API.

BBlack moved this task from Triage to General on the Traffic board.Oct 4 2016, 12:45 PM
Restricted Application added a project: ContentTranslation. · View Herald TranscriptOct 4 2016, 12:45 PM
BBlack moved this task from General to Watching on the Traffic board.Oct 4 2016, 12:45 PM
tom29739 removed a subscriber: tom29739.
tom29739 added a subscriber: tom29739.

Change 350505 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[operations/puppet@production] Decom legacy citoid service hostname

https://gerrit.wikimedia.org/r/350505

Change 350748 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[operations/dns@master] Remove the citoid.wm.org DNS record

https://gerrit.wikimedia.org/r/350748

Change 350748 merged by BBlack:
[operations/dns@master] Remove the citoid.wm.org DNS record

https://gerrit.wikimedia.org/r/350748

Change 350505 merged by BBlack:
[operations/puppet@production] Decom legacy citoid service hostname

https://gerrit.wikimedia.org/r/350505

So at this point basically only cxserver is remaining. Work on that is ongoing in T107914.

GWicke moved this task from doing to blocked on the Services board.
GWicke edited projects, added Services (blocked); removed Services (doing).
GWicke raised the priority of this task from Normal to Needs Triage.
GWicke triaged this task as Normal priority.

Change 471246 had a related patch set uploaded (by Mvolz; owner: Mvolz):
[operations/mediawiki-config@master] Removed decomissioned citoid url

https://gerrit.wikimedia.org/r/471246

Change 471246 merged by jenkins-bot:
[operations/mediawiki-config@master] Removed decomissioned citoid url

https://gerrit.wikimedia.org/r/471246

Mentioned in SAL (#wikimedia-operations) [2018-11-05T21:38:38Z] <thcipriani@deploy1001> Synchronized wmf-config/CommonSettings.php: [[gerrit:471246|Removed decomissioned citoid url]] T133001 (duration: 00m 53s)