Page MenuHomePhabricator

Set up Varnish purging and caching for RESTBase end points
Closed, ResolvedPublic

Description

One of the main goals behind moving to a REST Content API is enabling caching of API responses across the board. This will help us scale our APIs to keep up with growing demand, and lowers latency for clients by leveraging our geo-distributed caching infrastructure.

RESTBase requests are already proxied through the regular text varnishes, but caching of responses is still disabled by setting headers to that effect. Before we can allow Varnish to cache specific GET responses, we'll need to set up Varnish purging for those end points. The production logic for this lives in HTCPPurge. The UDP logic itself doesn't look too hard. The config data referenced there is simple enough (one IP address & one TTL) to let us manually replicate it in the RESTBase config for now.

Event Timeline

GWicke raised the priority of this task from to Needs Triage.
GWicke updated the task description. (Show Details)
GWicke added subscribers: GWicke, BBlack.
GWicke set Security to None.
GWicke edited subscribers, added: Pchelolo, mobrovac; removed: Aklapper.

We just spoke about this on IRC. It turns out that my fears about the config complexity were unfounded:

$wgHTCPRouting = array( '' => array( 'host' => '239.128.0.112', 'port' => 4827));
$wgHTCPMulticastTTL = 8;

So, I think for now we can add the same data in the RESTBase config & port the HTCPPurge method over to JS. It's probably worth making this a separate npm module, so that other services can potentially use it as well. Perhaps call it 'htcp-purge'?

GWicke renamed this task from Figure out a way to purge Varnishes from RESTBase to Set up Varnish purging and caching for RESTBase end points.Aug 21 2015, 5:50 PM
GWicke added a subscriber: mark.
GWicke triaged this task as Medium priority.Jan 25 2016, 9:18 PM

As request volumes for HTML resources (especially without an explicit revision) grow, it would be nice to start purging & caching those.

We should figure out whether it makes more sense to do this in RESTBase, or the change propagation service (T117933). I have gone back & forth on this before, but am recently leaning towards RESTBase, mainly so that RESTBase is in full control of its public URL layout & cache setup. @Pchelolo, @mobrovac, @Eevans: I'm curious what you think on this.

As a follow-up, we have decided to purge URLs directly from RESTBase for the time being (cf PR 515). The long-term plan is to leverage the Event-Platform system to expose all resource changes (cf T126687: RFC: Publish all resource changes to a single topic) which would be fed to a purging service.

Finally, this is done. RESTBase endpoints are cached and actively purged where appropriate. Some more improvements still to be done, but those are covered by T126571 and T127387