Page MenuHomePhabricator

Make WDQS responses cacheable by eliminating chunking
Closed, ResolvedPublic

Description

Currently, WDQS responses are sent with Transfer-encoding: chunked which prevents varnish from caching the results. We should find a way to make these results cacheable at least for smaller query results.

Event Timeline

Smalyshev created this task.Mar 2 2016, 9:32 PM
Restricted Application added projects: Wikidata, Discovery. · View Herald TranscriptMar 2 2016, 9:32 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Gehel added a subscriber: Gehel.Mar 3 2016, 8:42 PM
Smalyshev triaged this task as Normal priority.Mar 3 2016, 10:10 PM
Smalyshev added a subscriber: BBlack.

According to @BBlack it may be a problem peculiar to our varnish config, which then will be solved with upgrade to Varnish 4, planned sometime in Q4. With this, I think the best course of action, unless we find an easy solution soon, would be to just wait for that.

OTOH, it looks like RestBase has caching with chunked, so maybe there's still something missing here.

Turns out the problem with chunking was actually a config error in misc cluster:

sub misc_fetch_large_objects {
    // Stream objects >= 1MB in size
    if (std.integer(beresp.http.Content-Length, 1048576) >= 1048576 || beresp.http.Content-Length ~ "^[0-9]{8}") {
        set beresp.do_stream = true;
        // hit_for_pass on objects >= 10MB in size (no effect on backends that always (pass) anyways)
        if (std.integer(beresp.http.Content-Length, 10485760) >= 10485760 || beresp.http.Content-Length ~ "^[0-9]{9}") {
            return (hit_for_pass);
        }
    }
}

This assigns length of 10M to any request that has no Content-Length and then decides not to cache it.

Smalyshev closed this task as Resolved.Mar 3 2016, 11:48 PM
Smalyshev claimed this task.

Further handling of it in T128813