Page MenuHomePhabricator
Create Task
Maniphest T137238

Tune WDQS caching headers
Closed, ResolvedPublic
Actions

Description

At the moment, WDQS exposes quite a few caching headers, some of them not making sense. For example, ETags are generated by nginx and are different on each nginx server (nginx generates ETags based on last modified time, which is not garanteed to be coherent).

As we are now adding a hash of the file content to page resources (CSS, JS, ...), we can aggressively increase the caching time of those resources.

Queries themselves are probably only cacheable for a very fairly short time.

Some discussion on this has already happened on T133026. @BBlack input on this would be more than welcomed!

Details

SubjectRepoBranchLines +/-
WDQS caching headersoperations/puppetproduction+6 -0
Don't publish etags for WDQSoperations/puppetproduction+5 -0
Customize query in gerrit

Related Objects

Event Timeline

Gehel created this task.Jun 7 2016, 6:40 PM
Restricted Application added a project: Wikidata. · View Herald TranscriptJun 7 2016, 6:40 PM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald Transcript
Smalyshev subscribed.Jun 7 2016, 7:30 PM

Yes, after the switch to built GUI we can cache at least the hashed CSS/JS pretty much forever, they'd never change. Non-hashed ones probably fine with time like 1 day or so, we change them about once a week. If we want to have different rules, it may require some nginx configs, but should be possible.

Not sure about etags, didn't study the subject in depth.

Queries are now cached for 5 mins, I think we can keep it this way unless we find any reason for change.

Smalyshev moved this task from Incoming to Watching / Waiting on the Wikidata-Query-Service board.Jun 8 2016, 12:08 AM
Gehel added a comment.Jun 9 2016, 12:12 PM

It seems that at the moment only .js and .css files which ar in the js / css directories are processed by filerev. Is there any reason to not process other files as well? It seems that we have .json files that could benefit from the same processing.

gerritbot subscribed.Jun 9 2016, 12:16 PM

Change 293492 had a related patch set uploaded (by Gehel):
Don't publish etags for WDQS

https://gerrit.wikimedia.org/r/293492

gerritbot added a project: Patch-For-Review.Jun 9 2016, 12:16 PM
• Jonas added a comment.Jun 9 2016, 1:02 PM

We can only use filerev for files that are referenced from the html file, because the application itself is not aware about filerev and then will not find its resources.

The json files are for i18n, so there will be no problem when they are outdated.

Gehel added a comment.Jun 9 2016, 4:00 PM

We shoudl be able to also process at least:

  • vendor/jquery.uls/css/jquery.uls.css
  • logo.svg

They are unlikely to change frequently but still it would be nice...

At this point I'm not entirely sure which resources can be cached and which cannot. My guess:

  • main page (index.html) -> no cache (or very short)
  • everything under /css and /js -> long cache (7 days to start ?)
  • everything else -> short cache (5 minutes?)

Does this sound reasonable to you?

• Jonas added a comment.Jun 9 2016, 4:37 PM
In T137238#2368243, @Gehel wrote:

We shoudl be able to also process at least:

  • vendor/jquery.uls/css/jquery.uls.css

This is fixed.

  • main page (index.html) -> no cache (or very short)
  • everything under /css and /js -> long cache (7 days to start ?)
  • everything else -> short cache (5 minutes?)

Does this sound reasonable to you?

Yes, @Smalyshev ?

I am not sure what the general strategy should be, but I guess we don't need aggressive caching at the moment, because we don't have a lot of load...

Smalyshev added a comment.Jun 9 2016, 4:58 PM
  • main page (index.html) -> no cache (or very short)

I would cache it for the same as below. It doesn't change that much - pretty much once a week now. While re-fetching it is no big deal, I think we still can cache it for like 5 mins.

  • everything under /css and /js -> long cache (7 days to start ?)
  • everything else -> short cache (5 minutes?)

Makes sense.

gerritbot added a comment.Jun 10 2016, 1:35 PM

Change 293492 merged by Gehel:
Don't publish etags for WDQS

https://gerrit.wikimedia.org/r/293492

Gehel mentioned this in rOPUP76e9e66febc3: Don't publish etags for WDQS.Jun 10 2016, 1:36 PM
Gehel mentioned this in rOPUP9455cbb5b87a: Don't publish etags for WDQS.Jun 17 2016, 6:06 PM
Addshore moved this task from incoming to monitoring on the Wikidata board.Jun 22 2016, 1:02 PM
MoritzMuehlenhoff triaged this task as Medium priority.Jul 8 2016, 10:14 AM
Smalyshev added a comment.Aug 23 2016, 4:22 AM

@Gehel is anything left to do in this ticket?

gerritbot added a comment.Aug 23 2016, 8:44 AM

Change 306163 had a related patch set uploaded (by Gehel):
WDQS caching headers

https://gerrit.wikimedia.org/r/306163

Gehel added a comment.Aug 23 2016, 8:45 AM

@Smalyshev yes there is: adding some cache-control headers. Change submitted. Thanks for reminding me!

Gehel mentioned this in rOPUPa6f4b5ef23f7: WDQS caching headers.Aug 23 2016, 8:51 AM
Gehel mentioned this in rOPUP4664b321a09f: WDQS caching headers.Aug 24 2016, 9:55 AM
gerritbot added a comment.Aug 24 2016, 9:56 AM

Change 306163 merged by Gehel:
WDQS caching headers

https://gerrit.wikimedia.org/r/306163

Gehel closed this task as Resolved.Aug 24 2016, 10:30 AM
Gehel claimed this task.

New caching headers deployed. Checked with chrome, cache-control headers are showing up.

debt moved this task from Needs triage to WDQS on the Discovery-ARCHIVED board.Aug 24 2016, 5:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL