Page MenuHomePhabricator

Add cache headers to the datasets.wikimedia.org/limn-public-data/metrics folder {lion} [5 pts]
Closed, ResolvedPublic

Event Timeline

Milimetric claimed this task.
Milimetric raised the priority of this task from to Needs Triage.
Milimetric updated the task description. (Show Details)
Milimetric added a project: Analytics-Kanban.
Milimetric added a subscriber: Milimetric.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 2 2015, 5:01 PM
kevinator renamed this task from Add cache headers to the datasets.wikimedia.org/limn-public-data/metrics folder to Add cache headers to the datasets.wikimedia.org/limn-public-data/metrics folder {lion}.Jun 2 2015, 5:02 PM
kevinator set Security to None.
kevinator triaged this task as High priority.Jun 3 2015, 9:14 PM
kevinator renamed this task from Add cache headers to the datasets.wikimedia.org/limn-public-data/metrics folder {lion} to Add cache headers to the datasets.wikimedia.org/limn-public-data/metrics folder {lion} [? pts].Jun 12 2015, 4:15 PM
Milimetric renamed this task from Add cache headers to the datasets.wikimedia.org/limn-public-data/metrics folder {lion} [? pts] to Add cache headers to the datasets.wikimedia.org/limn-public-data/metrics folder {lion} [5 pts].Jun 15 2015, 4:59 PM
Milimetric edited projects, added Analytics-Kanban; removed Analytics-Backlog.

Change 218534 had a related patch set uploaded (by Milimetric):
Add cache headers for datasets.wikimedia.org

https://gerrit.wikimedia.org/r/218534

Change 218534 merged by Ottomata:
Add cache headers for datasets.wikimedia.org

https://gerrit.wikimedia.org/r/218534

kevinator closed this task as Resolved.Jun 19 2015, 10:45 PM
kevinator added a subscriber: kevinator.

Change 218534 merged by Ottomata:
Add cache headers for datasets.wikimedia.org
https://gerrit.wikimedia.org/r/218534

This change added max-age=86400 to the cache headers for ALL json/yaml/csv/tsv files served from datasets.wikimedia.org. This includes things like https://datasets.wikimedia.org/aggregate-datasets/search/app_event_counts.tsv , which change much more frequently. There's a cron job that rsyncs the aggregate-datasets directory in from stat1003 every half hour, so a caching max-age of 24 hours is not appropriate.

This caused the two misc-web Varnishes to get split-brained about the contents of this file, serving data that was hours out of date: https://gist.github.com/catrope/016fd02d5bc2ee7b4c35

Catrope reopened this task as Open.Jun 22 2015, 11:59 PM
Milimetric closed this task as Resolved.Jun 23 2015, 3:40 PM

This is resolved, as we need 24 hour cache on most of the files served here. We have to figure out how to handle exceptions to that rule without requiring ops intervention.