Page MenuHomePhabricator

Code changes in VCL to add a Last-Access cookie to track "top domain" uniques [13 pts] {bear}
Closed, ResolvedPublic

Event Timeline

Nuria created this task.Mar 11 2015, 7:47 PM
Nuria updated the task description. (Show Details)
Nuria raised the priority of this task from to High.
Nuria claimed this task.
Nuria added projects: Analytics, Analytics-Kanban.
Nuria added subscribers: kevinator, ggellerman.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 11 2015, 7:47 PM

Change 196009 had a related patch set uploaded (by Nuria):
Adding a Last-Access cookie to text and mobile requests

https://gerrit.wikimedia.org/r/196009

Nuria moved this task from Next Up to In Progress on the Analytics-Kanban board.Mar 12 2015, 2:16 PM
kevinator renamed this task from Code changes in VCL to add a Last-Acess cookie to track "top domain" uniques to Code changes in VCL to add a Last-Acess cookie to track "top domain" uniques {bear}.
kevinator set Security to None.
Nuria renamed this task from Code changes in VCL to add a Last-Acess cookie to track "top domain" uniques {bear} to Code changes in VCL to add a Last-Access cookie to track "top domain" uniques {bear}.Mar 12 2015, 3:36 PM
kevinator renamed this task from Code changes in VCL to add a Last-Access cookie to track "top domain" uniques {bear} to Code changes in VCL to add a Last-Access cookie to track "top domain" uniques [13 pts] {bear}.Mar 12 2015, 3:48 PM
kevinator moved this task from In Progress to In Code Review on the Analytics-Kanban board.
Nuria reassigned this task from Nuria to kevinator.Apr 22 2015, 5:00 PM

Change 196009 had a related patch set uploaded (by BBlack):
Adding a Last-Access cookie to text and mobile requests

https://gerrit.wikimedia.org/r/196009

BBlack added a subscriber: BBlack.Apr 23 2015, 1:44 PM

^ Aside from the technical-level work in PS10, a few other things have changed that affect other code, and I have a few further questions, etc:

  1. I've changed the format of the WMF-Last-Access value in X-Analytics to look like: "01-Jun-2000". It had to change from how it looked previously because (a) Cookie values shouldn't have whitespace or commas (among other bits) and (b) it's not worth the efficiency hit to convert whatever's in the cookie to a different format for X-Analytics within Varnish. It's still implicitly in the UTC timezone in terms of date cutoffs.
  2. I've tentatively changed the cookie expiration from +30 days to +32 days, on the theory that this ensures it does live for a full "month" regardless of month-size or the time-of-day it was set at (and whoever's digging through this for stats can always cut out the data for cookies which thus end up a day or two past their usefulness - they'd have to do this for at least Feb anyways). We can easily go back if you'd rather have it expire on exactly 30d.
  3. Should we be anonymizing the expiry value a bit? What it leaks in its current form is "The first time during each UTC day that the user hit the site, precisely down to the second", but it doesn't leak anything about further hits that day. We could round it off to the nearest 6 hours or something like that and avoid that small leak, if anyone feels it's worth it.
BBlack moved this task from Triage to In Progress on the Traffic board.Apr 23 2015, 1:49 PM
bd808 added a subscriber: bd808.Apr 23 2015, 3:16 PM
BBlack added a comment.EditedApr 23 2015, 4:59 PM

PS11 refactors a bit further to work around the Cookie-access issues by splitting into recv and deliver halves, which I think was the last real outstanding issue in the way of this code working in prod.

I've tested the core code of PS11 in an isolated default varnish server and confirmed that it seems to behave as expected for real requests as well, but not yet in beta with a full load of normal VCL alongside it.

The 3 questions/points above still need clarification.

+ Dan (milimetric) from Analytics Engineering now that Nuria is on leave

Thanks Grace. Brandon, I'm getting caught up on this and will respond shortly. I'll own the patch going forward. Btw, if you need me I'm milimetric on IRC as well, and I'm always in the ops channel.

Brandon, I've commented on the patch, I basically agreed with your first two changes and also agreed with your proposed third change. But I didn't update the patch. Let me know if you'd rather I take a shot next time, I'm happy to do that. Thanks very much for the thoughtful work.

hey @BBlack, when will this be deployed to all varnishes? I'm getting really excited we're so close to being able to generate a report.

This is running now for the beta text varnish, e.g. on http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page . You can look at the headers and play with sending it various cookies there (and even see the X-Analytics output in the response headers as well, of course). If we're in agreement that it functions correctly there, we can push the VCL part to prod either "very soon", or Monday. Generally things like this on Friday afternoons are frowned on at the ops level.

Milimetric closed this task as Resolved.

Brandon, that looks great. I'm going to mark this as resolved and move it into our "ready to deploy" column. Let us know if anything goes wrong with the deployment Monday.

Change 196009 merged by BBlack:
Adding a Last-Access cookie to text and mobile requests

https://gerrit.wikimedia.org/r/196009

Should be live everywhere at this point.

BBlack moved this task from In Progress to Done on the Traffic board.Apr 27 2015, 1:54 PM