Page MenuHomePhabricator

Percentage of users with DNT on
Closed, ResolvedPublic

Description

It will be really helpful for the ongoing discussions about DNT and other questions we run into, such as QAs in tasks like T127101 to know what percentage of our users have DNT on. It would be even better if we know this per population of logged-in and logged-out users, but this latter part is a nice-to-have.

Event Timeline

To truly answer this question, we'd probably want to look at a full week's worth of requests. But as a quick hack, I captured 300,000 requests on one of the Varnish servers and counted the percentage with a DNT: 1 request header, adapting my script from T124954#2009050.

#!/usr/bin/env bash
N="${1:-100000}"  # How many requests to process
printf "Capturing %'d requests...\n" "$N"
NEW="$(/usr/bin/varnishncsa -c -n frontend -mRxRequest:GET -F "%{DNT}i" | pv -eplSs "$N" | grep -c "1")"
echo "$NEW $N" | awk '{ printf "%.2f%% of requests have DNT: 1 header\n", (100*$1)/$2 }'

I ran it three times (analyzing 100,000 requests in each invocation) and got 11.59%, 11.42%, and 11.57%.

This is quite close to the global figures provided by CookieQ:

DNT2.png (461×734 px, 33 KB)

Thanks @ori. Agreed it would be nice to have snapshots taken over the course of a week. On Friday nights UTC traffic tends to be almost equally split between mobile an desktop, I suspect the % to be much higher during weekdays when desktop traffic dominates (and so should users on browsers with DNT on by default). Speaking of which, do we have a recent browser breakdown somewhere? cc @Tbayer

Thanks @ori. Agreed it would be nice to have snapshots taken over the course of a week. On Friday nights UTC traffic tends to be almost equally split between mobile an desktop, I suspect the % to be much higher during weekdays when desktop traffic dominates (and so should users on browsers with DNT on by default).

(Regarding "almost equally split", see also the last Readership metrics report with this chart. Saturday and Sunday have indeed a much higher mobile percentage than the other days of the week.)

Speaking of which, do we have a recent browser breakdown somewhere? cc @Tbayer

Not quite my focus area, but I happen to know that @mforns et al. are already generating internal datasets for this that should be available on stat1002, see https://wikitech.wikimedia.org/wiki/Analytics/Cluster/BrowserReports and
T126282 . (I say "should" because I'm right now getting an error when trying to retrieve an individual report following the instructions there.)

Not quite my focus area, but I happen to know that @mforns et al. are already generating internal datasets for this that should be available on stat1002, see https://wikitech.wikimedia.org/wiki/Analytics/Cluster/BrowserReports and T126282 . (I say "should" because I'm right now getting an error when trying to retrieve an individual report following the instructions there.)

Thanks @Tbayer for pointing @DarTar to that link. Yes, those are the current traffic reports broken down by browser (and OS). Regarding the error @Tbayer gets, I couldn't reproduce it, I did exactly what is in the doc you linked.

As mentioned, we Analytics are working on T126282 this quarter, which will allow visualizing those reports in a dashboard.

Not quite my focus area, but I happen to know that @mforns et al. are already generating internal datasets for this that should be available on stat1002, see https://wikitech.wikimedia.org/wiki/Analytics/Cluster/BrowserReports and T126282 . (I say "should" because I'm right now getting an error when trying to retrieve an individual report following the instructions there.)

Thanks @Tbayer for pointing @DarTar to that link. Yes, those are the current traffic reports broken down by browser (and OS). Regarding the error @Tbayer gets, I couldn't reproduce it, I did exactly what is in the doc you linked.

(Not to go too far offtopic, but I just tried again and this time it worked for me too.)