Once we make the feature default on Chinese, Greek and Catalan wikipedias we'll need to have a sampling rate lesser than 1.
Page views per month (from http://stats.wikimedia.org/EN/Sitemap.htm)
| Language | Page views per month | Link |
|---|---|---|
| Chinese | 601,992,893 | http://stats.wikimedia.org/EN/SummaryZH.htm |
| Greek | 30,853,193 | http://stats.wikimedia.org/EN/SummaryEL.htm |
| Catalan | 38,498,078 | http://stats.wikimedia.org/EN/SummaryCA.htm |
We also have ~22,620 people that have the feature activated on English Wikipedia.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Declined | None | T88882 Move Hovercards to OOjs and OOjs UI | |||
| Resolved | None | T92555 Turn off 2 month test release of Hovercards on Catalan, and Greek Wikipedias on 2015-06-18 | |||
| Resolved | Tnegrin | T88164 Make hovercards default for Catalan and Greek Wikipedias for 2 months starting 2015-04-15 | |||
| Resolved | Prtksxna | T88173 Decide on sampling rate for EventLogging | |||
| Resolved | leila | T88166 Finalize on schema for the new instrumentation |
@Prtksxna I guess we just need to know events per pageview to calculate how many events per second will we be seeing
How do you suggest we go about that? Currently our events only log sessionId and has no page specific information. Is there a way to estimate this?
Let's see:
What questions do you want to answer with the schema?
Is every user to which a page is presented going to see a hovercard?
If they hover over a link, then yes!
Does the hovercard view trigger 1 event or many?
One per link hovered or clicked on.
What questions do you want to answer with the schema?
That is a discussion we (@Jaredzimmerman-WMF @DarTar and @leila) have had a couple of times. I'll update T88164 soon to reflect our discussion.
If you are potentially login several events per pageview and (adding all your pageviews above) you have about 250 pageviews per second I will try to log no higher than 2 event per second or so, that would mean 1/100 (rounding big time) -if events were sent once per pageview- but if you think that users might click, say, 5 times per page to see hovercards I would divide this by 5 so 1/500.
That would give you about 200k of events every day, you can run the experiment as long as needed. Note that we are not only concern about a reasonable amount of throughput that EL can sustain but also about getting an amount of data that we can look at, if we log too much we have to throw data away as queries will not run.
Thanks @Nuria! We'll use that formula to come to a number. In the case that we collect too many or too few events in the first week, we can revisit and tweak the number. Is that alright?
@leila, how many events per day do you think we need to do a successful analysis?
@Jaredzimmerman-WMF, do you have as estimate on how many links people hover on on a single page view? Can you extrapolate on the basis of your own usage?
@Prtksxna, it depends what questions you want to answer at the end of the three months. Let's touch-base on this when we meet this week. We can update this thread afterwards.
I think you stated the question quite clearly on Trello. I had copied it to the description of T88166 too.
@Nuria I think we have to look to someone with more familiarity in EventLogging to know what sampling rate would be safe, and not cause issues. Who is in the best place to make that recommendation based on the number of active PV on those wikis?
I already commented on a ticket about this, @Prtksxna answered. Let me see if i can find the ticket.
Ah, it was this ticket!
"If you are potentially login several events per pageview and (adding all your pageviews above) you have about 250 pageviews per second I will try to log no higher than 2 event per second or so, that would mean 1/100 (rounding big time) -if events were sent once per pageview- but if you think that users might click, say, 5 times per page to see hovercards I would divide this by 5 so 1/500.
That would give you about 200k of events every day, you can run the experiment as long as needed. Note that we are not only concern about a reasonable amount of throughput that EL can sustain but also about getting an amount of data that we can look at, if we log too much we have to throw data away as queries will not run. "
Is this enough?
@leila do you think that gives you enough data for a high level of confidence when comparing enabled vs disabled user events (obviously we don't know what the opt out rate will be) gergo said they used 1/1000 for mediaviewer
The rate is not so important as the "amount" of data you will have at the end to do your analysis. Each feature has different usage and thus different rates of sampling.
@Jaredzimmerman-WMF, how you sample depends on what you want to do with the data.
if you want to be able to study how users use Hovercards in sessions, then you should sample sessions. if you want to see how users interact with Hovercards over the course of 90 days, then you should have userTokens with expiry date set to >90 days and you should sample users (something which is not in the schema as of now). If you go with the former, we should know something about the probability of each action or otherwise we can't use the sampled data to reconstruct the actual number of each action type. If this schema is already collecting data in Beta, we can use that data to compute the probabilities.
The biggest question for sampling and schema itself is: what do you want to be able to say after 90 days? What should happen that you call the feature successful and call for pushing it to all wikis? The sampling and schema should reflect that goal.
Seems like an easy place to start would be couting hoovercards displays per pageview, that would be easier than doing sessions (mediawiki only has true sessions for logged in users).
We can radomly sample pageviews and get that data.
Like:
if rand(1,10) > 5 // sample 50% send all hoovercard impressions for this page_id also send whether user was logged in.
Gather data over a long enough timeperiod that we think we have data for the majority of the user base (anonymous or otherwise)
If you want to get session data things become a bit more complicated as you need to have a token that persists across page transitions.
More detail:
So long as we can compare hovercard views to page views…
*Jared Zimmerman * \\ Director of User Experience \\ Wikimedia Foundation
M +1 415 609 4043 \\ @Jaredzimmerman http://loo.ms/g0
https://gerrit.wikimedia.org/r/196190 adds a sampling rate of 1.
@leila Do we have a number now? I'll update the patch accordingly.
@Prtksxna, I just reviewed 11528589. Can you specify what events will be logged when the Hovercard is turned off by the user?
As per @leila: Sampling rate has been decided as 1 out 10 events.
We'll revisit this number if needed