Story
As an analyst, I want to gauge the success of the A/B test performed on Italian and Russian wikipedias.

Description
Analyze the following and compare between Italian, Russian, and from the results of T139319: Analyze results of Hovercards A/B test - Hungarian:

Rate of hovercards being valuable v. disruptive

• Do the number of links hovered per page increase or decrease?
• Do they increase over time (by previewCountBucket)
• What is the rate of erroneous triggers?
• Is the usage of the back button reduced?

What % of people dislike hovercards?

• Percent of sessions where people disable hovercards?
• Percent changes over time
• Distribution of hovers before someone disables?

Impact on pageviews and overall engagement

• What is the impact on session length? (control vs test)
• Define page actions as:
  • Hovering a page
  • Opening new (in same tab or new window)
  • Compare page actions in control vs test group

• What is the impact on links or hovers clicked? (one possibility to quantify this: compare the average number of clicks per page view session - i.e. the sum of all three click actions counted in the schema - divided by the number of pageLoaded events, or the number of different pageTokens recorded for all events, within a certain timespan)

• What is the impact on session depth (with regard to page views)? (session depth defined as e.g. the number of different pageTokens recorded for that session ID within a certain timespan. Compare averages first

• What is the impact on the number of link interactions (as defined in the schema) per session?

• Same for the sum of page views (= session depth) and non-click link interactions (i.e. those that don't result in a new page view anyway), as a rough measure of overall engagement

Are they getting in the way of people who just want the next article?

• Calculate ratio of hovers where the reader opens the linked page after seeing the hovercard.

If 100% of the time, people continue onto article after seeing a hover, this is bad. We want to keep the ratio below 70%.
If 0% of the time people continue onto the article, that is also a sign that we are making it too hard. We want to keep this ratio above 10% of hovers where user continues onto article.

• Filter by previewCountBucket

Descriptive

• How long do people spend hovering before they click?
• How long do people spend hovering before they dismiss the hover?
• What is the change in the above metrics depending on learning curve? (by previewCountBucket)

Diagnostic

• Is it ever the case that “perceived” wait (amount of time before hover shows) exceeds “popup delay” (amount of time to trigger) by more than 200 ms? What % of the time?
• What is the frequency of error states (per hover)