Change Details

Analyze the data from the experiment launched on huwiki last month (see T134778 and [[https://meta.wikimedia.org/wiki/Schema:Popups| Schema:Popups]]) regarding the following aspects: **Rate of hovercards being valuable v. disruptive** [x] Do the number of links hovered per page increase or decrease? * --> [[https://phabricator.wikimedia.org/T139319#2476014| result]] (with data caveat) [x] What is the rate of erroneous triggers? (Estimated by taking the dwelledbutAbandoned time length in control group and seeing how many are above the trigger time. Apply # of these per page to the number of hovers in experimental.) * --> [[https://phabricator.wikimedia.org/T139319#2475811 | result]] (with data caveat) **What % of people dislike hovercards?** [x] % of sessions where people disable hovercards? [x] ^ over time [x] Distribution of # of hovers before someone disables? * --> [[https://phabricator.wikimedia.org/T139319#2467334 |result]] (with data caveat) **Impact on pageviews and overall engagement** (and on fundraising - being tracked at T131366) [x] What is the impact on links or hovers clicked of being in the experimental group? //(one possibility to quantify this: compare the average number of clicks per page view session - i.e. the sum of all three click actions counted in the schema - divided by the number of pageLoaded events, or the number of different pageTokens recorded for all events, within a certain timespan)// * --> [[https://phabricator.wikimedia.org/T139319#2476096 | result]] (with data caveat) [x] What is the impact on session depth (with regard to page views) of being in the experimental group? //(session depth defined as e.g. the number of different pageTokens recorded for that session ID within a certain timespan. Compare averages first// * --> [[https://phabricator.wikimedia.org/T139319#2476096 |result]] (with data caveat) about averages; perhaps compare medians too later. [x] What is the impact on the number of link interactions (as defined in the schema) per session? * --> [[https://phabricator.wikimedia.org/T139319#2476191 |result]] (with data caveat) [x] Same for the sum of page views (= session depth) and non-click link interactions (i.e. those that don't result in a new page view anyway), as a rough measure of overall engagement * --> [[https://phabricator.wikimedia.org/T139319#2440480 | result]] (with data caveat) **Are they getting in the way of people who just want the next article?** [x] Calculate ratio of hovers where the reader opens the linked page after seeing the hovercard. If 100% of the time, people continue onto article after seeing a hover, this is bad. We want to keep the ratio below 70%. If 0% of the time people continue onto the article, that is also a sign that we are making it too hard. We want to keep this ratio above 10% of hovers where user continues onto article. * --> [[https://phabricator.wikimedia.org/T139319#2475704 | result]] **Descriptive** [x] How long do people spend hovering before they click? [x] How long do people spend hovering before they dismiss the hover? (compare also https://phabricator.wikimedia.org/T131315#2192044 for last year's instrumentation) * --> [[https://phabricator.wikimedia.org/T139319#2475704 | results]] **Diagnostic** [x] Is it ever the case that “perceived” wait (amount of time before hover shows) exceeds “popup delay” (amount of time to trigger) by more than 200 ms? What % of the time? * ---> [[https://phabricator.wikimedia.org/T139319#2475143 | result]] (+[[https://phabricator.wikimedia.org/T139319#2510867 |PS]]) [x] Frequency of error states (per hover) (comment from @dr0ptp4kt: Note well T137059. It's probably necessary to define this for a ratio above 300ms or something like that until the bugfix is built and deployed.) * --> [[https://phabricator.wikimedia.org/T139319#2475936 | result]] ----- (adapted from @JkatzWMF's notes, see also https://www.mediawiki.org/wiki/Beta_Features/Hovercards#2016_a.2Fb_tests ) Overall comment from @dr0ptp4kt: Beware T137203 in analysis of authenticated Hovercards OFF users. Probably easiest to just exclude those funnels. ----- Followup questions (apart from instrumentation verification/debugging work like T140485, T139319#2507540 , ... ): [ ] Compare time spent per session (suitably defined, cf. [[https://phabricator.wikimedia.org/T139319#2449474 | notes]])

Analyze the data from the experiment launched on huwiki last month (see T134778 and [[https://meta.wikimedia.org/wiki/Schema:Popups| Schema:Popups]]) regarding the following aspects: **Rate of hovercards being valuable v. disruptive** [x] Do the number of links hovered per page increase or decrease? * --> [[https://phabricator.wikimedia.org/T139319#2476014| result]] (with data caveat) [x] What is the rate of erroneous triggers? (Estimated by taking the dwelledbutAbandoned time length in control group and seeing how many are above the trigger time. Apply # of these per page to the number of hovers in experimental.) * --> [[https://phabricator.wikimedia.org/T139319#2475811 | result]] (with data caveat) **What % of people dislike hovercards?** [x] % of sessions where people disable hovercards? [x] ^ over time [x] Distribution of # of hovers before someone disables? * --> [[https://phabricator.wikimedia.org/T139319#2467334 |result]] (with data caveat) **Impact on pageviews and overall engagement** (and on fundraising - being tracked at T131366) [x] What is the impact on links or hovers clicked of being in the experimental group? //(one possibility to quantify this: compare the average number of clicks per page view session - i.e. the sum of all three click actions counted in the schema - divided by the number of pageLoaded events, or the number of different pageTokens recorded for all events, within a certain timespan)// * --> [[https://phabricator.wikimedia.org/T139319#2476096 | result]] (with data caveat) [x] What is the impact on session depth (with regard to page views) of being in the experimental group? //(session depth defined as e.g. the number of different pageTokens recorded for that session ID within a certain timespan. Compare averages first// * --> [[https://phabricator.wikimedia.org/T139319#2476096 |result]] (with data caveat) about averages; perhaps compare medians too later. [x] What is the impact on the number of link interactions (as defined in the schema) per session? * --> [[https://phabricator.wikimedia.org/T139319#2476191 |result]] (with data caveat) [x] Same for the sum of page views (= session depth) and non-click link interactions (i.e. those that don't result in a new page view anyway), as a rough measure of overall engagement * --> [[https://phabricator.wikimedia.org/T139319#2440480 | result]] (with data caveat) **Are they getting in the way of people who just want the next article?** [x] Calculate ratio of hovers where the reader opens the linked page after seeing the hovercard. If 100% of the time, people continue onto article after seeing a hover, this is bad. We want to keep the ratio below 70%. If 0% of the time people continue onto the article, that is also a sign that we are making it too hard. We want to keep this ratio above 10% of hovers where user continues onto article. * --> [[https://phabricator.wikimedia.org/T139319#2475704 | result]] **Descriptive** [x] How long do people spend hovering before they click? [x] How long do people spend hovering before they dismiss the hover? (compare also https://phabricator.wikimedia.org/T131315#2192044 for last year's instrumentation) * --> [[https://phabricator.wikimedia.org/T139319#2476368 | results]] **Diagnostic** [x] Is it ever the case that “perceived” wait (amount of time before hover shows) exceeds “popup delay” (amount of time to trigger) by more than 200 ms? What % of the time? * ---> [[https://phabricator.wikimedia.org/T139319#2475143 | result]] (+[[https://phabricator.wikimedia.org/T139319#2510867 |PS]]) [x] Frequency of error states (per hover) (comment from @dr0ptp4kt: Note well T137059. It's probably necessary to define this for a ratio above 300ms or something like that until the bugfix is built and deployed.) * --> [[https://phabricator.wikimedia.org/T139319#2475936 | result]] ----- (adapted from @JkatzWMF's notes, see also https://www.mediawiki.org/wiki/Beta_Features/Hovercards#2016_a.2Fb_tests ) Overall comment from @dr0ptp4kt: Beware T137203 in analysis of authenticated Hovercards OFF users. Probably easiest to just exclude those funnels. ----- Followup questions (apart from instrumentation verification/debugging work like T140485, T139319#2507540 , ... ): [ ] Compare time spent per session (suitably defined, cf. [[https://phabricator.wikimedia.org/T139319#2449474 | notes]])

Analyze the data from the experiment launched on huwiki last month (see T134778 and [[https://meta.wikimedia.org/wiki/Schema:Popups| Schema:Popups]]) regarding the following aspects: **Rate of hovercards being valuable v. disruptive** [x] Do the number of links hovered per page increase or decrease? * --> [[https://phabricator.wikimedia.org/T139319#2476014| result]] (with data caveat) [x] What is the rate of erroneous triggers? (Estimated by taking the dwelledbutAbandoned time length in control group and seeing how many are above the trigger time. Apply # of these per page to the number of hovers in experimental.) * --> [[https://phabricator.wikimedia.org/T139319#2475811 | result]] (with data caveat) **What % of people dislike hovercards?** [x] % of sessions where people disable hovercards? [x] ^ over time [x] Distribution of # of hovers before someone disables? * --> [[https://phabricator.wikimedia.org/T139319#2467334 |result]] (with data caveat) **Impact on pageviews and overall engagement** (and on fundraising - being tracked at T131366) [x] What is the impact on links or hovers clicked of being in the experimental group? //(one possibility to quantify this: compare the average number of clicks per page view session - i.e. the sum of all three click actions counted in the schema - divided by the number of pageLoaded events, or the number of different pageTokens recorded for all events, within a certain timespan)// * --> [[https://phabricator.wikimedia.org/T139319#2476096 | result]] (with data caveat) [x] What is the impact on session depth (with regard to page views) of being in the experimental group? //(session depth defined as e.g. the number of different pageTokens recorded for that session ID within a certain timespan. Compare averages first// * --> [[https://phabricator.wikimedia.org/T139319#2476096 |result]] (with data caveat) about averages; perhaps compare medians too later. [x] What is the impact on the number of link interactions (as defined in the schema) per session? * --> [[https://phabricator.wikimedia.org/T139319#2476191 |result]] (with data caveat) [x] Same for the sum of page views (= session depth) and non-click link interactions (i.e. those that don't result in a new page view anyway), as a rough measure of overall engagement * --> [[https://phabricator.wikimedia.org/T139319#2440480 | result]] (with data caveat) **Are they getting in the way of people who just want the next article?** [x] Calculate ratio of hovers where the reader opens the linked page after seeing the hovercard. If 100% of the time, people continue onto article after seeing a hover, this is bad. We want to keep the ratio below 70%. If 0% of the time people continue onto the article, that is also a sign that we are making it too hard. We want to keep this ratio above 10% of hovers where user continues onto article. * --> [[https://phabricator.wikimedia.org/T139319#2475704 | result]] **Descriptive** [x] How long do people spend hovering before they click? [x] How long do people spend hovering before they dismiss the hover? (compare also https://phabricator.wikimedia.org/T131315#2192044 for last year's instrumentation) * --> [[https://phabricator.wikimedia.org/T139319#24757046368 | results]] **Diagnostic** [x] Is it ever the case that “perceived” wait (amount of time before hover shows) exceeds “popup delay” (amount of time to trigger) by more than 200 ms? What % of the time? * ---> [[https://phabricator.wikimedia.org/T139319#2475143 | result]] (+[[https://phabricator.wikimedia.org/T139319#2510867 |PS]]) [x] Frequency of error states (per hover) (comment from @dr0ptp4kt: Note well T137059. It's probably necessary to define this for a ratio above 300ms or something like that until the bugfix is built and deployed.) * --> [[https://phabricator.wikimedia.org/T139319#2475936 | result]] ----- (adapted from @JkatzWMF's notes, see also https://www.mediawiki.org/wiki/Beta_Features/Hovercards#2016_a.2Fb_tests ) Overall comment from @dr0ptp4kt: Beware T137203 in analysis of authenticated Hovercards OFF users. Probably easiest to just exclude those funnels. ----- Followup questions (apart from instrumentation verification/debugging work like T140485, T139319#2507540 , ... ): [ ] Compare time spent per session (suitably defined, cf. [[https://phabricator.wikimedia.org/T139319#2449474 | notes]])