Page MenuHomePhabricator

Share results of Hovercard A/B Test on Hungarian Wikipedia
Closed, ResolvedPublic

Description

After the analysis of the results of the Hungarian A/B test are complete, present the following results to the group:

  1. Qualitative hovercards results
  2. Fundraising test data (from the period when a/b test was happening)
  3. A/B test data intrumentation issues resolvement

Note: due to instrumentation issues https://phabricator.wikimedia.org/T141922 , the data prior to 8/30 (or further) is not an appropriate representation of trends.

I'll help to write and request translations of the results. This will help communicate what we found and will be used in further work around promoting Hovercards out of Beta status.

See also: T139319

Event Timeline

Qgil triaged this task as Medium priority.Jul 6 2016, 11:26 AM
Qgil added a project: Page-Previews.
Qgil subscribed.

Assuming normal priority, better than no priority.

What needs to happen in order to complete this task in September? Are there any blockers?

I think I just need to post the results to a mailing list or two and make sure it's accessible from pages like https://www.mediawiki.org/wiki/Beta_Features/Hovercards (which it is).

@JKatzWMF is that right?

https://www.mediawiki.org/wiki/Extension:Popups/Fundraising_test_1

@CKoerner_WMF yes, but there are 2 components to the a/b test. 1. is fundraising data which is done, but 2. is the engagement data we will get from the a/b test, but due to instrumentation issues that data is not yet available (T141922).

As a side note, when the qualitative analysis is complete, we will want to share that too--it is expected within a week or so. Should I post that as a separate ticket or edit this one's description to be more inclusive.

@ovasileva to carry fwd.

@CKoerner_WMF
a note: due to the some of the instrumentation issues and the tasks that followed it, we are unsure on whether we can retrieve the correct data from the beginning of the a/b test, but can present the data since the fix (early this week)

Let's modify this ticket to reflect reality. We'll most likely want to present all the results together as a cohesive effort.

ovasileva updated the task description. (Show Details)

@CKoerner_WMF - I edited the description to reflect the current situation. I think 2 and 3 are a cohesive effort, as a part of the a/b test overall, and 1 can fit nicely as the qualitative analog

Thank you, the situation is clearer now. Since this is an individual quarterly goal for Community-Relations-Support, I am still interested in knowing whether there is a chance to complete this task by the end of this quarter.

Olga and I talked this morning. The answer is 50%

Qualitative - Yes. Sherah is finishing up a presentation that covers qualitative hovercard research to be shared in the next week or so.

Quantitative - No. There's outstanding work to review the results from the A/B that probably won't be done by the end of this quarter. See T139319 for more details.

The results of the test on Hungarian Wikipedia have been published: https://www.mediawiki.org/wiki/Beta_Features/Hovercards/2016_A/B_Tests

Next steps are asking for help in translating a short message to post to the Hungarian Village Pump (technical).

CKoerner_WMF closed this task as Resolved.EditedDec 12 2016, 9:38 PM

I asked the Hungarian Wikipedia if they would like to enable Hovercards as part of the team's rollout plan. With this request I point out the results of the A/B test. With this annoucncment and the previous mailing list announcement I'd consider this task accomplished.

I'm going to re-open this. Based on a Google Translate view, it looks like there wasn't really any opportunity for them to comment on enablement of Hovercards. All I saw were comments indicating they couldn't comment on it because (at that time) testing of Hovercards was disabled.

If there was a newer discussion that I missed, then this can certainly be reclosed.

This task was created to share the results. The results were shared in the ask (which revealed the beta feature wasn't available ).

With T156290#3023750 the beta feature is now available again. I just created T158089 to address the feedback there.

Thanks @Alsee, I had it on my list to do, but was waiting for the fix to be in place. Off to the Hungarian Wikipedia! :)