In order to run an a/b test on Italian Wikipedia, we will need to make sure there are no objections from the community.
|Resolved||Dereckson||T68374 Enable Hovercards on se.wikimedia.org (Swedish chapter wiki)|
|Resolved||Jdlrobson||T70860 [GOAL] Graduate Page Previews feature (Popups extension) out of Beta Feature|
|Resolved||ovasileva||T131366 Measure impact of HoverCards on Central Notice interaction|
|Invalid||None||T141775 Initiate Hovercards A/B Test 3 with Central Notice: October 2016|
|Resolved||CKoerner_WMF||T143170 Start discussion with Italian community about running a/b test for hovercards in October|
A very rough draft is here: https://meta.wikimedia.org/wiki/User:CKoerner_(WMF)/Hovercard_A/B_test_Italian_Wikipedia_post
As we're waiting on the other A/B tests first the language and timing of this test may change.
Spoke with @Jseddon from Fundraising to discuss a rough timeline for community discussion.
Fundraising potentially will run their campaign from 4 Oct to 4 Nov.
We will have a draft post prepared by the end of August or early September. We'll share it with Wikimedia Italia for review and feedback in early September. Then post the message two weeks in advance of the campaign on Sept 16th to the Italian Village Pump. That should give community members and Fundraising/Reading folks time to discuss any concerns or issues.
@Elitre would you be so kind as to review this plan and our draft once it is complete?
@CKoerner_WMF last time around we intentionally had Hovercards up and running for a while in head-to-head A/B test mode before the campaign started, so that users in the test bucket could get accustomed to it before the campaign started. @Jseddon, @Elitre et al, do you think it would be possible to start the campaign a week and a half later or so, such that we could enable the A/B test of Hovecards for a while to get users accustomed to the feature ahead of the banners?
@dr0ptp4kt, this test affects logged-out users only, right? If enabling Hovercards in advance (for a subset of users) helps collecting more accurate and reliable data, then this shouldn't really be a question I guess? (If you want to "compare" with the Hungarian test for example, shouldn't the setting be identical?)
Hi @Elitre, yes, that helped :) Yes, we're trying to recreate a similar setting as Hungarian. Also, we're hoping not to overwhelm users with the campaign and hovercards at the same time. So if the fundraising campaign starts 4th Oct, we would want to launch the hovercard test the week of 19th of sept and post to the village pumps the week of the 4th (next week)
Alternatively, if it is possible to postpone the fundraising test to the following week, the 11th of October, we could launch the hovercard a/b test the week of the 26th and post the message on the 16th.
On the announcement side of things, a few suggestions.
- mention why it.wp was chosen. Mention they can also get additional support in their language if necessary because of me.
- shortly explain how the test works: "on day X, we do this and this is what people will see [...]. On day Y, we [...]. On day Z, everything goes back to normal".
- remind HC is still a beta feature when you say that nothing changes for logged-in people.
- Post announcement Monday, Sept 5th
- Start A/B test on September 19th
- Start Fundraising campaign on October 4th
I also updated the announcement with feedback from folks. I need some help filling in the blanks on:
- the percentages of people who will be included in the A/B test
- end dates for both the hovercards test and fundraising campaign.