Page MenuHomePhabricator

Identify two wikis to run a research study and get approval from their respective communities
Closed, ResolvedPublic


I would like to conduct this research study, hopefully next quarter: T187299: User-perceived page load performance study

It would require showing a survey to a low sample of users. I would probably leverage QuickSurveys for that, to avoid reinventing the wheel.

Ideally, this wouldn't be a one-off, but something we ask users (again, with very low sampling) continuously, to gauge whether or not people's perception of site performance evolves with time.

I would like for this to run on two wikis, to verify whether there are cultural differences or not in the results.

The wikis needs to have a fair amount of traffic, but they don't have the biggest. I.e. it's not necessary to "sell" this to enwiki for example - where from what I've heard there's an ongoing debate about research done directly on users - as the results would be equally valid on a smaller wiki.

I need help figuring out which wikis would be most likely to help us by letting us run this study, and validating community approval using whichever process those particular wikis use for that.

Event Timeline

Thanks. I'll let you know something ASAP.

Gilles, I think this is possible. Do you have any requirements for the wikis? For example, do they both need to be Wikipdias? Do you want the two wikis to be on different continents or noticeably different cultures?

Some possibilities that have occurred to me in the last two minutes:

  • Catalan Wikipedia, because they have a small, awesome community that understands the value of tech
  • French Wikipedia, because they are large and you can be certain of ideal translations
  • English Wikivoyage, because it's not a Wikipedia
  • Pairing Czech and Korean, because these communities are the communities for the New-Editor-Experiences project

(The enwiki policy was updated recently, but it is not intended to interfere with projects like yours.)

QuickSurveys is quite limited (which is why I've proposed a fully featured microsurvey tool at T89970: Enable microsurveys for long-term tracking of editing experience ), but QuickSurveys might be sufficient for your purposes. Have you used it before?

It would be desirable if we have at least 2 wikipedias, but it could also be interesting to have a non-wikipedia in addition to that.

I haven't used QuickSurveys yet, I suspect it will require some adaption. The main value I see is not reinventing the wheel in terms of displaying a survey and recording responses, but the "hooking up" will have to be different for this project.

What's the maximum number of wikis you want to consider?

I understand that QuickSurveys can only be displayed on article pages, but that might be ideal for your purposes.

Do you need to collect any information beyond what the user answers (e.g., logged-in vs logged-out status? geolocation?).

A lot more information about the performance of the page and anonymous information about the browser, etc., which we already collect with the NavigationTiming extension. Essentially the survey results need to be tied to the performance data (and performance measured when we show the survey - or rather the survey shown on a subsample of pages where we measure performance). It's fine if they go to different EventLogging schemas (I presume QuickSurveys is set up with a default schema). I'll probably modify NavigationTiming rather than QuickSurveys to achieve that linking.

Article-only is fine. Preferably this would be displayed as an overlay, not in-article, as it's about the whole page.

What's the next step? Asking the community of these 3 wikis (cawiki, frwiki, enwikivoyage) if they would be willing to let us run the study on their wiki?

I think so. How soon do you think this could be ready to start? (No need to talk to people in March, if the study will happen in August...)

Should be ready to go in a few weeks, the code is up for review already (based on QuickSurveys)

Posting notices about two weeks before the start date seems to be the sweet spot. Two weeks is early enough that we can change plans if there are unexpected complications (e.g., "Not then! That's when ____ happens, too!"), but not so early that everyone will have forgotten about it by the time it finally starts.

Have you settled on the text for the first question?

Is it possible to get a screenshot of the survey (ideally, in each language)? Every now and again, people see a survey question and worry that it's some kind of malware, so if we can show them a picture, it can be reassuring to them.

Yes, I've settled on the (initial, at least) English and French wording of the survey. I'll make screenshots. I'll need to pester a Catalan speaker for translations.

Those screenshots are exactly what I want. Could you upload them to Commons? (I could do it for you, if you prefer.)

We are now unblocked by legal, here's the next wording below the survey:

Capture d'écran 2018-04-30 10.09.02.png (478×483 px, 46 KB)

Linking to:

I'm going to translate the "for more information..." sentence on translatewiki for French. And if it's not translated to Catalan in a few days, I guess we can ping someone.

That sounds good. Do you have a date for when this will actually start? Are we ready to post the notes this week?

It can start whenever we want. Basically we can say 2 weeks after you post the notes, since I seem to recall you said that was the ideal timeframe.

Elitre triaged this task as Medium priority.May 3 2018, 3:53 PM

Argh, I missed this message last week! Yes, now's fine. I'll post to enwikivoyage and cawiki now, and you can post to frwiki whenever you want. If you need to change the dates, I'll update those communities.

I also posted the English screenshot at enwiki's VPT, in case someone takes questions there. I don't have a Catalan screenshot. Do we still need that translation?

Here's an updated screenshot for Catalan:

Capture d'écran 2018-05-11 12.23.14.png (399×425 px, 49 KB)

I see that the special mention at the bottom that the Legal team requested hasn't been translated on translatewiki yet, though. I've just checked and I didn't know that Catalan had such a huge backlog of untranslated messages for MediaWiki (close to 17000 messages!).

With the hackathon happening in Barcelona next weekend, I'll just find a native speaker there to translate that sentence.

("Not sure" also doesn't sound very Catalan :p nonetheless, uploaded.)

Haha, didn't spot that. That shows how outdated the MediaWiki translations are, since that particular message has been in there for years.

Hi, I saw the announcement on English Wikipedia’s technical village pump and thought that I could ask: is there a way for other projects to chime in on this? Engineers in Russian Wikipedia would’ve greatly appreciated such research being conducted for our readers. I can translate a note to wider community about this if you would give this a green light.

If you would need localised strings in Russian:

  • Did this page load fast enough? – Достаточно ли быстро загрузилась эта страница?
  • For more information, see this survey’s privacy statement – Чтобы узнать больше, см. [link политику конфиденциальности] данного опроса.

Hi @stjn yes, absolutely, we can run this study on more projects! Thank you for your help, I've added your translations to the WikimediaMessages extension. This is what the survey looks like now in Russian when I test it locally:

Capture d'écran 2018-05-14 09.29.56.png (247×333 px, 21 KB)

If you could translate the notice we've posted on English wikivoyage: and post it on the equivalent board for Russian Wikipedia, that would be great! With a survey start date 2 weeks after the notice is posted.

Change 432955 had a related patch set uploaded (by Gilles; owner: Gilles):
[mediawiki/extensions/WikimediaMessages@master] Add russian translations for performance perception survey

Change 432955 merged by jenkins-bot:
[mediawiki/extensions/WikimediaMessages@master] Add russian translations for performance perception survey

If you could translate the notice we've posted on English wikivoyage: and post it on the equivalent board for Russian Wikipedia, that would be great! With a survey start date 2 weeks after the notice is posted.

I will keep you posted about any concerns that might come up in the discussion.

The updated Catalan screenshot:

Capture d’écran 2018-05-19 à 10.39.42.png (458×642 px, 40 KB)

Change 433929 had a related patch set uploaded (by Gilles; owner: Gilles):
[mediawiki/extensions/WikimediaMessages@master] Add Catalan translation for performance perception survey

Change 433930 had a related patch set uploaded (by Gilles; owner: Gilles):
[mediawiki/extensions/QuickSurveys@master] Add Catalan translation

Change 433930 merged by jenkins-bot:
[mediawiki/extensions/QuickSurveys@master] Add Catalan translation

Change 433929 merged by jenkins-bot:
[mediawiki/extensions/WikimediaMessages@master] Add Catalan translation for performance perception survey

Hi. The QuickSurveys extension and the custom WikimediaMessages we use are avalaible for translation at as far as I can see. Could you please post the translations there instead and let the bots update the code? Thanks!


@Whatamidoing-WMF are we good to go to start the survey on cawiki and enwikivoyage tomorrow as planned?

I have seen no objections at cawiki or enwikivoyage, andédia:Le_Bistro/11_mai_2018#Micro-sondage_sur_la_performance_d'affichage_des_pages has had no comments since the first day. I think we are ready to go!

The survey has been launched on cawiki and enwikivoyage earlier today. Looking at the data, I discovered that QuickSurveys actually work on mobile, so this is being displayed on the mobile sites of these wikis as well.

So far we've had 30ish impressions of the survey, but nobody has responded yet. I'll check after the weekend if it's worth increasing the sampling rate a little to get more responses. With frwiki's much larger traffic I think we'll get a decent idea of the impression/response ratio to expect.

@stjn what's the consensus on the ruwiki discussion? Can I start the survey?

The survey is live on ruwiki. Thanks a lot to everyone who helped with this! When we start getting meaningful results I'll write about it on our team's phabricator blog:

As for preliminary results and discussion about the survey data, it will happen on the parent task: T187299

If we decide to update the wording of the survey along the way, I'll create a new task for that purpose. I think we can close this one now.

I'm happy to see this work underway. @stjn, thank you for recommending that it be run at the Russian Wikipedia as well. I appreciate your help in expanding the project's reach.