Page MenuHomePhabricator

deploy quicksurvey for editors on eswiki and arwiki (for Growth IP editors research)
Closed, ResolvedPublic

Description

TLDR: host Quicksurvey on Arabic and Spanish Wikipedias that links to a Google Forms survey in support of Growth team IP editing research


Background:

Hi all, this is my attempt at opening a ticket for getting an external Quicksurvey hosted on Arabic and Spanish Wikipedias. Ideally, I'd like to hit logged-in editors who have made 10+ edits and invite them to complete an external survey hosted on Google Forms. Please let me know where/how I can provide more information or clarification.


Acceptance criteria

Sampling rate: 0.1% (based on rates seen in previous Quicksurvey deployments)
Length: 2 weeks
Wikis: eswiki (Spanish Wikipedia) and arwiki (Arabic Wikipedia
Audience: minEdits: 10; anons: false (hoping to hit logged-in editors who have made a minimum of 10 edits)
Layout: External
Platform: desktop users
Privacy statements: Spanish and Arabic
External survey links (Google Forms): Spanish and Arabic


The language for the quicksurveys:


Ayúdenos a saber más sobre la edición anónima

Deseamos aprender mas sobre las impresiones de su comunidad hacía la edición anónima, también conocida como la edición de IP. Le rogamos que considere completar esta encuesta corta, y quizás le invitaremos a hablar sobre sus experiencias si indica su interés. ¡Gracias!

[button text]

Visitar encuesta [this button takes you to the survey https://docs.google.com/forms/d/e/1FAIpQLSdYmR-07CA4nKcIcwdPWaezkw3Q2TDFh27-Y4NX8dm-tVgblA/viewform?usp=sf_link]
No gracias [this is the 'no thanks' button]

[privacy policy text]

Encuesta está alojada por Google Forms. Véase la declaración de privacidad.


[Arabic should ideally be aligned right -->]

ساعدنا في معرفة المزيد عن التحرير المجهول

نحاول أن نتعلم أكثر عن انطباعات مجتمعك تجاه التحرير المجهول، أي التحرير من بروتوكول الإنترنت (تحرير الأي بي). الرجاء النظر في إكمال هذا الاستبيان القصير، وقد ندعوك إلى جلسة بحثية لمناقشة تجاربك إذا سجّلت اهتمامك بذلك داخل الاستبيان. شكراً!

[button text]

زيارة الاستبيان [this button takes you to the survey https://docs.google.com/forms/d/e/1FAIpQLScosxA-WTxhU8OlAq3zuml1ltC8z0tJVxKuwoY9eukIXsiHoQ/viewform?usp=sf_link]
لا شكراً [this button is the 'no thanks' button]

[privacy policy text]

الاستبيان يستضيفه Google Forms. أنظر بيان الخصوصية الخاص بالاستبيان.


English (for reference and clarification only)

Help us learn about anonymous editing on Wikipedia

We are trying to learn more about the impressions of your community about anonymous editing, also referred to as IP editing. Please consider filling out this short survey, and we might invite you to a research session to chat about your experiences if you indicate your interest. Thank you!

[button text]
Visit survey [takes you to survey link]
No thanks

[privacy policy text]

Survey is hosted by Google Forms. See the privacy policy [link]


  • Deploy first with a sample size of 0
  • Test using a query string parameter
  • Deploy to sampling rate defined above

Event Timeline

MRaishWMF updated the task description. (Show Details)

Change 736035 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/mediawiki-config@master] Prepare a QuickSurvey for Growth IP research

https://gerrit.wikimedia.org/r/736035

Change 736035 merged by jenkins-bot:

[operations/mediawiki-config@master] Prepare a QuickSurvey for Growth IP research

https://gerrit.wikimedia.org/r/736035

Mentioned in SAL (#wikimedia-operations) [2021-11-01T19:29:26Z] <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: cba805cb8aaa88d814bfff19b82e8f57ace4fafd: Prepare a QuickSurvey for Growth IP research (T294568) (duration: 00m 55s)

The Quicksurveys have been successfully tested on Arabic and Spanish. Everything links to the correct place and the text is appearing perfectly.

Change 736043 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/mediawiki-config@master] QuickSurveys: Show Growth IP editors survey to 0.1% of users

https://gerrit.wikimedia.org/r/736043

Change 736043 merged by jenkins-bot:

[operations/mediawiki-config@master] QuickSurveys: Show Growth IP editors survey to 0.1% of users

https://gerrit.wikimedia.org/r/736043

Mentioned in SAL (#wikimedia-operations) [2021-11-02T09:39:38Z] <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: b2594347041ae61ef88661bc0d5aa57fc501540d: QuickSurveys: Show Growth IP editors survey to 0.1% of users (T294568) (duration: 00m 57s)

In theory, should be live. I wasn't lucky enough to see the survey, but let's hope this works.

Change 736332 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/mediawiki-config@master] Growth IP research survey: Fix coverage

https://gerrit.wikimedia.org/r/736332

Change 736332 merged by jenkins-bot:

[operations/mediawiki-config@master] Growth IP research survey: Fix coverage

https://gerrit.wikimedia.org/r/736332

Mentioned in SAL (#wikimedia-operations) [2021-11-03T06:35:09Z] <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 34888b034e54ec35ca3b6745336fc0881e50c9b0: Growth IP research survey: Fix coverage (T294568) (duration: 01m 04s)

Note to self: find and document somewhere in repo markdown or ext manual page the query parameter for triggering the survey on demand.

We usually monitor surveys passively and look out for any impact. In this case, since it's (one of) the first surveys to go out using Vue.js, I suggest we run a few profiles in our synthetic infra and compare side-by-side with and without the survey on to get an updated baseline understanding of its cost/impact.

this is the URL you need to force it on: https://es.wikipedia.org/w/index.php?title=Spain&quicksurvey=external-survey-growth-ip-research
1% of traffic is quite low however, considering this is for logged in users with < 10 edit count. You probably want 100% coverage for anything that's logged in only.

Let me document it here so we know for future deploys:

MetricArticleWith quick survey
Total transfer size349.0 KB433.1 KB
Total JavaScript transfer size241.1KB324.7 KB
Total JavaScript content size (unpacked)832.4 KB1.1 MB
CPU long tasks812
CPU longest task duration196 ms535 ms

I did run the tests on my Mac so I wouldn't focus so much on the exact CPU timings or number of tasks, only that it looks like it could be some room for improvement to minimise the long tasks. It will be interesting when we collect long tasks in our RUM.

The interesting part is that the quick survey changes when the largest contentful paint (LCP) happens (that is one of the https://wikitech.wikimedia.org/wiki/Performance/GoogleWebVitals). For a small public that will not be a problem but if we deploy it to a large audience it will be a problem. The LCP changed from 1 second to 3 seconds and that's because when we run the survey, the survey is the largest painted content on the screen. Without the survey its the logo that is registered as the largest paint (red part is the largest painted area registered by Chrome):

largestContentfulPaint.png (921×2 px, 393 KB)

But with the survey it looks like this:

largestContentfulPaint.png (921×2 px, 468 KB)

this is the URL you need to force it on: https://es.wikipedia.org/w/index.php?title=Spain&quicksurvey=external-survey-growth-ip-research
1% of traffic is quite low however, considering this is for logged in users with < 10 edit count. You probably want 100% coverage for anything that's logged in only.

It's >10, not <10. At least it's _supposed to be_ :-).

@Jdlrobson that's a great point. I had hoped to hit editors with 10+ edits ( >10, as @Urbanecm noted). Thanks for flagging that the % of traffic may need to be drastically increased.

Hey @Peter what can we do to improve things there? One thing we are considering is inserting the survey outside the article body for the modern Vector experience to avoid reflows.

What about if the quick survey is off the page? For most wikis the quick surveys will be at the bottom of the first section which may not be in view.

How does this compare to the older OOUI version? The old version loaded significantly more JavaScript (T159738#7134776)

@Jdlrobson I'm thinking reflow/shift doesn't matter as long as we run the banner :) When we insert that, the rest of the content is moved (the red area). It's the same when run the survey.

layoutShift.jpg (921×2 px, 259 KB)

For most wikis the quick surveys will be at the bottom of the first section which may not be in view.

Is there a way I can try that so I can see what it looks like and test it out?

How does this compare to the older OOUI version? The old version loaded significantly more JavaScript

When I add up what you wrote in that task, it looks much better (the new version I mean).

@Urbanecm shouted out in the #humans-of-the-web Slack channel about why he wasn't seeing QuickSurveyInitiation events for this survey. I believe that this is a regression, which I've written up in T295683: Regression: QuickSurveys doesn't log QuickSurveysInitiation events.

Change 738974 had a related patch set uploaded (by Phuedx; author: Phuedx):

[operations/mediawiki-config@master] Growth IP research survey: Fix platforms

https://gerrit.wikimedia.org/r/738974

Change 738974 merged by jenkins-bot:

[operations/mediawiki-config@master] Growth IP research survey: Fix platforms

https://gerrit.wikimedia.org/r/738974

Mentioned in SAL (#wikimedia-operations) [2021-11-15T18:31:19Z] <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 1be2d3941530bbed54632dafb0b804d0ddf41299: Growth IP research survey: Fix platforms (T294568) (duration: 00m 55s)

After being open for some time, the Spanish survey has netted 37 responses and the Arabic has netted ~10, however both of these numbers are sufficient for my purposes and I am ok to disable these surveys at this point. I have the impression that .1% of editors and >10 edits was too restrictive, although it would be helpful to know how many times the surveys were viewed, which I gather is not currently possible. For the future I would plan to change both the % of editors and edit count, especially on smaller wikis. Thank you all!

Change 742268 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/mediawiki-config@master] Disable Growth IP research survey

https://gerrit.wikimedia.org/r/742268

I'll deploy the disabling change on Monday.

Change 742268 merged by jenkins-bot:

[operations/mediawiki-config@master] Disable Growth IP research survey

https://gerrit.wikimedia.org/r/742268

Mentioned in SAL (#wikimedia-operations) [2021-11-29T10:52:14Z] <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: d01652ec22f6cb3413b419a3c9b0a7a08d79960f: Disable Growth IP research survey (T294568) (duration: 00m 56s)

Survey was disabled again, so this is done now.