Page MenuHomePhabricator

QuickSurveys: Add survey on article page when coming from a wiki search
Closed, DeclinedPublic3 Estimate Story Points

Description

In T117831 we dismissed all the blockers for having a survey on the article page when we come from a wiki search.

This ticket is to track the implementation of the quick survey:

1/ Make sure Quick Surveys are ready to use
2/ Writing the survey definition
3/ Add a query param when we come from a wiki search (&quicksurvey=name_of_the_survey)
4/ Clean up the page url as soon as user lands on the page

Details

Related Gerrit Patches:
mediawiki/extensions/WikimediaEvents : master[Temp] Add a survey on article pages when users come from SERP

Event Timeline

JGirault claimed this task.
JGirault raised the priority of this task from to Needs Triage.
JGirault updated the task description. (Show Details)
JGirault added a subscriber: JGirault.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptNov 17 2015, 12:39 AM
This comment was removed by JGirault.
JGirault updated the task description. (Show Details)Nov 17 2015, 12:44 AM
JGirault set Security to None.

Change 254089 had a related patch set uploaded (by JGirault):
[Temp] Add a survey on article pages when users come from SERP

https://gerrit.wikimedia.org/r/254089

I moved this ticket to Pending Review.
Right now, only the part that takes care of the query parameter is done.

The configuration array for the survey is:

array(
    'name' => 'search-satisfaction',
    'type' => 'internal',
    'question' => 'survey-search-satisfaction-question',
    'answers' => array(
        'survey-search-satisfaction-yes',
        'survey-search-satisfaction-no',
    ),
    'description' => 'survey-search-satisfaction-desc',
    'enabled' => true,
    'coverage' => 0.0,
    'platform' => array(
        'desktop' => array( 'stable' ),
        'mobile' => array( 'stable' ),
    ),
),

It will have to be put into this array for production: https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/InitialiseSettings.php#L15816-L15833

Meanwhile, you can enable all this locally by:

wfLoadExtension( 'QuickSurveys' );

$wgQuickSurveysConfig = array(
    array(
        'name' => 'search-satisfaction',
        'type' => 'internal',
        'question' => 'survey-search-satisfaction-question',

        'answers' => array(
            'survey-search-satisfaction-yes',
            'survey-search-satisfaction-no',
        ),
        'description' => 'description',
        'enabled' => true,
        'coverage' => 0.0,
        'platform' => array(
            'desktop' => array( 'stable', 'beta', 'alpha' ),
            'mobile' => array( 'stable', 'beta', 'alpha' ),
        ),
    ),
);

Since there are some performance concerns (see comment on gerrit), I am waiting for more green light before moving any forward.

There are some PERFORMANCE concerns: the fact that we add a query param will break caching. Note that this seems to be a current limitation of the QuickSurveys module, and that seems hard to work around (as of now). Second note, the survey (and the query param) is included ONLY for articles when you come FROM the search engine results page (a limited amount of articles, considering our search usage).

JGirault edited a custom field.Dec 9 2015, 10:53 PM
debt added a subscriber: debt.Jan 25 2016, 5:55 PM
debt renamed this task from Add a survey on the article page when coming from a wiki search to QuickSurveys: Add survey on article page when coming from a wiki search.Jan 28 2016, 4:51 PM
debt triaged this task as High priority.
dr0ptp4kt added a subscriber: dr0ptp4kt.

I added this to sprint 66 for Reading Web, which starts 15-February-2016, for visibility.

I think this kind of survey is biased because it only takes the users who clicked on a search result into account. All other users who were not satisfied with search results may just abandon the page without navigating forward. For this reason I'd question the validity of the conclusions drawn from the survey results.

If I'm not mistaken, the earlier parts of the funnel and dwell time analysis will play into fuller analyses of explicit and implicit satisfaction of users who didn't tap through. I think what they're looking for here is whether users who actually did tap through in fact found the content germane to their search. Imagine if for example the user didn't pay close attention to the snippet in the fulltext search results, or if the fulltext results didn't closely mirror the top of the page, etc.

Is there a particular patch you need help from the Reading web team for? I'm a little confused.

@dr0ptp4kt I don't have the time to coordinate this on my end right now, so the real world has caused this to sink in priority somewhat. Sorry about that.

Per a recent Discovery retrospective, I'm going to try to step in and get this task unstuck. My memory from a recent meeting was that the current blocker is legal, related to the privacy policy. I have pinged @JGirault on a couple tasks, so we can be sure we all understand the true state of things.

Thanks for the updates.

Moving to the Portal backlog board for now until we can get https://phabricator.wikimedia.org/T127119 unblocked

debt added a comment.Apr 14 2016, 5:08 PM

I believe we've unblocked and completed everything that needs to happen in order for this task to move forward.

Can we discuss what we need/want to do next with this ticket?

So…looking at this task, it seems like the only thing left is to actually deploy the survey and start collecting data? @MSyed should have the specific wording we decided on in a meeting a long time ago. If we're not at that stage, then we need to figure out what else is left.

@debt this is marked high priority, but hasn't seen any activity for over a year now. Should it be declined and the patches abandoned?

@debt @EBernhardson for the record, I'd love for this to be done and for us to start collecting feedback :)

I do agree that the priority should be adjusted to be more realistic :)

debt added a comment.Jun 1 2017, 12:42 AM

I'm not sure that the Discovery team has anything that we'd want to use for quick surveys at this time. I'm inclined to close this out and get the patches abandoned. Is there anyone on this ticket (other than @mpopov) that would be interested in using Quick Surveys in the near future?

MaxSem closed this task as Declined.Feb 26 2018, 7:13 PM
MaxSem added a subscriber: MaxSem.

Is there anyone on this ticket (other than @mpopov) that would be interested in using Quick Surveys in the near future?

Apparently not.

Change 254089 abandoned by MaxSem:
[Temp] Add a survey on article pages when users come from SERP

Reason:
The underlying ticket has been declined.

https://gerrit.wikimedia.org/r/254089

As I understand it'll be used in other contexts, but I'm not aware of a recently expressed need on this particular use case (CC @leila in case I'm missing something, though).

leila added a comment.Feb 26 2018, 8:09 PM

We have not directly needed this feature, but, the absence of it can create bias in the data we collect as part of our studies, too. We do have an extensive debiasing step that may take care of this part of bias introduced as long as the features we use for debiasing can fully explain the differences between the users who come to a Wikipedia article from another article or external search but not internal search. This is a big claim that we as researchers can never make (that our features capture all differences), so I'd love to see this issue addressed if we have enough resources on this project.

While we're at it: I'd love if addressing this feature turns into addressing a bigger question/problem of control over where QSs can be shown. For example, it would be great if there are a set of conditions we can agree on that define the whole space of conditions under which QS can be shown to the user, and we allow the researchers to choose specific conditions for the specific study. Suppose, for example, I want to know certain information from users who have visited both uk and en Wikipedias in the past 10 clicks (as part of their current session, however we define it).