Page MenuHomePhabricator

Create user feedback gadget for technical documentation on Wikitech pages
Closed, ResolvedPublic

Description

Proposal: We need a gadget to measure user perceptions of usefulness and usabilty of technical documentation on Wikitech. The gadget should initially provide very basic feedback about whether documentation is useful or not to our users. The feedback will help guide an iterative documentation process focused around user needs and help us to target areas where we can best apply our resources.

Initially, the gadget would be available on Wikitech documentation pages only.

The idea is to start with existing documenation and to have a feedback mechanism for users with a simple question:

Was this documentation helpful?

The answer should be (Yes or No | thumbs up or thumbs down)?

If the answer is no, users would be presented with a text box to gather further information about why the feedback was not useful.

Another version of this gadget might be a scale or star rating with the same basic idea behind it (less than 4 stars would trigger a mechanism to gather more user feedback)

Though the idea is to have a fairly simple mechanism intially, we might consider a variety of questions going forward:

  • What is the purpose of your visit today?
  • Did you find the information you were looking for?
  • What do you like most / least about the information on this page?
  • Is there anything you care to share about your vist to this page?
  • How did you find this page?
  • How can we improve?

If possible, it would be nice to be able to gather some additional metrics.

  • How many times a page had been viewed?
  • How long the average page views were?

There are a couple "wish to haves" for the feedback gadget.

  • A simple thank-you message that comes up after the visitor has particpated in the survey. (It takes time to fill out a survey.)
  • A persistent link to help/support. (The theory is that it should be easy and comfortable for users to reach out and ask questions when they have them.)

Event Timeline

srodlund created this task.May 19 2018, 9:09 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 19 2018, 9:09 PM
bd808 added a subscriber: bd808.Jun 4 2018, 8:01 PM
Vvjjkkii renamed this task from Create user feedback gadget for technical documentation on Wikitech pages to cncaaaaaaa.Jul 1 2018, 1:09 AM
Vvjjkkii triaged this task as High priority.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: Aklapper.
CommunityTechBot renamed this task from cncaaaaaaa to Create user feedback gadget for technical documentation on Wikitech pages.Jul 1 2018, 5:38 PM
CommunityTechBot raised the priority of this task from High to Needs Triage.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot added a subscriber: Aklapper.
srishakatux added a subscriber: srishakatux.

(tech engagement offsite follow-up)

bd808 added a comment.Oct 17 2018, 5:02 PM

I think that EventLogging could be made to work as the data collection backend for a gadget like this. In addition to feedback results we may want to have a "gadget shown" type of data point collected that we could use to determine percentage of users giving any feedback at all. This would be useful in attempts to unbias the data we receive (I am assuming that response rate will be low generally and biased towards negative feedback).

Once the data is captured by EventLogging we will need to do additional work to turn the raw data into something that we can visualize and act on, but I would consider that a second phase of the project. Until we can get some data we don't need to be too concerned about visualizations, etc.

For background research, I looked into existing gadgets or extensions around rating/voting and found a couple interesting ones:

  • Extension:ArticleFeedback5 - Made a lot of progress with different iterations before it was disabled on Wikimedia wikis in ~2013. The extension was disabled a) due to philosophical reasons and b) there weren’t more resources to put in development efforts. It supported anonymous data collection and for that relied heavily on AbuseFliter extension. The work on AbuseFilter was on halt for a while. Also, it wasn't clear how to handle the amount of data collected via the extension and how to use it. There were some major bugs with the archiving process too.
  • Extension:VoteNy - Allows voting on articles. The design might need a bit of tweak, but it is exactly what we need. The voting widget can be embedded on a page with a wiki markup. <vote /> outputs a green square voting box and <vote type=1 /> outputs five voting stars.

From work on ArticleFeedback, it seems like it might be too complicated to collect anonymous qualitative feedback, and to archive and process it. The simple solution would be a thumbs up and thumbs down button that allows (un)voting. We can also display number of votes that an article has received. If a user clicks on thumbs down button, we redirect them to provide feedback on the talk page.

EventLogging seems like a robust approach to log data entries, JSON schema, probably easy to update entries, but the data can only be stored for ~90 days. I am inclined towards using Extension:VoteNy, as it is functional and works. But, also wondering what would be the advantage of using EventLogging over database logging and extension over gadget for this purpose? How about the idea of using a customized version of VoteNy on Wikitech, to begin?

bd808 added a comment.Dec 6 2018, 8:12 PM

EventLogging seems like a robust approach to log data entries, JSON schema, probably easy to update entries, but the data can only be stored for ~90 days. I am inclined towards using Extension:VoteNy, as it is functional and works. But, also wondering what would be the advantage of using EventLogging over database logging and extension over gadget for this purpose? How about the idea of using a customized version of VoteNy on Wikitech, to begin?

First, thank you for doing some "build vs buy" research. It is always a good idea to stop and think about how to reuse or extend existing things before diving into building new things from scratch.

Deploying a new extension to a Wikimedia wiki is not a trivial undertaking. "A customized version" also seems to imply a fork of the existing project which is almost never a good idea in my opinion either.

Here are my reasons for suggesting a gadget + eventlogging approach for the initial project:

  • EventLogging works as a data collection pipeline and as pointed out in T195119#4798468 already deals with cleaning up historical data which I consider to be a feature rather than a problem
  • Gadgets do not require security review and are easy to tweak and to uninstall
  • We really do not know what we will settle on for the UI/UX of this and would like to be able to experiment
  • We don't need large amounts of historically data, most of what we will be interested in is recent data and historical trends

The gadget + eventlogging parts are only a partial solution. We would still need to build some backend scripts against the eventlogging data store to produce whatever reporting we need to analyze the votes. What kind of reporting we want/need should be fleshed out a bit more before anything is built as well so we can test the assumption that any of these things would/could work. I think we should consider some of the lessons of ArticleFeedback in that collecting anonymous text feedback could be problematic. It might be a much better idea to have the gadget make it easy/obvious to add suggestions for improving a page to its talk page or a central talk page that might make responding to feedback easier and more public.

@bd808 This is a helpful summary! Also, learned from Gilles that their team used QuickSurveys to understand site performance on wikis: T188503: Identify two wikis to run a research study and get approval from their respective communities

It looks like QuickSurveys is easy to use, but the constraint is that for asking questions we can only stick to a specific format. It looks like our initial survey format is going to be similar to the one followed by the performance team: one question with a yes and no choice. If not, there are Microsurveys too. Plus, it relies on EventLogging and we can use it to host external surveys via Google/Qualtrics service.

Though, in its current state and as per @Quiddity's comment T194099#4189182, we are able to show the survey to % of users, we could consider enhancing the extension’s capabilities to add support for specific pages.

Not trying to be lazy (:P). Any opinions on this?

bd808 added a comment.Dec 7 2018, 9:50 PM

we could consider enhancing the extension’s capabilities to add support for specific pages.

We could, but again this is high cost development (changing an existing deployed extension to serve a new use case) rather than the hoped for 'light' development cost of a gadget that can be changed with relatively little oversight or friction with other use cases. I haven't looked at the implementation of QuickSurveys to see how invasive the changes would need to be to make it something that we could configure to present to 100% of people looking at pages in the Help namespace on Wikitech, but since this deviates in multiple ways from the documented use of the extension I am assuming that it is non-trivial.

Update > User script is ready and is in testing mode. Some technical details:

  • Feedback form is displayed at the very bottom of a content page that is in a namespace supported by the script. Feedback form screenshots are here.
  • For a “thumbs-up” or “thumbs-down” vote:
    • User’s vote gets recorded as per Schema:UserFeedback
    • Cookie set with vote value and expire in 10 minutes. A user can vote again only after cookie clears.
    • Confirmation messages get shown; and in case of a thumbs-down vote, a user is being asked to provide more feedback on the article's talk page.

Current status: Tried testing on the EventLogging devserver and reported the errors I ran into here T153641#48252231.

Interesting external example of a "Help and feedback" footer section: 4 tabs, each with 2-3 entries - https://docs.gitlab.com/ee/development/i18n/translation.html#help-and-feedback

srishakatux closed this task as Resolved.EditedApr 17 2019, 11:32 PM

User feedback gadget can currently be seen in action in the help namespace on Wikitech. Scroll to the bottom of the following page: https://wikitech.wikimedia.org/wiki/Help:Toolforge to view the documentation feedback form.

Code for gadget lives on Wikitech:
https://wikitech.wikimedia.org/wiki/MediaWiki:Gadget-userfeedback.css
https://wikitech.wikimedia.org/wiki/MediaWiki:Gadget-userfeedback.js

Eventlogging schema used by the gadget is on Meta:
https://meta.wikimedia.org/wiki/Schema:UserFeedback

Steps followed to deploy this gadget are in: T213782#4956527

Instructions to fetch eventlogging data are in: T213782#5103235

Blog post on Medium: https://medium.com/@Srish_Aka_Tux/documentation-feedback-gadget-for-wikis-c280531e3108