Page MenuHomePhabricator

[Spike 2 days] Work out how to automate reported user feedback
Closed, ResolvedPublic3 Estimated Story Points

Description

Background

We are collecting feedback on https://www.mediawiki.org/wiki/Reading/Web/Accessibility_for_reading/Reporting. Sorting through feedback is currently manual. We should work out a strategy for how to handle feedback going forward so it gets to the right people. If we don't do anything with this feedback it will reflect badly on the team.

User story

As a reader who reports issues I want to make sure my issues get heard.

Requirements

Answer the following questions:

  • What ways are there to report the number of clicks received on the report links to communities? (note: the expected format would be
page namenumber of clicks to report link
"this is a page"x

For example, we can take inspiration from the current contrast reporting

BDD

  • For QA engineer to fill out

Test Steps

  • For QA engineer to fill out

Design

  • Add mockups and design requirements

Acceptance criteria

  • Add acceptance criteria

Communication criteria - does this need an announcement or discussion?

  • Add communication criteria

Rollback plan

  • What is the rollback plan in production for this task if something goes wrong?

This task was created by Version 1.2.0 of the Web team task template using phabulous

Event Timeline

ovasileva renamed this task from Work out how to automate reported user feedback to [Spike] Work out how to automate reported user feedback.Jul 25 2024, 1:19 PM
ovasileva updated the task description. (Show Details)
ovasileva renamed this task from [Spike] Work out how to automate reported user feedback to [Spike 2 days] Work out how to automate reported user feedback.Jul 29 2024, 5:32 PM
ovasileva set the point value for this task to 3.

How are anonymous users using the feedback button. What volume of users are clicking it every day? Is there anything useful we can get from their feedback?

On English Wikipedia, I looked at a 6 hour period of data. I logged about 50 clicks an hour for logged in and logged out users. From the limited test run I did before disabled by ops, there were no patterns - there were only about 10 articles with more than one click, and when looking at those pages it was not obvious what issues they were referring to (the articles were more accessible in light mode than dark mode)

What ways are there to report the number of clicks received on the report links to communities?

For anonymous users, I was going to recommend we build Grafana dashboards like so that present the top most clicked pages as it would have provided an easier path to visualizing data in Grafana. Unfortunately, ops have weighed in and asked us not to do this.

This really leaves us with two options:

  1. Support and spec out work on T361214 - and wait for a solution we can use.
  2. Manually get an engineer and a data engineer to generate these reports manually as needed. This would likely be 1-2 days worth of work each time we need to do it, as the data is protected by NDA so likely would require some anonymization and review process. While this is not a long term solution this could be a solution if this data is only important say for a short period. Before doing this we would need to be sure that work is worthwhile!

What ways are there to automatically post feedback received in written form on https://www.mediawiki.org/wiki/Reading/Web/Accessibility_for_reading/Reporting to the specific wiki that feedback is from?

I don't think there is a way to automate this. As somebody who was monitoring this: The feedback ranges from known issues to general feedback as well as actual issues. I would hope over time we'd see more of the latter which is the most useful.
Rather than guessing what editors want, I think we should re-pose this as a question for the community members that are showing up to these pages. The question being "What would help you here?", "Is this page helpful?"

I would recommend however that this feedback is sent to the wiki itself rather than mediawiki.org. Although some editors are showing up at www.mediawiki.org it doesn't seem like the appropriate venue. For example many editors rely on Watchlists to learn about new feedback and this is outside their project watchlist. We should reach out to the users that are using it and ask them what they want: https://xtools.wmcloud.org/articleinfo/www.mediawiki.org/Reading/Web/Accessibility_for_reading/Reporting/en.wikipedia.org

This seems like a logical next step here.

Proposed next steps based on the above (@NBaca-WMF, @Jdlrobson, @JScherer-WMF, @sgrabarczuk):

  1. We remove the report link for logged-out users. Based on the data above, it doesn't feel like we're getting quality feedback from this.
  2. We begin publishing the feedback manually on-wiki in weekly batches - @sgrabarczuk - I think this is what we had agreed on doing originally but have not prioritized it. Once you're back from break, I'd suggest we prioritize it and try to see how it's working before looking at further options for logged-in users
  3. We continue brainstorming ways for improving dark mode on small and medium-sized wikis. I can set up a meeting around this over the next few weeks.

Let me know if you have concerns with the proposal. Otherwise, I'll set up a ticket for removing the link for logged-out users for next sprint and resolve this.

Discussed in steering with @NBaca-WMF and @JScherer-WMF - will proceed with the plan above.