|Declined||• srodlund||T248892 Look into eventlogging data for the user feedback gadget for technical docs whether it's "useful"|
|Declined||None||T290018 Plan for feedback gadget|
- Mentioned In
- T290018: Plan for feedback gadget
T287301: Define accuracy criteria for style
T254147: Wikimedia GSoD 2020 Proposal: Audience research and user experience
- Mentioned Here
- T290018: Plan for feedback gadget
T195119: Create user feedback gadget for technical documentation on Wikitech pages
T213782: Deploy user feedback gadget
Hello! I'm very interested in this data and would be happy to help any way I can. I don't have the required access, but I'd be happy to start looking at the data if someone else can retrieve it.
hive (event)> select event.page_name as pagename, count(*) as total,sum(case when event.vote = 'yes' then 1 else 0 end) as yes, sum(case when event.vote = 'no' then 1 else 0 end) as no from userfeedback where year = 2020 group by event.page_name;
pagename total yes no API:Allfileusages/en 1 1 0 API:Allmessages/Sample_code_1 1 1 0 API:Categorymembers 2 2 0 API:ChangeContentModel 1 1 0 API:ChangeContentModel/fr 1 1 0 API:Client_code 1 0 1 API:Client_code/Java 1 1 0 API:Client_code/Java/fr 1 1 0 API:Edit 1 1 0 API:Edit_-_Set_user_preferences 1 0 1 API:Errors_and_warnings 1 0 1 API:Geosearch 1 0 1 API:Get_the_contents_of_a_page 1 0 1 API:Images 1 1 0 API:Import 1 0 1 API:JSON_version_2 2 2 0 API:Langlinks 1 1 0 API:Languagesearch 1 0 1 API:Lists 1 0 1 API:Login 2 1 1 API:Main_page 33 18 15 API:Main_page/de 1 0 1 API:Main_page/id 1 1 0 API:Main_page/ja 1 1 0 API:Main_page/nl 1 1 0 API:Main_page/ru 1 0 1 API:Meta 2 1 1 API:Options 3 2 1 API:Page_info_in_search_results 4 2 2 API:Parsing_wikitext 2 2 0 API:Parsing_wikitext/Sample_code_1 1 1 0 API:Properties 2 2 0 API:Query 5 2 3 API:Query/de 1 0 1 API:REST_API/Media_files 1 1 0 API:REST_API/Pages 1 1 0 API:REST_API/Search 1 0 1 API:Random/af 1 0 1 API:Raw_query_continue 1 1 0 API:Recent_changes_stream 1 1 0 API:Redirects 1 1 0 API:Search 1 1 0 API:Search_and_discovery 1 1 0 API:Tokens 3 1 2 API:Tutorial 1 0 1 API:Web_APIs_hub 38 22 16 Help:Access_to_Cloud_VPS_instances_with_PuTTY_and_WinSCP 1 0 1 Help:Accessing_Cloud_VPS_instances 2 2 0 Help:Adding_Disk_Space 1 0 1 Help:At_a_glance:_Cloud_VPS_and_Toolforge 3 1 2 Help:Cloud_Services_Introduction 24 12 12 Help:Cloud_Services_communication 1 0 1 Help:Cloud_VPS_project 1 1 0 Help:Create_a_Wikimedia_developer_account 6 3 3 Help:Force_all_users_to_log_in_afresh 1 0 1 Help:Labs_labs_labs 1 0 1 Help:Manage_floating_IP_addresses_assigned_to_Cloud_VPS_instances 1 01 Help:SSH_Fingerprints/gerrit.wikimedia.org:29418 1 1 0 Help:Shared_storage 1 0 1 Help:Toolforge 4 3 1 Help:Toolforge/Database 1 1 0 Help:Toolforge/How_to 1 0 1 Help:Toolforge/My_first_Flask_OAuth_tool 1 1 0 Help:Toolforge/My_first_NodeJS_OAuth_tool 1 0 1 Help:Toolforge/Pywikibot 2 0 2 Help:Toolforge/Right_to_fork_policy 1 1 0 Help:Toolforge/Rules 3 3 0 Help:Toolforge/Terms_and_conditions 1 0 1 Help:Toolforge/Toolforge_standards_committee 1 0 1 Help:Toolforge/Version_Control_in_Toolforge 1 0 1
Amazing! Thanks, @bd808!
Overall, it seems like the number of responses is pretty low on most pages. For example, the page with the most responses, API:Main Page, got 38 responses out of 67,000+ pageviews.
It seems like there may be a correlation between shorter pages and number of responses, with shorter pages like API:Main Page, API:Web APIs Hub, and Help:Cloud Services Introduction getting significantly more responses than longer pages like API:Query (which saw more traffic than API:Web APIs Hub). It may be the case that shorter pages make the gadget more visible than having it at the bottom of a longer page. We could consider moving the gadget to a more prominent location (maybe as part of the API sidebar?) temporarily to see if that increases participation.
Generally speaking, I’d hope to see 100+ responses per month to be able to compare trends over time. This would help us answer questions like: Do pages with consistently positive responses share characteristics that we could use to improve pages with more negative responses?
As an example using the data we have, we could compare API:Parsing wikitext (2 yes, 0 no) and API:Options (2 yes, 1 no) with API:Query (2 yes, 3 no). Although these numbers are too small to really be sure, we could hypothesize that the presence of sample code in multiple programming languages on API:Parsing wikitext and API:Options is resulting in more positive responses, and we could add similar samples to API:Query and see if the responses improve.
Let me know what you think! CC: @srodlund
@bd808 and @apaskulin My gut tells me that probably one of the reasons we aren't seeing more feedback on this is because of design. My guess is that if we move the gadget on the page, make it more prominent/visible, we might see more engagement. I think it would be useful to talk with some folks in design research and ask them if they concur and what their recommendations for best practices for the design and placement of the gadget.
I agree. One idea I had quite a while ago that might help: add html+js to the gadget that would make a element of some sort that "floats" along with scrolling of the page so that there is a feedback call to action that folks can always see. The current layout makes the gadget hard to find, basically it blends into the footer and is only visible if you end up scrolling al the way to the bottom of a help article. Some UX expert feedback on this would be great though because doing it in a way that is accessible and usable without being intrusive and annoying is not a trivial thing.
I agree with you that improving the design (including the placement of this gadget) might help. There is also something to learn from the design of the survey form used for this study: https://meta.wikimedia.org/wiki/Research:Study_of_performance_perception_on_Wikimedia_projects,
https://commons.wikimedia.org/wiki/File:Wikimedia_performance_perception_survey_screenshot_in_English.png. Happy to put some work on it in the next quarter :-)
@srishakatux Yes! It would be awesome to do some more experimentation/development on this -- even if it just means moving the current gadget to a more prominent location and comparing usage.
The example you shared is interesting. I wonder how they came up with that particular design? I think it's definitely worth looking into what would make for the most effective.
Followed https://wikitech.wikimedia.org/wiki/Analytics/Systems/Superset#Access and logged into https://superset.wikimedia.org/
In the "SQL Lab", I am unable to filter via e.g. wiki="mediawikiwiki" (says "Column 'mediawikiwiki' cannot be resolved") and I'm unable to parse the JSON (?) in eventwhich has the format [int id, string "page name", string "yes"|"no"] (trying event.vote="no" says "Column 'no' cannot be resolved"; I took that format from here). So back to fiddling with CSV exports.
Data is stored for last 90 days. August had 53 replies; September had 53 replies. 40 times "no"; 66 "yes":
@srishakatux may disable this gadget for the time being. In some future, @srodlund and @apaskulin maybe could fiddle out a plan how to use the data, keep the data somewhere somehow longer for 90 days while keeping it legal; and how to gather feedback (e.g. in a central place as looking at each talk page does not scale), etc...
Regarding 👍👎 as binary feedback options, IIRC Alex (?) once also mentioned that "consensus is that a starred rating system is the best way to collect feedback on pages" when working on dev portal content strategy. Could be something to also think about when rebooting this (or not)...