Page MenuHomePhabricator

Look into eventlogging data for the user feedback gadget for technical docs whether it's "useful"
Closed, DeclinedPublic

Description

The gadget got deployed in T195119#5121390. We had some very early data in T213782#5103235.
(AFAIK the data is stored for 90 days according to WMF's data retention guidelines.)

Hence wondering how many people click on that gadget and if it creates "enough useful" data. Take a look and explore! :)

Event Timeline

Hello! I'm very interested in this data and would be happy to help any way I can. I don't have the required access, but I'd be happy to start looking at the data if someone else can retrieve it.

hive (event)> select event.page_name as pagename, count(*) as total,sum(case when event.vote = 'yes' then 1 else 0 end) as yes, sum(case when event.vote = 'no' then 1 else 0 end) as no from userfeedback where year = 2020 group by event.page_name;
pagename        total   yes     no
API:Allfileusages/en    1       1       0
API:Allmessages/Sample_code_1   1       1       0
API:Categorymembers     2       2       0
API:ChangeContentModel  1       1       0
API:ChangeContentModel/fr       1       1       0
API:Client_code 1       0       1
API:Client_code/Java    1       1       0
API:Client_code/Java/fr 1       1       0
API:Edit        1       1       0
API:Edit_-_Set_user_preferences 1       0       1
API:Errors_and_warnings 1       0       1
API:Geosearch   1       0       1
API:Get_the_contents_of_a_page  1       0       1
API:Images      1       1       0
API:Import      1       0       1
API:JSON_version_2      2       2       0
API:Langlinks   1       1       0
API:Languagesearch      1       0       1
API:Lists       1       0       1
API:Login       2       1       1
API:Main_page   33      18      15
API:Main_page/de        1       0       1
API:Main_page/id        1       1       0
API:Main_page/ja        1       1       0
API:Main_page/nl        1       1       0
API:Main_page/ru        1       0       1
API:Meta        2       1       1
API:Options     3       2       1
API:Page_info_in_search_results 4       2       2
API:Parsing_wikitext    2       2       0
API:Parsing_wikitext/Sample_code_1      1       1       0
API:Properties  2       2       0
API:Query       5       2       3
API:Query/de    1       0       1
API:REST_API/Media_files        1       1       0
API:REST_API/Pages      1       1       0
API:REST_API/Search     1       0       1
API:Random/af   1       0       1
API:Raw_query_continue  1       1       0
API:Recent_changes_stream       1       1       0
API:Redirects   1       1       0
API:Search      1       1       0
API:Search_and_discovery        1       1       0
API:Tokens      3       1       2
API:Tutorial    1       0       1
API:Web_APIs_hub        38      22      16
Help:Access_to_Cloud_VPS_instances_with_PuTTY_and_WinSCP        1       0      1
Help:Accessing_Cloud_VPS_instances      2       2       0
Help:Adding_Disk_Space  1       0       1
Help:At_a_glance:_Cloud_VPS_and_Toolforge       3       1       2
Help:Cloud_Services_Introduction        24      12      12
Help:Cloud_Services_communication       1       0       1
Help:Cloud_VPS_project  1       1       0
Help:Create_a_Wikimedia_developer_account       6       3       3
Help:Force_all_users_to_log_in_afresh   1       0       1
Help:Labs_labs_labs     1       0       1
Help:Manage_floating_IP_addresses_assigned_to_Cloud_VPS_instances       1      01
Help:SSH_Fingerprints/gerrit.wikimedia.org:29418        1       1       0
Help:Shared_storage     1       0       1
Help:Toolforge  4       3       1
Help:Toolforge/Database 1       1       0
Help:Toolforge/How_to   1       0       1
Help:Toolforge/My_first_Flask_OAuth_tool        1       1       0
Help:Toolforge/My_first_NodeJS_OAuth_tool       1       0       1
Help:Toolforge/Pywikibot        2       0       2
Help:Toolforge/Right_to_fork_policy     1       1       0
Help:Toolforge/Rules    3       3       0
Help:Toolforge/Terms_and_conditions     1       0       1
Help:Toolforge/Toolforge_standards_committee    1       0       1
Help:Toolforge/Version_Control_in_Toolforge     1       0       1

Amazing! Thanks, @bd808!

Overall, it seems like the number of responses is pretty low on most pages. For example, the page with the most responses, API:Main Page, got 38 responses out of 67,000+ pageviews.

It seems like there may be a correlation between shorter pages and number of responses, with shorter pages like API:Main Page, API:Web APIs Hub, and Help:Cloud Services Introduction getting significantly more responses than longer pages like API:Query (which saw more traffic than API:Web APIs Hub). It may be the case that shorter pages make the gadget more visible than having it at the bottom of a longer page. We could consider moving the gadget to a more prominent location (maybe as part of the API sidebar?) temporarily to see if that increases participation.

Generally speaking, I’d hope to see 100+ responses per month to be able to compare trends over time. This would help us answer questions like: Do pages with consistently positive responses share characteristics that we could use to improve pages with more negative responses?

As an example using the data we have, we could compare API:Parsing wikitext (2 yes, 0 no) and API:Options (2 yes, 1 no) with API:Query (2 yes, 3 no). Although these numbers are too small to really be sure, we could hypothesize that the presence of sample code in multiple programming languages on API:Parsing wikitext and API:Options is resulting in more positive responses, and we could add similar samples to API:Query and see if the responses improve.

Let me know what you think! CC: @srodlund

@bd808 and @apaskulin My gut tells me that probably one of the reasons we aren't seeing more feedback on this is because of design. My guess is that if we move the gadget on the page, make it more prominent/visible, we might see more engagement. I think it would be useful to talk with some folks in design research and ask them if they concur and what their recommendations for best practices for the design and placement of the gadget.

@bd808 and @apaskulin My gut tells me that probably one of the reasons we aren't seeing more feedback on this is because of design. My guess is that if we move the gadget on the page, make it more prominent/visible, we might see more engagement. I think it would be useful to talk with some folks in design research and ask them if they concur and what their recommendations for best practices for the design and placement of the gadget.

I agree. One idea I had quite a while ago that might help: add html+js to the gadget that would make a element of some sort that "floats" along with scrolling of the page so that there is a feedback call to action that folks can always see. The current layout makes the gadget hard to find, basically it blends into the footer and is only visible if you end up scrolling al the way to the bottom of a help article. Some UX expert feedback on this would be great though because doing it in a way that is accessible and usable without being intrusive and annoying is not a trivial thing.

If there is a way to have it functional on desktop and mobile, that would be ideal.

I agree with you that improving the design (including the placement of this gadget) might help. There is also something to learn from the design of the survey form used for this study: https://meta.wikimedia.org/wiki/Research:Study_of_performance_perception_on_Wikimedia_projects,
https://commons.wikimedia.org/wiki/File:Wikimedia_performance_perception_survey_screenshot_in_English.png. Happy to put some work on it in the next quarter :-)

@srishakatux Yes! It would be awesome to do some more experimentation/development on this -- even if it just means moving the current gadget to a more prominent location and comparing usage.

The example you shared is interesting. I wonder how they came up with that particular design? I think it's definitely worth looking into what would make for the most effective.

Followed https://wikitech.wikimedia.org/wiki/Analytics/Systems/Superset#Access and logged into https://superset.wikimedia.org/

In the "SQL Lab", I am unable to filter via e.g. wiki="mediawikiwiki" (says "Column 'mediawikiwiki' cannot be resolved") and I'm unable to parse the JSON (?) in eventwhich has the format [int id, string "page name", string "yes"|"no"] (trying event.vote="no" says "Column 'no' cannot be resolved"; I took that format from here). So back to fiddling with CSV exports.

Data is stored for last 90 days. August had 53 replies; September had 53 replies. 40 times "no"; 66 "yes":

1https://wikitech.wikimedia.org/wiki/Help:Accessing_Cloud_VPS_instances , no
2https://wikitech.wikimedia.org/wiki/Help:At_a_glance:_Cloud_VPS_and_Toolforge , no
3https://wikitech.wikimedia.org/wiki/Help:At_a_glance:_Cloud_VPS_and_Toolforge , no
4https://wikitech.wikimedia.org/wiki/Help:At_a_glance:_Cloud_VPS_and_Toolforge , yes
5https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication , no
6https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction , no
7https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction , no
8https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction , no
9https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction , no
10https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction , no
11https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction , no
12https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction , yes
13https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction , yes
14https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction , yes
15https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction , yes
16https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction , yes
17https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction , yes
18https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction , yes
19https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction , yes
20https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction , yes
21https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction , yes
22https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_Instances , no
23https://wikitech.wikimedia.org/wiki/Help:Create_a_Wikimedia_developer_account , yes
24https://wikitech.wikimedia.org/wiki/Help:Create_a_Wikimedia_developer_account , yes
25https://wikitech.wikimedia.org/wiki/Help:Getting_Started , yes
26https://wikitech.wikimedia.org/wiki/Help:IRC , yes
27https://wikitech.wikimedia.org/wiki/Help:Labs_labs_labs , yes
28https://wikitech.wikimedia.org/wiki/Help:Puppet-compiler , yes
29https://wikitech.wikimedia.org/wiki/Help:Security_groups , yes
30https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/dev.toolforge.org , yes
31https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/gerrit.wikimedia.org:29418 , yes
32https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/login.toolforge.org , no
33https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/login.toolforge.org , yes
34https://wikitech.wikimedia.org/wiki/Help:Toolforge , no
35https://wikitech.wikimedia.org/wiki/Help:Toolforge , no
36https://wikitech.wikimedia.org/wiki/Help:Toolforge , yes
37https://wikitech.wikimedia.org/wiki/Help:Toolforge/Dumps , yes
38https://wikitech.wikimedia.org/wiki/Help:Toolforge/Elasticsearch , yes
39https://wikitech.wikimedia.org/wiki/Help:Toolforge/Java , no
40https://wikitech.wikimedia.org/wiki/Help:Toolforge/Pywikibot , yes
41https://wikitech.wikimedia.org/wiki/Help:Toolforge/Pywikibot , yes
42https://wikitech.wikimedia.org/wiki/Help:Toolforge/Pywikibot , yes
43https://wikitech.wikimedia.org/wiki/Help:Toolforge/Terms_and_conditions , yes
44https://wikitech.wikimedia.org/wiki/Help:Toolforge/Version_Control_in_Toolforge , yes
45https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web , no
46https://www.mediawiki.org/wiki/API:Account_creation , yes
47https://www.mediawiki.org/wiki/API:Allpages , yes
48https://www.mediawiki.org/wiki/API:Article_ideas_generator , yes
49https://www.mediawiki.org/wiki/API:Authmanagerinfo , no
50https://www.mediawiki.org/wiki/API:Categories , yes
51https://www.mediawiki.org/wiki/API:Categorymembers , yes
52https://www.mediawiki.org/wiki/API:Client_code , no
53https://www.mediawiki.org/wiki/API:Client_code/Evaluations/Pywikibot , yes
54https://www.mediawiki.org/wiki/API:Data_formats , yes
55https://www.mediawiki.org/wiki/API:Data_formats/ru , yes
56https://www.mediawiki.org/wiki/API:FAQ/ja , yes
57https://www.mediawiki.org/wiki/API:FAQ/pt-br , no
58https://www.mediawiki.org/wiki/API:Geosearch , no
59https://www.mediawiki.org/wiki/API:Geosearch , yes
60https://www.mediawiki.org/wiki/API:Information_in_Wikimedia_projects , no
61https://www.mediawiki.org/wiki/API:Information_in_Wikimedia_projects , yes
62https://www.mediawiki.org/wiki/API:Information_in_Wikimedia_projects , yes
63https://www.mediawiki.org/wiki/API:Main_page , no
64https://www.mediawiki.org/wiki/API:Main_page , no
65https://www.mediawiki.org/wiki/API:Main_page , no
66https://www.mediawiki.org/wiki/API:Main_page , no
67https://www.mediawiki.org/wiki/API:Main_page , yes
68https://www.mediawiki.org/wiki/API:Main_page , yes
69https://www.mediawiki.org/wiki/API:Main_page , yes
70https://www.mediawiki.org/wiki/API:Main_page , yes
71https://www.mediawiki.org/wiki/API:Main_page/hi , yes
72https://www.mediawiki.org/wiki/API:Meta/ru , yes
73https://www.mediawiki.org/wiki/API:Opensearch , yes
74https://www.mediawiki.org/wiki/API:Page_info_in_search_results , yes
75https://www.mediawiki.org/wiki/API:Page_info_in_search_results , yes
76https://www.mediawiki.org/wiki/API:Parsing_wikitext , no
77https://www.mediawiki.org/wiki/API:Parsing_wikitext , yes
78https://www.mediawiki.org/wiki/API:Picture_of_the_day_viewer , no
79https://www.mediawiki.org/wiki/API:Purge , no
80https://www.mediawiki.org/wiki/API:Purge , yes
81https://www.mediawiki.org/wiki/API:Query , yes
82https://www.mediawiki.org/wiki/API:Resetpassword , no
83https://www.mediawiki.org/wiki/API:REST_API/Extensions , yes
84https://www.mediawiki.org/wiki/API:Restricting_API_usage , no
85https://www.mediawiki.org/wiki/API:Tokens , no
86https://www.mediawiki.org/wiki/API:Tokens , yes
87https://www.mediawiki.org/wiki/API:Tutorial , no
88https://www.mediawiki.org/wiki/API:Upload , yes
89https://www.mediawiki.org/wiki/API:Userinfo , yes
90https://www.mediawiki.org/wiki/API:Watchlist , yes
91https://www.mediawiki.org/wiki/API:Web_APIs_hub , no
92https://www.mediawiki.org/wiki/API:Web_APIs_hub , no
93https://www.mediawiki.org/wiki/API:Web_APIs_hub , no
94https://www.mediawiki.org/wiki/API:Web_APIs_hub , no
95https://www.mediawiki.org/wiki/API:Web_APIs_hub , no
96https://www.mediawiki.org/wiki/API:Web_APIs_hub , no
97https://www.mediawiki.org/wiki/API:Web_APIs_hub , no
98https://www.mediawiki.org/wiki/API:Web_APIs_hub , no
99https://www.mediawiki.org/wiki/API:Web_APIs_hub , yes
100https://www.mediawiki.org/wiki/API:Web_APIs_hub , yes
101https://www.mediawiki.org/wiki/API:Web_APIs_hub , yes
102https://www.mediawiki.org/wiki/API:Web_APIs_hub , yes
103https://www.mediawiki.org/wiki/API:Web_APIs_hub , yes
104https://www.mediawiki.org/wiki/API:Web_APIs_hub , yes
105https://www.mediawiki.org/wiki/API:Web_APIs_hub , yes
106https://www.mediawiki.org/wiki/API:Web_APIs_hub , yes

@srishakatux may disable this gadget for the time being. In some future, @srodlund and @apaskulin maybe could fiddle out a plan how to use the data, keep the data somewhere somehow longer for 90 days while keeping it legal; and how to gather feedback (e.g. in a central place as looking at each talk page does not scale), etc...

Regarding 👍👎 as binary feedback options, IIRC Alex (?) once also mentioned that "consensus is that a starred rating system is the best way to collect feedback on pages" when working on dev portal content strategy. Could be something to also think about when rebooting this (or not)...

Note that we should revisit this as part of the content strategy for updating key docs.

T290018 got declined, thus also declining this ticket.