Page MenuHomePhabricator

Make it easier for gadget authors to understand gadget usage and debug user-reported issues
Open, Needs TriagePublic

Description

Many community maintained MediaWiki gadgets don't have a practical way to understand how users are actually using them or to debug issues when something goes wrong. When bugs are reported, maintainers usually rely on anecdotal reports.

At the same time, WMF already has well-established systems for observability and debugging in other areas: Wikimedia-Logstash, Test Kitchen Event-Platform. Gadgets, however, are mostly outside of this ecosystem. As a result, it is hard for gadget authors to answer basic questions like how often a feature is used, which options users select, or what path led to a problematic result. For example, I have a gadget that suggests categories based on content from multiple language versions of a wiki, it is currently difficult to see which source language or wiki contributed to a bad suggestion that a user picked. This makes it harder to find and block low-quality sources (wikis).

It would be useful to have a simple, privacy-friendly way for gadget authors to get basic usage and debugging information. This could be aggregated metrics, opt-in debugging mode, or a lightweight instrumentation API for gadgets.

Event Timeline

Logstash does include bugs in gadgets. You could apply to get access to it, though it is not easy (you need to get a high level "sponsor" of your request). I think it would help if every major wiki had at least one user that could view Logstash.

I fixed some bugs reported only in Logstash in the past. Logstash is mostly nice to look at after deploying a new default gadget or a gadget expected to be used mostly by readers (so by users that are less likely to report on usual channels). Logstash can be hard to read, but It does help with observing the volume of new errors, especially those nasty ones (syntax errors, NPE aka TypeError etc).

It is however sometimes hard to figure out whether certain error is caused by a gadget, a core script, or an extension. It would help if there would be an easy way to filter out errors coming from the core and extensions.