Validate Assumption
The Citoid extension is used by most users of Visual Editor and Wikitext Editor 2017 to insert references. We have seen this in multiple discovery interviews, however, more quantitative data is required to validate this.

Steps

• Check if data already exists in VisualEditor/FeatureUse_data_dictionary and document they keys
• If not implement instrumentation using ve.track and document they keys

Questions and Metrics

• How often is automatic reference generation used across sessions, broken down by VisualEditor-MediaWiki-2017WikitextEditor and VisualEditor .
  • Coarse, summary answer is fine (e.g. "N% of all sessions have used the Citoid at least once")
  • Use Superset's SQL Lab to write a query grouping all presto.event.visualeditorfeatureuse events by edit session ID, selecting a count of citoid.automatic-generate events for that group. See the count_if function.
  • Save the query as a dataset.
  • Create charts based on this dataset, for example
• How often do users interact with the Citoid successfully. Defined by adding a reference by using the reference generation.
  • Chart "Citoid success"
• How often do users fail to add an automatic generated reference. Defined by interacting with the input field and then aborting the dialog.
  • The dialog "abort" event isn't comparable to successful insert events, because it includes more than just the automatic lookup sessions. Chart "Citoid success ratio" shows the successful inserts divided by lookups, so the conversion between clicking "Create" and "Insert".
• How often and in which situations does the tool fail in retrieving results? Defined by how often the error message is triggered.
  • Error Message: We couldn't make a citation for you. You can create one manually using the "Manual" tab above.
  • Instrument using ve.track and document the new action in the feature use dictionary.

Implemented:

• feature citoid actions
  • automatic-generate-fail - Automatic reference generation fails, error message is triggered
  • automatic-generate-fail-network- Tool fails to retrieve search results for generating a citation
  • automatic-generate-fail-searchResults - Tool encounters a network error while attempting to generate a citation.

Outcomes

Citoid usage and events board (superset): https://superset.wikimedia.org/superset/dashboard/518/

Citoid service dashboard (grafana): https://grafana.wikimedia.org/d/NJkCVermz/citoid?orgId=1&refresh=5m

Event data already includes:

• feature citoid actions
  • automatic-generate - "generate" button was pressed
  • automatic-insert - "insert" button was pressed
  • reuse-choose (TODO: document citoid vs Cite "reuse")
• feature citoidEducationPopup action show (TODO: screenshot)