Context
We're currently supporting the Burmese Wikipedia project to enable Citoid there. To understand the impact of our work, we want to monitor the data points listed below. Some of these may already exist, so our first step is to check whether the necessary instrumentation and data sources are available. Afterwards, we will implement any missing instrumentation or scraping and build the corresponding dashboards.
Metrics
Finding: The necessary tracking for monitoring Citoid on Burmese Wikipedia already exists in the[[ https://www.mediawiki.org/wiki/VisualEditor/FeatureUse_data_dictionary#Actions | VisualEditor/FeatureUse data dictionary ]] . The data dictionary and schema confirm that all key user interactions (feature, action) and user metadata (user_editcount, editingSessionId) are being logged.
Usage by user types (breakdown by newcomer and all others):
How many users attempt to use Citoid auto-generate tool, timeframe daily?
| action | feature | description |
|---|---|---|
| automatic-generate | citoid | The "generate" button was clicked to generate a citation based on a URL (or DOI or ISBN) |
- Query for this event with a set time range for daily timeframe
How many of those attempts are initiated by newcomers (edits <100)?
- Create a query to check the user_editCount for all users who have triggered the 'automatic-generate' event
What percentage of all newcomers make an edit using citoid auto-generate tool?
- Query for (Newcomers (user_editCount < 100) that successfuly used Citoid auto-generate tool ('automatic-insert' event)) / (All editing newcomers (user_editCount < 0)) * 100
Success rates:
How many of those attempts are successful or unsuccessful, broken down by user type?
- Successful Newcomers: Query for 'automatic-insert' event for users where user_editcount < 100.
- Unsuccessful Newcomers: Query for 'automatic-generate-fail' for users where user_editcount < 100.
- Successful experienced editors: Query for 'automatic-insert' for users where user_editcount > 100.
- Unsuccessful experienced editors: Query for 'automatic-generate-fail' for users where user_editcount > 100.
How often does the auto-generate tool fail?
| action | feature | description |
|---|---|---|
| automatic-generate-fail | citoid | Automatic citation generation failed and error message is displayed. |
What are the sources (list) that fail?
| action | feature | description |
|---|---|---|
| automatic-generate-fail-template-build | citoid | Automatic citation generation failed because the templates could not be built. E.g. due to misconfiguration, missing TemplateData or wrong data mapping. |
| automatic-generate-fail-api | citoid | Automatic citation generation failed due to an issue while retrieving the necessary information from the API. E.g. if there are no results or if there's a network issue. |
- Does the tool work for reputable Burmese sources? @Johannes_Richter_WMDE will try to understand from Burmese community what sources they deem as reputable.
Template usage:
Can we see an increase in the usage of citation templates vs. basic references?
| feature name | meaning |
|---|---|
| cite | Adding or editing citations using citation templates. |
| reference | Adding or editing citations using the basic, empty-box reference dialog (the only option when citation templates and Citoid are not set up on the wiki) |
- Query that compares the daily event counts of the cite feature to the daily event counts of the reference (Set time range)
How often are templates used, broken down by template type?
| feature name | meaning |
|---|---|
| cite-* | Adding or editing citations using specific citation templates (e.g. cite-book, cite-Internetsäit, cite-libro, cite-web, cite-any). |
funnel: How many clicks on the citation button lead to a reference being created / inserted into an article? (Hypothesis: the reference-creation rate improves compared to the previous status quo).
- To test if the reference-creation rate improves, first calculate the success rate of the pre-Citoid citation workflow, then calculate the new funnel rate from 'automatic-generate' event to 'automatic-insert' event within same session (editingSessionId from schema), then compare the two percentages.
Summary
It seems that with current events logged we can cover all needs in here. So at the end it boils down to setting up graphs and potentially make sure we take snapshots to see before/after.
The only thing that does not seem to be logged yet is
What are the sources (list) that fail?
afaik there's currently no record of, what the users put into the field when the request failed. -> see logstash this might be helpful for us