The specific metrics we want to show on the Dashboard will define what data we need to request from the Toolhunt backend.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | NicoleLBee | T324507 Develop a web app for editing Toolhub records | |||
| Resolved | NicoleLBee | T324674 [frontend][backend][dashboard] Decide what metrics we want to show and implement routes |
Hannah and I came up with the following preliminary list:
Would it be interesting to track changes over time?
I'm assuming that our backend will be making calls to the toolhub API for some of this information (for instance, the total number of tools). Are we going to set up our own database to hold other information? Because it occurs to me that one way to populate the leaderboard (as well as the tables showing past contributions), would be to save that data ourselves.
Yes, that's a way we could do it. I encourage you to open a new task where we can have this discussion.
Thinking about this more, the response we need should be in numbers meaning that what we want from the first two are
This means that the second and the third will be the same data so maybe we can do away with one of them.
It would be great to track the changes over time say in the last 1 month/6 months and or 1 year we could add this to our stretch goals.
Then a GET request to a hypothetical toolhunt/api/metrics endpoint would ideally look like;
{
"current_user_contributions": 2,
"global_contributions": 456,
"total_number_of_tools_in_toolhub": 45678,
"percentage_of_tools_with_missing_data": 57,
"frequently_missing_field": "field_20",
"frequently_edited_field": "field_12"
}Here are my proposals:
Leaderboard: a GET request to /toolhunt/api/leaderboard would return a JSON string that could be parsed into a JS object with the following form:
[{ "highscores_latest":
{ <username>: score,
<username>: score },
highscores_all_time:
{ <username>: score,
<username>: score }
}]This is if we want to display statistics both for the past 30 days and for all time; if we only want one table we can just return an array containing a single object, that contains the values that we'll use to populate the table.
Is array > object > object an unnecessary amount of nesting? Or is it preferable to do it this way, so that we can attach a label, e.g., "highscores_latest", that will explain the content/purpose of the returned data?
Dashboard: a GET request to toolhunt/api/dashboard would return:
{
"global_contributions_total": 456,
"total_number_of_tools_in_toolhub": 45678,
"percentage_of_tools_with_missing_data": 57,
"frequently_missing_field": <field name, e.g. available_ui_languages>,
"frequently_edited_field": <field name>,
"current_user_contributions": [
{ "date": <date string>,
"user": <username>,
"tool": <toolname> },
{ "date": <date string>,
"user": <username>,
"tool": <toolname> }
],
"global_contributions_recent": [
{ "date": <date string>,
"user": <username>,
"tool": <toolname> },
{ "date": <date string>,
"user": <username>,
"tool": <toolname> }
]
}I'd suggest creating a more flexible set of /metrics endpoints that could serve both the leaderboard and the dashboard, and that aren't tied to any specific implementation of these features. For instance, instead of having a /toolhunt/api/leaderboard endpoint that returns an object with data for the "last 30 days" and "all times", it would be nicer to have something like /toolhunt/api/metrics/edits/{user}/{start_date}/{end_date}, that could be used to retrieve edit data for arbitrary time frames, optionally filtered by user. This data could then be used for both user-specific and aggregate statistics, as well as for the leaderboard.
Some links for REST API inspiration:
Wikistats: https://stats.wikimedia.org/#/all-projects
Wikimedia REST API: https://wikimedia.org/api/rest_v1/
These links provide access to the Wikistats website and the documentation for the underlying REST API.
Sounds good. I had wondered whether it would be more efficient to make multiple calls from the frontend to the backend (e.g., one for each date range) or to make one call to the backend and then have the backend make multiple calls to the Toolhub API (one for 30 days, one without a date specified). But the backend will be sending multiple requests to the Toolhub API in either case, so I suppose it doesn't make much difference.
Having a flexible query also means that we could, for instance, allow the user to choose the range of dates (every 30 days, every 60 days, in the past week, etc.) -- but that falls under the "stretch goals!"
The frontend is built with Vue; could be interesting to take a look at as well:
https://github.com/wikimedia/analytics-wikistats2
We haven't designed our database yet, but we will most likely be fetching the data from there, rather than passing on requests to the Toolhub API in real time. For instance, if we want to display some stats about how much data is missing from Toolhub, we could run a daily job that gathers the data we need, computes the stats we want, and stores them on our backend for our API to expose.
I spent some time trying to puzzle out the code. The level of complexity makes my head spin, although perhaps it's less complicated than it looks.
Ok, thanks for the clarification. I presume that the tradeoff is accuracy vs. speed? As in, if we're running a daily job there's no guarantee that our information will be completely accurate (presumably we'd update the information in our database every time a user submitted a task, though, so it should be pretty close to accurate), but the upshot is that if we're not querying Toolhub each time, it will be faster?
(System design is another thing that I don't know anything about but am eager to learn.)
One thing I noticed looking at the Wikimedia REST API documentation is that the endpoints are extremely granular.
So rather than having one API call that returns a bucket of data, as in our original proposal, is it better practice to make multiple calls that each return one piece of data?
E.g., rather than:
/toolhunt/api/metrics returning, say, information about the number of fields missing, the most frequently edited field, and the least edited field, we'd make calls to
toolhunt/api/metrics/fields/total_missing toolhunt/api/metrics/fields/most_edited toolhunt/api/metrics/fields/least_edited
and then bundle and return the data we get back from our database?
I'm going to reopen this, as I think it's the most appropriate place to catalog desired metrics for the dashboard and the SQL queries that we can use to get them.
Suggestions welcome! I'm only listing a few that spring immediately to mind.
| Metric | Query |
|---|---|
| # of tools | SELECT COUNT(*) FROM tool; |
| # of incomplete tasks | SELECT COUNT(*) FROM task WHERE user IS NULL; |
| # of edits made using Toolhunt | SELECT COUNT(*) FROM task WHERE user IS NOT NULL; |
| # of edits made using Toolhunt in the past X days | SELECT COUNT(*) FROM task WHERE user IS NOT NULL AND timestamp >= {other timestamp}; |
| # of incomplete tasks by field name | SELECT field_name, COUNT(*) FROM task WHERE user IS NULL GROUP BY field_name; |
| the field that's most often empty | as above, plus ORDER BY COUNT(*) DESC LIMIT 1 |
| the field that's least often empty | as # of incomplete tasks, plus ORDER BY COUNT(*) LIMIT 2; (and take the 2nd, because the first will always be "replaced_by") |
| current user's contributions in the last X days | SELECT COUNT(*) FROM task WHERE user = {username} AND timestamp >= {other timestamp} |
| current user's total contributions | SELECT COUNT(*) FROM task WHERE user = {username} |
We could also do things like most complete tool, least complete tool, etc. etc... it's very exciting!
We settled on going halfway between a completely granular approach of one endpoint for every metric, and a single endpoint that grabbed all metrics, and will instead categorize metrics by type (roughly).
Current user metrics: /api/metrics/user
total # of contributions
number of contributions in the last 30 days
Contributions metrics: /api/metrics/contributions
total contributions made using toolhunt
contributions made in the last 30 days
Tasks metrics: /api/metrics/tasks
number of incomplete tasks in the db
Tools metrics: /api/metrics/tools
number of tools in the db
number of tools with missing data
Future expansions could include information about fields (e.g., most/least frequently empty field, most often edited field, etc.)
I'll expand this as I go along.
I've implemented this first set of metrics and opened a PR for review: https://github.com/wikimedia/toolhunt/pull/39
The basic set of metrics routes have been implemented and merged into the main branch. We can always (and probably will) expand upon them later.