In order to have a better understanding of the maps userbase, we should add another graph that will show per country maps usage. Even though we could analyse both unique users (unique IP + user agent combination), as well as all tiles, the unique users should be enough for now.
Goal 1: show top 9 countries plus 1 for "other" as lines.
Goal 2: allow drill-down into any country and to select which countries to show
We should probably collect data for both, but only implement goal 1 for now.
IP and user agent are nothing like unique users, though; they're massively inaccurate.
You should collect data for neither and implement neither until it's been prioritised by @Deskana, who I have added to this ticket since he's the product manager for the feature.
@Ironholds, yes, well aware of that - our "users" graph on the dashboard is not as exact as we would like it to be, but good enough approximation until we have something better. As for data - I do not "collect" anything. You do it as part of your dashboard scripts.
Now that we have figured out how to do this in a straightforward manner (see T123347), it should be fairly straightforward to knock this one out. Pulling into the sprint. It's fairly low priority at the minute, though.
I think we covered the grafana question.
Looking at this it's going to be sliightly more complicated than initially expected; the EL schema doesn't expose country data, so we're gonna have to rely on Hive. Not a problem but will require some more work.
Change 272797 had a related patch set uploaded (by OliverKeyes):
Add data collection scripts for maps users-per-country
Change 272797 merged by Bearloga:
Add data collection scripts for maps users-per-country
We're only collecting for Goal 1 at the moment since Goal 2 has privacy implications.
Change 273511 had a related patch set uploaded (by OliverKeyes):
Fix geodata retrieval code for Maps
Change 273526 had a related patch set uploaded (by OliverKeyes):
[WIP] Add geographic visualisation code
@Deskana @Yurik We've noticed that having daily tile usage data going all the way back to September is impacting the performance of the Maps dashboard, at least for the first user who opens the dashboard on a given day. Do y'all need that kind of granularity?
I'm thinking maybe we could have a 30-day rolling window for daily data and have a separate view that's just monthly median values?
Alternatively, we could get rid of aggregation by tile size and just have a daily tiles served, users, and median tiles per user. Do y'all use the tiles by zoom level view (http://discovery.wmflabs.org/maps/#tiles_total_by_zoom)?
@mpopov, I would like to keep that data for the future usage, but it is not needed to be shown on the graph. So if you can keep it in the database and tell me how to access it, you can make it into a slide window.
@Yurik okay, thanks
@Ironholds Okay, do you want to have a non-visualized dataset that has all the daily data and a visualized dataset that just has a rolling 30 day window? (For geo data and tile usage data.) In both cases, we can have two writing calls: conditional_write (for all data) and conditional_rewrite (for windowed data). Thoughts?
Change 273998 had a related patch set uploaded (by OliverKeyes):
Add code to generate a rolling window of data
@Yurik http://discovery-beta.wmflabs.org/maps/#geo_breakdown
@Ironholds Looks like the geo data needs backfilling for Feb 12-28. Also there needs to be a missing data check for the geo data.
This update will be deployed to production (discovery.wmflabs.org) in the next deployment "train" (we prefer to bundle up multiple dashboard deployments).