Page MenuHomePhabricator

Metrics for new Kartographer feature usage
Open, Needs TriagePublic0 Estimated Story Points

Description

We are implementing new features in the Maps (Kartographer) extension and want to track use and effectiveness. This is the epic to document what we will track. Sub-tickets can be added for implementation.

  • Add as events so data can be segmented by wiki

T304076: Nearby feature for maps
Blockers for deploying MVP

  • Count first click on show nearby button, per fullscreen map view (show clicks only, ignore hide clicks) T318678
  • Count first click on an article link in a nearby pop up, per fullscreen map view T318678

Can happen after MVP, ordered by priority

  • Count every time a Kartographer map is opened in full screen mode T318678
  • Compare across wikis: % of times someone who opens a full screen map, also clicks on show nearby
  • Compare across wikis: % of times someone who clicks on show nearby button, also clicks on pop up
  • Number of markers initially loaded (do not send another metric after zoom/pan) T318678
  • Lag time between clicking the button and seeing nearby markers. T318678

T307695: Display coordinate markers in Kartographer maps from QID

  • Number of articles with geopoints added via QID T318927

T302297: Display geopoint SPARQL queries in a map in Wikipedia (GeoPoints)

  • Number of articles with geopoints added via SPARQL query T318927

Event Timeline

Lena_WMDE renamed this task from Create metrics for maps usages to Metrics for new Kartographer feature usage.Aug 31 2022, 3:07 PM
Lena_WMDE added a project: Epic.
Lena_WMDE updated the task description. (Show Details)
Lena_WMDE moved this task from Backlog to Ready for pickup on the WMDE-GeoInfo-FocusArea board.
awight subscribed.

Pulling into the sprint as part of preparing the Nearby feature for deployment.

I would make the following assumptions about what data to collect—does this look right?

  • Count clicks on show nearby button T318678
    • Only record "show" clicks and ignore "hide" clicks.
  • Count every time a Kartographer map is opened in full screen mode
    • If possible, we'll instrument just the clicking interaction to open a full-screen map. But this might also include the rare cases when a map is opened in full screen mode because the user navigated to a direct link like this.
  • % of times someone who opens a full screen map, also clicks on show nearby
    • This depends on recording the user's unique session token, so we'll keep that for 90 days.
  • Number of articles with geopoints, added via QID or SPARQL

I think we've missed an interesting metrics possibility, when looking at the frequent question of "number of articles in the database", vs. "number of articles viewed". What if we send an event for every article *added* as you say, which uses geopoints. These events are easy to record and can be deduplicated by article ID, giving an exact count. It's the important component of "number of articles in the database over time" that we would want to graph anyway, and serves as a good replacement for that total. (Basically, we allow ourselves to be slightly blind to the integral of this metric for now.)

Article ID would be IMHO necessary for these "geopoint added" events, and maybe interesting to include in the other event types as well. Eg. we could learn what types of article are more often viewed with nearby, etc.

Data retention / privacy:

  • No identifying information about the user, no edit count bucket, no IP address.
  • After 90 days, we'll keep the metrics but will drop session_token. This means we can no longer associate eg. full-screen and nearby events on this archived data, but we can still compare the totals over time.

Random suggestion:

  • Count the first mouse interaction with a dynamic thumbnail map, as a complement to the full-screen count.

Change 835621 had a related patch set uploaded (by Awight; author: Awight):

[schemas/event/secondary@master] [WIP] Maps event schema

https://gerrit.wikimedia.org/r/835621

Change 835662 had a related patch set uploaded (by Awight; author: Awight):

[mediawiki/extensions/Kartographer@master] [WIP] Log nearby-show events

https://gerrit.wikimedia.org/r/835662

Change 836804 had a related patch set uploaded (by Awight; author: Awight):

[operations/mediawiki-config@master] Wire new event stream for maps interactions

https://gerrit.wikimedia.org/r/836804

awight set the point value for this task to 0.Sep 29 2022, 1:30 PM

Change 835621 merged by jenkins-bot:

[schemas/event/secondary@master] Maps interaction event schema

https://gerrit.wikimedia.org/r/835621

Change 835662 merged by jenkins-bot:

[mediawiki/extensions/Kartographer@master] Log basic nearby and fullscreen events

https://gerrit.wikimedia.org/r/835662

Change 836804 merged by jenkins-bot:

[operations/mediawiki-config@master] Wire new event stream for maps interactions

https://gerrit.wikimedia.org/r/836804

Mentioned in SAL (#wikimedia-operations) [2022-10-04T13:29:42Z] <awight@deploy1002> Started scap: Backport for [[gerrit:836804|Wire new event stream for maps interactions (T315972 T318678)]]

Mentioned in SAL (#wikimedia-operations) [2022-10-04T13:30:05Z] <awight@deploy1002> awight and awight: Backport for [[gerrit:836804|Wire new event stream for maps interactions (T315972 T318678)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet

Change 838097 had a related patch set uploaded (by Awight; author: Awight):

[mediawiki/extensions/Kartographer@wmf/1.40.0-wmf.4] Log basic nearby and fullscreen events

https://gerrit.wikimedia.org/r/838097

Mentioned in SAL (#wikimedia-operations) [2022-10-04T13:36:31Z] <awight@deploy1002> Finished scap: Backport for [[gerrit:836804|Wire new event stream for maps interactions (T315972 T318678)]] (duration: 06m 49s)

Change 838097 merged by jenkins-bot:

[mediawiki/extensions/Kartographer@wmf/1.40.0-wmf.4] Log basic nearby and fullscreen events

https://gerrit.wikimedia.org/r/838097

Mentioned in SAL (#wikimedia-operations) [2022-10-04T14:03:35Z] <lucaswerkmeister-wmde@deploy1002> Synchronized php-1.40.0-wmf.4/extensions/Kartographer/modules/dialog: Backport: [[gerrit:838097|Log basic nearby and fullscreen events (T315972, T318678)]] (no wikis use wmf.4 yet, but the code exists, so the change needs to be synced) (duration: 03m 42s)

Change 845639 had a related patch set uploaded (by Awight; author: Awight):

[mediawiki/extensions/Kartographer@master] [WIP] polymorphic nearbyLayer has all the behavior

https://gerrit.wikimedia.org/r/845639