Page MenuHomePhabricator

Instrument permalink timestamps
Open, MediumPublic

Description

This would add tracking to when users click on comment timestamps to copy the permalink.

Event Timeline

Note that this doesn't measure if the permalink is used. We could perhaps track on-wiki usage by looking for #c- or #h- in wikitext diffs, but not off-wiki use:
https://en.wikipedia.org/w/index.php?search=insource%3A%2F%5C%23c%5C-%2F&title=Special%3ASearch&profile=advanced&fulltext=1&ns0=1&ns1=1&ns2=1&ns3=1&ns4=1&ns5=1&ns6=1&ns7=1&ns8=1&ns9=1&ns10=1&ns11=1&ns12=1&ns13=1&ns14=1&ns15=1&ns100=1&ns101=1&ns118=1&ns119=1&ns710=1&ns711=1&ns828=1&ns829=1

On en.wiki (the only wiki where the feature isn't deployed yet) there appears to be about ~2,600 links already. de.wiki and fr.wiki have fewer than 500 links each.

Unless we want a whole new schema for this, it's probably a decent fit for <desktop/mobile>webuiactionstracking. (We already log clicks to the various add-topic buttons there.)

It'd be pretty easy to hook into the same code that's highlighting permalinked comments and log to that schema from there, if we want to track usage, too.

Change 1008928 had a related patch set uploaded (by DLynch; author: DLynch):

[schemas/event/secondary@master] Desktopwebuiactionstracking: add missing `show` to action enum

https://gerrit.wikimedia.org/r/1008928

Change 1008929 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/DiscussionTools@master] Instrument when permalinks are copied

https://gerrit.wikimedia.org/r/1008929

Change 1008930 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/DiscussionTools@master] Instrument when permalinks are followed

https://gerrit.wikimedia.org/r/1008930

https://gerrit.wikimedia.org/r/1008929 can be merged immediately and just logs when a permalink timestamp is clicked. (action: click, action_source: discussiontools.permalink-copied)

https://gerrit.wikimedia.org/r/1008930 might want closer review -- it'd log as a show event in the webuiactionstracking schemas when something that seems like an attempted permalink pageload happens. (It'll log it as a show in that schema, with action_source being either discussiontools.permalink-found or discussiontools.permalink-missing depending on whether it's actually on the current page.)

@MNeisler may want to confirm that this is okay as a way to log it, also.

It'd be pretty easy to hook into the same code that's highlighting permalinked comments and log to that schema from there, if we want to track usage, too.

Note that this wouldn't distinguish between user-generated permalinks, and the permalinks we use in comment notifications. The Echo notifications have a tell-tale ?markasread query string, but that gets removed immediately by ext.echo.init.js: https://github.com/wikimedia/mediawiki-extensions-Echo/blob/f5ebd41ec959c1f3d568ce58d45454907d444f51/modules/ext.echo.init.js#L14

Change 1008929 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@master] Instrument when permalinks are copied

https://gerrit.wikimedia.org/r/1008929

Could probably hook in and make that removal of ‘markasread’ leave some trace that we could filter by if need be, but I’m also fine just not merging that patch if we don’t think we need it. (Part of the reason I made two patches. 🤩)

Assigning this over to @MNeisler for review.

MNeisler triaged this task as Medium priority.Mar 7 2024, 3:44 PM
MNeisler edited projects, added Product-Analytics (Kanban); removed Product-Analytics.

I agree that the webuisactionstracking schema is probably the best fit for this but there are a few limitations this will introduce including per wiki sampling rates and data retention periods that will impact how the metrics in T359489 are calculated. See notes below:

Tracking clicks to copy permalinks

The instrumentation added in https://gerrit.wikimedia.org/r/1008929 (action: click, action_source: discussiontools.permalink-copied) looks good to me.

  • The webuiactionstracking schema does not track user id info so we will need to join with EditAttemptStep to obtain user and edit info using the pageToken field. This pageToken field is not retained in the webuiactionstracking schemas past 90 days so we only be able to review users that tried generating a permalink for the past 90 days vs since enabled.
  • I believe the webuiactionstracking schemas still apply sampling rates so we will be able to look at percentages but the actual numbers reported will be impacted.

Tracking how many times do permalinks appear within wikis

It sounds like we're still looking to find a scalable way to track when a wikitext talk page comment/topic is published per T359489#9613887.

The patch in https://gerrit.wikimedia.org/r/1008930 would instrument when a permalink pageload happens/when they are followed, which seems slightly different to me and I'm not sure this is needed based on the identified metrics.

cc @ppelberg @DLynch

I believe the webuiactionstracking schemas still apply sampling rates so we will be able to look at percentages but the actual numbers reported will be impacted.

Yeah, it's at 20% most places. Lower on mobile -- 10% default, with many wikis overridden to 20%.

It sounds like we're still looking to find a scalable way to track when a wikitext talk page comment/topic is published per T359489#9613887.

The patch in https://gerrit.wikimedia.org/r/1008930 would instrument when a permalink pageload happens/when they are followed, which seems slightly different to me and I'm not sure this is needed based on the identified metrics.

It's a slightly better proxy for what we want to know, which is "how much are people using permalinks?". Tracking when they're shared on-wiki would be a step there, but it wouldn't give us any insight at all into off-wiki usage, so I'm not convinced that it's particularly useful -- particularly given that it's a pain to instrument. Whereas tracking when someone has arrived at a page from a permalink would be logged regardless of the source...

Hey folks, is this being coordinated with the Web team who are currently in the middle of migrating *UIActionTracking to the Metrics Platform? T344274: Adopt Web Team Instrumentation to Metrics Platform

@mpopov We haven't coordinated with them about it. That said, we've also not been touching the schemas (this patch did because desktop and mobile are inconsistent, but it wasn't necessary and so nobody has merged it), so is there actually anything to coordinate around the migration?

I think it depends on the technical specifics of the migration, which @KSarabia-WMF would be able to verify. Essentially my main concern was potential presence of new permalink-copied events coming from original instrument that might not be present in the new MP-based instrument, which would impact the data QA. If the new instrument automatically produces these new permalink events, then all is good and there's no concern.

So long as the migration didn't involve massively changing the schema, I don't think it should matter -- the name field is in the schema as a freeform string currently.

After examining:

Everything is probably good? If my understanding is correct, the new permalink events would be present in both tables (legacy desktopwebuiactionstracking and new mediawiki_web_ui_actions) without any additional steps.

Change #1008928 merged by jenkins-bot:

[schemas/event/secondary@master] Desktopwebuiactionstracking: add missing `show` to action enum

https://gerrit.wikimedia.org/r/1008928

Moving to Editing QA to verify that when people click (desktop)/tap (mobile) a permalink timestamp, there is a client-side event being emitted/logged.