| • Ramsey-WMF | |
| Mar 29 2018, 8:51 PM |
| F26712693: flow_analysis.png | |
| Oct 22 2018, 6:17 PM |
| F26012056: events.png | |
| Sep 18 2018, 3:05 PM |
Planning how we measure usage of the upcoming multilingual file caption feature. This should include:
We're looking to collect this data to measure things like abandon rates, etc.
| Subject | Author | Repo | Branch | Lines +/- | |
|---|---|---|---|---|---|
| Track interactions | Matthias Mullie | mediawiki/extensions/UploadWizard | master | +340 -73 |
What kind of questions do we expect to get answers for?
I'm looking for more detail to figure out exactly what we need to log.
A simple example: we could easily log when a user opens the language selection dropdown, but
or:
We can narrow things down by also tracking the amount of "Add caption in another language" clicks and amount of "Filled out a caption", but those could also mean "never even filled out anything for that other caption" or "edited the same caption over and over" - we still might not have all the context required, depending on the kind of answers we're looking for.
If we're only looking to track whether those elements are being used, we don't need much context (we may not even need to log anything and just query how many captions get submitted in what kind of languages)
If we're looking to answer questions like "do most people consistently change the language because the default selection is bad", we may need to log more data (e.g. interface language, default language, selected language)
So...
It's not too hard to log events in UploadWizard right now - just need to know exactly what we're looking for to make sure we have enough context when we eventually want to process the data.
Hi @Tbayer . Would you be able to add some insight for Matthias' questions above? Thanks!
Hi @matthiasmullie . I think I have answers for your questions now :)
If we're looking to answer questions like "do most people consistently change the language because the default selection is bad", we may need to log more data (e.g. interface language, default language, selected language)
We do want to be able to answer questions like that. As we make more efforts to improve multilingual support all across Commons, we'll need to know which approaches work/don't work AND why. Having the contextual data will hopefully help.
One particular stat we're hoping to capture is abandon rates: how often someone clicked add a caption, maybe even started to fill something in, but never actually finished the process. If it helps, I'd suggest using this as the primary use case for metrics gathering: answering whether the info we capture gives us enough clear data to say for sure if people get confused/lost in the process and don't finish, and perhaps narrowing down which element is the one people get turned off by.
To add: for the abandon rate and other questions, we need to connect all the different events logged during the same upload session, via a token. This could be something like the flowId field used in the exisiting/past UploadWizard EL schemas, or perhaps we might even end up reusing and augmenting some of these directly.
@matthiasmullie correctly points out that it would be useful to document the main research questions in more detail (@Ramsey-WMF and I had an initial brainstorming about these a while ago but haven't arrived at a full finalized list yet); I think this task was filed more as an initial placeholder in the beginning. I'll flesh it out a bit more now based on the discussion above.
(Sorry, I missed the above questions and ping earlier this month while being on/catching up from a longer vacation.)
Change 431776 had a related patch set uploaded (by Matthias Mullie; owner: Matthias Mullie):
[mediawiki/extensions/UploadWizard@master] Track interactions
I added a whole lot of tracking to UW, for pretty much every element that can be interacted with (full list below)
All of these would go into the UploadWizardFlowEvent schema, where only the flow id, flow position & event name are logged.
This should be enough to figure out *where* people drop out.
I'll wait for further input from @Tbayer before pushing to get this code-reviewed, in case we want to change things (log more context, ...)
Another question: with ~85000 monthly UploadWizard uses, logging every action will generate an enormous amount of data.
We should probably only log a fraction; what would a good sampling ratio be? 10%? 5%?
List of events to be logged:
Hi @Tbayer - have you had a chance to look into this and flesh things out a bit more yet, or other thoughts on how we want to pursue this? Or other thoughts on above list of interactions that can be logged (without any context, in the current patch)?
Well the good news is that as of T203596, EL schemas need to be whitelisted for MySQL (previously they had to be blacklisted when volume of data was anticipated to be an issue) so as far as I know we're good to go without sampling because all of those events are going into Hadoop/Hive.
List of events to be logged:
…
Those seem alright to me! I would like clarifications for the following and when they are triggered:
Since we are interested in assessing the usage of the multilingual caption feature, there should be a context or extra string field where we can include additional information for certain events as JSON. For example, since we're interested in usage of multilingual captions feature then it could be helpful if the thanks-file-upload event looked like:
{
"cap_langs": ["en", "de", "fr"],
"desc_langs": ["de"]
}To indicate that the file was uploaded with English, German, and French captions and description in German.
And the first event in an upload session would benefit from the context/extra field having information like:
{
"display_lang": "ru",
"input_lang": "ru"
}to capture data about the user's language settings, like @matthiasmullie suggested in an earlier comment.
We've recently made a bunch of not insignificant changes to UW, so the initial patch would need some work before we could merge it, but I think we've not really changed much about what things can be interacted with.
So, some more details about these actions:
See below screenshot to see what thingy triggers what event
Below 5 are equivalent, but for "caption" rather than "description" (not yet in production)
So the big question is: what kind of information would we like to get out of this?
The original patch will log every interaction without any additional context. This would mostly only be useful for figuring out where people drop out or find behavioural patterns.
If we want to collect data about languages, just capturing all events may not suffice and we may want to add more context (like @mpopov described), possibly creating (a) new schema(s).
What kind of things do we want to log/measure/get answers for?
I've thought about this and I think the current event-per-interaction approach should be scrapped in favor of a more forward-thinking solution. Analytics Engineering has some guidelines in place for creating EventLogging schemas in a way that the events are easily ingested into Druid, which makes them easy to visualize in Turnilo/Superset, which is usable by non-analysts which means @Ramsey-WMF et al. wouldn't be blocked by, say, the unavailability of a data analyst ;)
If we can send a single, flow session-summarizing event at the end of the process (either when user finishes and the upload is success OR when user exits by closing tab/window or goes to a different page within the tab) then the schema might look like:
{
dt: 'ISO-8601 formatted timestamp',
// Primary context dimensions:
wizard_version: 'Upload Wizard version',
display_language: 'User preference: language used for menus, e.g. "de"',
input_language: 'User preference: language used for writing, e.g. "en"',
flow_step: '"tutorial", "file", "deeds", "details", or "thanks"',
// Secondary dimensions for additional context:
file_extension: 'Extension of the file being uploaded',
ownership: '"self" or "someone else"',
deed: 'e.g. "cc-by-sa-4.0"; "custom" if not using one of built-in options',
location_set: '"yes" if a location was added, "no" otherwise',
// Metrics:
measure_elapsed: 'Seconds since user initially started the process',
measure_current: 'Seconds elapsed since leaving the previous flow step',
measure_descriptions: 'Number of languages that user provided description in',
measure_captions: 'Number of languages that user provided caption in',
measure_restarts: 'Number of times user started over (returned to "file" step)'
}Depending on when exactly the user abandons the process some of those dimensions would be "N/A" (e.g. deed would be "N/A" if user abandoned the process before selecting a license).
@Nuria @matthiasmullie: what do you think of this?
Here's an initial draft of how we'd compute daily per-step upload abandonment rates & overall upload success rate under this system:
Since the schema would have context dimensions such as Upload Wizard version and users' language preferences, we'd be able to compute these rates on a per-version, per-language basis.
(Updated the funnel analysis diagram because I had a brain toot that made me write "users" in place of "uploads")
@mpopov I'm all in favor of the schema you proposed. It'd generate a lot less noise and would be a hell of a lot easier to process compared to a neverending stream of every single interaction.
My main/only concern is that this schema would be excellent to track overall health of the application, but wouldn't really be useful to track specific interactions with specific elements (as described in this task's description) - not unless/until we specifically add them into the schema anyway (but what are the exactly are the interactions we'd like to measure?)
Schema looks good and *I think* you would be able to visually see abandonment rate with a bar chart on flow_step ( '"tutorial", "file", "deeds", "details", or "thanks"') given a period of time.
measure_current: 'Seconds elapsed since leaving the previous flow step',
Note that this measure will get "bucketized" on druid as plain "numbers" are not "ingestable" into druid, if that makes sense . https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/Schema_Guidelines#Bucketing_time_measures
My main/only concern is that this schema would be excellent to track overall health of the application, but wouldn't really be useful to track specific interactions with specific elements
Our strong recommendation is to split different types of data to different schemas.
Our strong recommendation is to split different types of data to different schemas.
100% agree - just want to make sure we don't lose track of these things (if they're still relevant)
Just met with Ramsey & Cormac. We are going to go with 2-schema system as was recommended :)
We'll have the high-level schema (described in T191081#4686318, will follow-up with an actual specification on Meta that incorporates Nuria's feedback) for tracking abandonment rates during upload process.
In addition to that, we will have EventLogging for this particular feature – which will track the individual interactions in the Upload page and when the user is adding captions to existing files. Once we feel that we're done making changes to the UI, we will disable/remove the EL for it. Going forward we will continue to have per-feature EL that supplements the high-level EL. For now, we can remove tracking of all individual interactions that are not relevant to the captions feature.
On the file pages:
On the both the file pages and the Upload Wizard page, we'll want to track the following on a per-caption basis:
@Ramsey-WMF does that capture what we discussed earlier?
After I make the schema spec, I'll follow up with @Cparle to go over it and the instrumentation details.
P.S. Some existing instrumentations that may be useful to refer to are: Search Satisfaction (based on TestSearchSatisfaction2 rev:16909631) and Reading Depth (based on ReadingDepth rev:18201205).
Change 431776 abandoned by Matthias Mullie:
Track interactions
Reason:
Will go in different direction
No longer needed; Captions have been in production for quite some time now and we are measuring usage via statements created.