Page MenuHomePhabricator

[Spike] What schema should be used to track usage of Replying v1.0?
Closed, ResolvedPublic

Description

Context

Situation
In January 2020, the Editing Team will start deploying interventions as part of the Talk pages project. The first of which will be an improved workflow for replying to specific comments on talk pages, across namespaces: Talk pages project/replying.

Later in Q3 (January - March, 2020), the Editing Team also plans to deploy an improved workflow for starting new discussions on talk pages, across namespaces. See this task: T233446 (to be completed in Jan., 2020).

As part of these deployments, the team would like to log talk page and talk page-related events to help understand the impact of these interventions.

Complication
Trouble is, we are not yet sure where the best place is to log these events. Should they be stored in EditAttemptStep? Should a new talk page or communication-specific schema be created? Is there another option that has not yet been discussed?

This ticket is intended to represent the work with answering this question: Where should talk page and talk page-related events be logged?

The beginnings of a document that contains some thoughts David L. surfaced in chat can be found here: Talk pages/schema

"Open questions"

  • Where should full talk page editing events live? In EditAttemptStep? In the new schema?
  • If we proceed with introducing a new schema, what are the different schema those events should be sent to?
  • What is the relationship between new schema and EditAttemptStep?

"Done"

  • Resolve the "Open questions"
  • Determine what schema should be used to log the existing full page talk page editing events
  • Determine what schema should be used to log the new DiscussionTools-related talk page events

References

Relevant links

Replying | wiki/Talk_pages_project/replying

Starting a new discussion | project page to be created

  • Prototype: planed for Feb. - Mar., 2020
  • Measurement plan: planed for Feb. - Mar., 2020

Event Timeline

Updating task description: 13-Dec

  • ADDED: background information about this task, why it is important and relevant links

@DLynch + @MNeisler, if you think the description could be made more clear, please let me know or boldly affect it as you see fit.

ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)
kzimmerman triaged this task as Medium priority.
kzimmerman added a subscriber: kzimmerman.

Peter needs an analyst to lead this decision; unassigning him

Met with Peter and David over this week to finalize project kpi, replying kpi, impact kpi as well as Workflow Engagement metrics. Also discussed workflow events and scenarios that we will need to track.
Topics discussed in 01/17 meeting between DL-PP-MK:

  • What talk page and replying-related events would we like to track?
  • In what schema should these events be logged?
    • DL: seeming more and more like a new schema is necessary; contexts are sufficiently different; dashboarding would become tricky if metrics for talk pages and articles were collapsed into one schema
  • Current talk pages tracking
    • Assume things we are tracking on articles we are tracking on talk pages EXCEPT for VE-related metrics (all happens in Schema:EditAttemptStep - Meta)
    • Mostly this is just EditAttemptStep. People using NWE will also be submitting VisualEditorFeatureUse for any wikitext tools they use.
  • Talk pages/Replying/Instrumentation spec
    • Workflow completion timing
      • Complicated, but potentially not impossible to create a 1:1 comparison between replying workflow and full page b/c full page workflow could – in the case of contributors using classic wikitext editor – involve a contributor visiting 3 distinct pages. Then becomes difficult to work out when exactly the session started
      • Also: more thought needs to be put to when exactly the workflow starts in both experience (full page and new replying workflow)

Peter has scheduled meeting with Jason, Mikhail, David L., Maya on 01/21 for schema discussions:
Agenda

  • Where should all talk page, and talk page-related events, should be logged?
  • Who and how will we document, implement and test talk page instrumentation?
  • Bonus: how might we be able to better understand the kinds of edits contributors are making using full talk page editing?
    • E.g. how do we tell whether a contributor is replying to a comment or fixing a typo?

Notes

  • Talk page schema requirements:
    • Reliable
    • Extensible
    • Maintainable
    • Contextual (e.g. "editing" a talk page is diff. than "editing" an article)
  • Considerations
    • We'd like to have the ability to dashboard talk page metrics

Meeting notes recorded in this document

21-Jan update

Notes from today's meeting below...

Decisions

  • We will experiment with logging the new DiscussionTools-related events in a new schema

Resulting actions

Open questions

  • Where should full talk page editing events live? In EditAttemptStep? In the new schema? Some other configuration?
  • If we proceed with introducing a new schema, what are the different schema those events should be sent to?
  • What is the relationship between new schema and EditAttemptStep?

Full meeting notes: Meeting/Product Analytics ⇄ Editing

ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)

24-Jan Update

Met with David and Peter and to understand how the EditAttemptStep schema can be used for Replying Talk pages.
Will discuss next week during All Hands or after, regarding the new talk page schema.

Action
Which events in the events tab of the instrumentation spec are deserving of a schema that is distinct from EditAttemptstep?

06-Feb Update

Met with @DLynch over the last week to discuss, (focussing only on the current release V1.0) which events need to be instrumented and which schema they will live. We have come to the conclusion that for the features being implemented as a part of Replying V1.0, we will not need a new schema and everything we need can be tracked using EditAttemptStep.
T243363 is not a blocker for V1.0

Please note that as we move forward with the release of V1.1 and V2.0, we will discuss and may require a new schema.

JTannerWMF updated the task description. (Show Details)
JTannerWMF added a subscriber: JTannerWMF.

This spike can be closed out. I migrated the open questions to T243363. The original use of this task was to determine if for 1.0 we needed to log events in EditAttemptStep or a new schema. We made that decision. The work related to creating the new schema should be captured in the aforementioned task.

ppelberg renamed this task from [Spike] What schema should be used to track talk page usage? to [Spike] What schema should be used to track usage of Replying v1.0?.Feb 19 2020, 8:31 PM

19-February

Documenting the answers to our remaining open questions...

  • Where should full talk page editing events live? In EditAttemptStep? In the new schema?
    • For Replying v1.0, all events will be logged in EditAttemptStep. See task: T243467
  • If we proceed with introducing a new schema, what are the different schema those events should be sent to?
    • We do not need to answer this question for v1.0. Instead, we will answer this question as part of v2.0. That work will happen in this task: T243363
  • What is the relationship between new schema and EditAttemptStep?
    • This will be determined as part of the work to complete his tasks T243363
ppelberg updated the task description. (Show Details)