Page MenuHomePhabricator

Determine what – if any – new events need to be added to the existing section=new workflow
Closed, ResolvedPublic

Description

This task is about determining what – if any – new events need to be added to the existing section=new workflow order to analyze the metrics these tickets are asking for:

Background

T265100 explains what events in the existing section=new workflow are being logged.

New events

  • A new event is needed to differentiate between edits to existing sections and the creation of new sections. The work to implement this event will happen in: T272544.

Done

  • Determine whether new events need to be added to the existing section=new workflow order to analyze the metrics T263053 and T263054 is asking for.
  • If new events need to be added, list them in the ===New events section above

Event Timeline

ppelberg moved this task from Backlog to Triaged on the DiscussionTools board.Dec 12 2020, 3:35 AM
MNeisler triaged this task as Medium priority.Dec 21 2020, 4:50 PM
MNeisler added a project: Product-Analytics.
MNeisler moved this task from Triage to Upcoming Quarter on the Product-Analytics board.

@DLynch - I have a follow-up question regarding the behavior of the init_mechanism field described in T265100#6618083.

WikiEditor:

  • If there's no HTTP referer, it'll always be url
  • If the referer is present it'll be new if section=new or click if section=[some-number]

Is it possible for wikitext section edits with no HTTP referrer (where editor_interface = wikitext, init_type = section, init_mechanism = url) to include edits to start a new section?

Context: I'm currently reviewing the current instrumentation to confirm if we are able to distinguish between talk page edits to start a new section using the section=new workflow versus edits to an already created section.

ppelberg reassigned this task from MNeisler to DLynch.Wed, Jan 13, 8:15 PM
ppelberg added a subscriber: MNeisler.

Assigning this over to @DLynch to respond to the question @MNeisler posed above.

Is it possible for wikitext section edits with no HTTP referrer (where editor_interface = wikitext, init_type = section, init_mechanism = url) to include edits to start a new section?

That's currently what could happen, yes. Init events with type=section and mechanism=url are going to be a mixture of new and existing sections. Intuitively I'd expect mechanism=url to be comparatively uncommon, on the assumption that most ways we'd create a new section involve just clicking the link on the page.

This predates my involvement in the project, but I'd speculate that this particular logging was never intended to let us determine this particular bit of data. If the mechanism=url events are common, we could certainly add a new mechanism variant that would distinguish it while still preserving the "this was direct URL access" semantics.

(Basically, it's very possible to make changes so that init_mechanism is more consistent between the various implementing editors. I haven't touched it because I've never been assigned a task which directly involved it, and changing the semantics randomly without knowing whether any ongoing analysis depends on these differences seemed like a bad idea...)

MNeisler claimed this task.Tue, Jan 19, 6:07 PM
MNeisler added a comment.EditedWed, Jan 20, 1:17 AM

Thanks for the clarification @DLynch.

Intuitively I'd expect mechanism=url to be comparatively uncommon, on the assumption that most ways we'd create a new section involve just clicking the link on the page.

I did a quick query of the number of wikitext section edits by init_mechanism from November through December 2020. init_mechanism = url events are the least common type (9.92% of all wikitext section edits) but given current instrumentation, there's no way to tell what percent of that is starting a new section vs editing a current section. I agree it's probably uncommon though.

editing_interfaceinit_mechanismedits% of edits by editor interfaece
Wikitextclick515273178.85%
Wikitexturl6479699.92%
Wikitextnew73417711.23%

If the mechanism=url events are common, we could certainly add a new mechanism variant that would distinguish it while still preserving the "this was direct URL access" semantics.

Based on my understanding, existing instrumentation can be used to identify all new section events created using the new discussion tool. Those events will be recorded as action = init , integration = discussiontools. and init_type = section and should only be recorded when a new section is started.

However, we are unable to identify new section events created using the existing workflow because the init_mechanism behavior is inconsistent. This is needed if we want to make any comparisons between the two workflows and to accurately calculate metrics such as Adoption metric #4 in T263053 (how many people used the New Discussion Tool to add X proportions of new topics ).

Based on this, I think it would make sense to go ahead and add a new mechanism to distinguish new section events if feasible. (If not, we will need to add a caveat to data on the existing new section workflow that it does not include new section edits where there is no HTTP referrer)

Notes from the conversation @MNeisler and I had today...

Based on this, I think it would make sense to go ahead and add a new mechanism to distinguish new section events if feasible. (If not, we will need to add a caveat to data on the existing new section workflow that it does not include new section edits where there is no HTTP referrer)

Let's do what Megan is suggesting above and what @DLynch alluded to doing in T269486#6746576. This work will happen in T272544.


Also @MNeisler, we did implement a revision tag for edits made with the New Discussion Tool. This work happened in T260467. See: the New Topic and Visual tags associated wit this edit: https://en.wikipedia.beta.wmflabs.org/w/index.php?title=Talk%3ACats&type=revision&diff=471357&oldid=470814.

ppelberg updated the task description. (Show Details)Wed, Jan 20, 9:06 PM

Done

  • Determine whether new events need to be added to the existing section=new workflow order to analyze the metrics T263053 and T263054 is asking for.

Following the investigation Megan did, the conversation she and I had today and me double confirming a revision tag has been created for edits made with the New Discussion Tool [i], we can confidently say T272544 is the only instrumentation that is currently needed for T263053.

As for T263054, Megan and I agreed that we are going to revisit the scope of that task once T269471 is resolved. At that point, we can determine whether additional instrumentation is necessary.

  • If new events need to be added, list them in the ===New events section above

Done per T269486#6763524. The task description's ===New events section has been updated.


i. T269486#6763524

For reference, I started a new discussion tool instrumentation spec where I mapped current metrics outlined in T263053 and T263054 to existing instrumentation. As @ppelberg mentioned, this can be updated and revised when we revisit the workflow engagement scope.

ppelberg closed this task as Resolved.Sat, Jan 23, 2:50 AM

For reference, I started a new discussion tool instrumentation spec where I mapped current metrics outlined in T263053 and T263054 to existing instrumentation.

Oh, this is helpful.