Page MenuHomePhabricator

[SPIKE] What events are being tracked within the "section=new" flow
Closed, ResolvedPublic

Description

This task is about understanding what events in, and metadata about, the section=new flow [i] are currently being tracked.

Background

One way we will evaluate the effectiveness of the New Discussion Tool is to look at whether people who use this new tool/workflow are more or less likely to publish new discussion topics than the people who use the existing workflow. [i]

In order to draw the comparison described above, we'll need to ensure the existing section=new workflow is instrumented sufficiently.

This task is about understanding how it is currently instrumented so we can decide what – if any – adjustments needs to be made to it.

Instrumentation documentation

For each of the three editing interfaces – WikiEditor, VisualEditor NWE, VisualEditor (visual) – the same events that are tracked in the article editing context are tracking in the section=new context with the key difference being the metadata that accompanies these events.

For all options, events will be logged to EditAttemptStep and VisualEditorFeatureUse. The only practical difference from article-editing is that EditAttemptStep init event will be logged with init_type=section whenever there's a section parameter in the URL.

There's inconsistent behavior of the EditAttemptStep init_mechanism` field:

  • WikiEditor:
    • If there's no HTTP referer, it'll always be url
    • If the referer is present it'll be new if section=new or click if section=[some-number]
  • VisualEditor:
    • If you clicked on the new section tab and it dynamically loaded it'll be click
    • If you clicked on the new section tab and it navigated to the new page it'll be url
  • It'll never be new
  • Visual / NWE will log all their normal VisualEditorFeatureUse events.

There's precisely one VisualEditorFeatureUse event logged from WikiEditor, which is editor-switch; visual-desktop` for when you're switching away to VisualEditor. This should only be possible to trigger if you're doing section=new on a non-Talk page, considering the "visual" VisualEditor is not available in talk namespaces.

Done

  • Document what – if any – events the current section=new workflow is emitting in the ===Instrumentation documentation section above
    • Assuming some events are being emitted, what are these events called and what schema are they being logged in?
  • Document what pieces of metadata are being logged about the section=new form in the ===Instrumentation documentation section above
  • Determine what – if any – additional instrumentation will need to be added to the existing workflow to complete the analyses T263053 and T263054 are asking for.
    • Note: work on this is now going to happen in T269486.

i. E.g. https://en.wikipedia.org/w/index.php?title=User_talk:PPelberg_(WMF)&action=edit&section=new

Event Timeline

Discussed in meeting: it's okay to summarize the table, since there's going to be a lot of VisualEditorFeatureUse events which are common to all integrations.

So! The answer here is "it depends", because there's several variants we could be talking about.

  • WikiEditor
  • VisualEditor NWE
  • VisualEditor (visually) -- which we can ignore, if we're only considering talk namespaces, but section=new does work outside of them

However, the summary for all of them is "you get exactly the events you get when editing an article, but with some metadata changed to reflect that you're in section mode".

For all options, events will be logged to EditAttemptStep and VisualEditorFeatureUse. The only practical difference from article-editing is that EditAttemptStep init event will be logged with init_type=section whenever there's a section parameter in the URL.

There's inconsistent behavior of the EditAttemptStep init_mechanism field:

  • WikiEditor:
    • If there's no HTTP referer, it'll always be url
    • If the referer is present it'll be new if section=new or click if section=[some-number]
  • VisualEditor:
    • If you clicked on the new section tab and it dynamically loaded it'll be click
    • If you clicked on the new section tab and it navigated to the new page it'll be url
    • It'll never be new

Visual / NWE will log all their normal VisualEditorFeatureUse events.

There's precisely one VisualEditorFeatureUse event logged from WikiEditor, which is editor-switch; visual-desktop for when you're switching away to VisualEditor. This should only be possible to trigger if you're doing section=new on a non-Talk page, of course.

Thank you for pulling this together, @DLynch. Before going into what, if anything, needs to be changed about the instrumentation of the existing section=new workflows to meet what will be asked for in T263053 and T263054, I'd like to confirm I am accurately understanding the current state.

With the above in mind, some clarifying questions...

So! The answer here is "it depends", because there's several variants we could be talking about.

  • WikiEditor
  • VisualEditor NWE
  • VisualEditor (visually) -- which we can ignore, if we're only considering talk namespaces, but section=new does work outside of them
  • Can you confirm this diagram accurately describes how events are being logged at a high level? Sometimes, seeing something visually helps me better understand it.
Diagram showing section=new logging
Screen Shot 2020-11-17 at 12.34.38 PM.png (1×2 px, 180 KB)
Link: Editing Team/Sketching

However, the summary for all of them is "you get exactly the events you get when editing an article, but with some metadata changed to reflect that you're in section mode".

  • Would it be accurate for me to understand "section mode" as any context where section=new is present in the URL?

For all options, events will be logged to EditAttemptStep and VisualEditorFeatureUse. The only practical difference from article-editing is that EditAttemptStep init event will be logged with init_type=section whenever there's a section parameter in the URL.

  • Would it be accurate for me to understand the above as follows? "section will always appear in the URL when someone is viewing the new section form, regardless of the affordance they clicked/tapped to cause the new form to be shown (e.g. "New section" tab, manually navigating to the form's URL, clicking a custom call to action (e.g. the "Ask a question" button at en.wiki's Teahouse)?
  • Would it be accurate for me to assume we do not currently have a way of detecting what someone clicked/pressed/tapped to open the section=new form?

Task description update
ADDED the below to the task description's ===Done section:

  • Determine what – if any – additional instrumentation will need to be added to the existing workflow to complete the analyses T263053 and T263054 are asking for.

Can you confirm this diagram accurately describes how events are being logged at a high level? Sometimes, seeing something visually helps me better understand it.

I have to make my own diagram for this, so hold on.

Would it be accurate for me to understand "section mode" as any context where section=new is present in the URL?

Yes, that should be accurate. Even the "I clicked the new section tab and it replaced the page contents" route should update the URL.

Would it be accurate for me to understand the above as follows? "section will always appear in the URL when someone is viewing the new section form, regardless of the affordance they clicked/tapped to cause the new form to be shown (e.g. "New section" tab, manually navigating to the form's URL, clicking a custom call to action (e.g. the "Ask a question" button at en.wiki's Teahouse)?

I believe so. I don't see any routes to it that wouldn't either trigger from the URL or update the URL to contain it, anyway.

Would it be accurate for me to assume we do not currently have a way of detecting what someone clicked/pressed/tapped to open the section=new form?

Nothing fine-grained, certainly. We could make guesses based on the referrer, but that's not reliably present.

@ppelberg to link new discussion measurement ticket for @DLynch to review and determine what if any new events need to be added to the workflow to do those analyses

@ppelberg to link new discussion measurement ticket for @DLynch to review and

The new discussion measurement tickets are linked here:


...determine what if any new events need to be added to the workflow to do those analyses

I've created a new ticket (T269486) where we can determine whether new instrumentation is needed.

In doing so, I am:

  1. Removing the following from this task's ===Done section:
  • Determine what – if any – additional instrumentation will need to be added to the existing workflow to complete the analyses T263053 and T263054 are asking for.
  1. Moving this task to "Ready for sign off." Note: I am leaving the task open until after @MNeisler and @DLynch indicate this ticket (T265100) has the information they need to work on what T269486 is asking for.
  1. Moving this task to "Ready for sign off." Note: I am leaving the task open until after @MNeisler and @DLynch indicate this ticket (T265100) has the information they need to work on what T269486 is asking for.

@MNeisler confirmed in our meeting just now that this ticket provides the information required to answer what's being asked in T269486.