Page MenuHomePhabricator

Test that clicktracking is collecting the data we need
Closed, ResolvedPublic

Description

Let's test that we are collecting the data we need through clicktracking:

  • Edit CTA clicks
  • Edit CTA impressions
  • Learn more CTA clicks
  • Learn more CTA impressions
  • Edit saves
  • Edit attempts
  • Feedback link clicks (Option C, vertical button)

This is based on this requirement:


Version: unspecified
Severity: blocker

Details

Reference
bz32992

Event Timeline

bzimport raised the priority of this task from to Unbreak Now!.Nov 22 2014, 12:03 AM
bzimport added a project: ArticleFeedbackv5.
bzimport set Reference to bz32992.

Here are the action items we discussed towards this goal:

  1. Reha will look at the clicktracking DB on prototype (with help from Roan if needed) to see if all the above data are being tracked as intended
  1. Reha will add any above data items that are not being tracked
  1. Reha will let us know by email what items are being tracked and what their labels are, as well as post this info on this ticket.

Here are some additional action items that will be needed on this issue:

  1. Dario will then check the the labels and data, and make sure that they are usable for data/metrics analysis
  1. We may need to separate the tables so that they do not interfere with AFTv4 data collection (Dario, Roan and Reha to figure out offline)
  1. The DB label for this clicktracking data also needs to be added to the AFTv5 tables

See data/metrics plan for more info on these issues.
http://meta.wikimedia.org/wiki/Research:Article_feedback/Data_and_metrics

This is the complete list of events/UI elements for which we need to collect clicktracking data. Note that this only covers code released as of the 1.0 launch. Please let me know off list if you have any questions, if the list is ok I will start adding labels for each of these events.

Feedback button

  • feedback button clicks

AFT5 Option 1 (Yes/No), bottom

  • help button clicks
  • terms of use link clicks
  • yes button clicks
  • no button clicks
  • submit button clicks

AFT5 Option 1 (Yes/No), overlay

  • impressions (overlay only)
  • help button clicks
  • disable button clicks (overlay only)
  • terms of use link clicks
  • yes button clicks
  • no button clicks
  • submit button clicks

AFT5 Option 2 (4 tabs), bottom

  • help button clicks
  • terms of use link clicks
  • tab-1 clicks
  • tab-2 clicks
  • tab-3 clicks
  • tab-4 clicks
  • submit button clicks

AFT5 Option 2 (4 tabs), overlay

  • impressions (overlay only)
  • help button clicks
  • disable button clicks (overlay only)
  • terms of use link clicks
  • tab-1 clicks
  • tab-2 clicks
  • tab-3 clicks
  • tab-4 clicks
  • submit button clicks

AFT5 Option 3 (Rating), bottom

  • help button
  • terms of use link
  • rating clicks (1 event per rating, I don't care about individual stars)
  • submit button

AFT5 Option 3 (Rating), overlay

  • impressions (overlay only)
  • help button
  • disable button (overlay only)
  • terms of use link
  • rating clicks (1 event per rating, I don't care about individual stars)
  • submit button

CTA-1 (Edit), bottom

  • impressions
  • help button
  • learn how to edit link clicks
  • Edit this page button clicks
  • Edit save attempts
  • Edit save completed
  • feedback saved backlink clicks

CTA-1 (Edit), overlay

  • impressions
  • help button
  • disable button (overlay only)
  • learn how to edit link clicks
  • edit this page button clicks
  • edit save attempts
  • edit save completed
  • feedback saved backlink clicks

CTA-2 (Learn more), bottom

  • impressions
  • help button
  • learn how to edit button clicks
  • feedback saved backlink clicks

CTA-2 (Learn more), bottom

  • impressions
  • help button
  • disable button (overlay only)
  • learn how to edit button clicks
  • feedback saved backlink clicks

Thanks, Dario, this is very thorough!

Yoni or Reha, do these requests work for you? Anything in there seem like a show-stopper for 1.0 launch?

I only have a couple comments on this list, which looks pretty comprehensive to me:

  1. Should we (can we) track clicks in the text area of the forms?
  1. CTA2's button is 'Learn more' -- not 'Learn to Edit' -- and we may not need to track the feedback-saved backlink clicks for it, since it doesn't take you to the edit page, just to the tutorial
  1. Dario, when preparing label names for OmniTI to use for this, I recommend using words that are very close to what the buttons or UI actually say, for clarity (e.g. 'Post' instead of 'Submit', etc.)

Other than that, I think you have covered the key data we are looking to track, thanks!

Fabrice, here's a few comments

  1. nice to have, definitely not a show-stopper
  1. that's correct, it's confusing that we have a different wording for CTA1 and CTA2, but I'll stick to whatever language we use. We do need to track feedback-saved backlink clicks as they are one of the possible exit points from the CTA. Just to clarify I am referring to the link that appears in the "Your feedback has been saved" – I am not sure what that currently links to.
  1. agreed

the last heading in the above list should actually read:

CTA-2 (Learn more), overlay

Here's a shorter, high-priority list of events if the above list is too long to be implemented before the 1.0 deployment:

Feedback button

  • feedback button clicks

AFT5 Option 1 (Yes/No), bottom

  • submit button clicks

AFT5 Option 1 (Yes/No), overlay

  • impressions (overlay only)
  • disable button clicks (overlay only)
  • submit button clicks

AFT5 Option 2 (4 tabs), bottom

  • submit button clicks

AFT5 Option 2 (4 tabs), overlay

  • impressions (overlay only)
  • disable button clicks (overlay only)
  • submit button clicks

AFT5 Option 3 (Rating), bottom

  • submit button

AFT5 Option 3 (Rating), overlay

  • impressions (overlay only)
  • disable button (overlay only)
  • submit button

CTA-1 (Edit), bottom

  • impressions
  • edit this page button clicks
  • edit save attempts
  • edit save completed

CTA-1 (Edit), overlay

  • impressions
  • disable button (overlay only)
  • edit this page button clicks
  • edit save attempts
  • edit save completed

CTA-2 (Learn more), bottom

  • impressions
  • learn how to edit button clicks

CTA-2 (Learn more), overlay

  • impressions
  • disable button (overlay only)
  • learn how to edit button clicks

reha wrote:

The short list is nearly done (as of r106251)

Here are the tracking ids I've selected (excluding the prefix to designate it for this version of AFTv5). I'm operating under the assumption that because you listed the overlay and bottom states separately, you wanted them tracked separately. Please let me know if I should make any of these share a tracking id.

Feedback button

  • feedback button clicks - 'vertical-link'
    • also 'toolbox-link' for clicks on the toolbar link (this was already being tracked)

AFT5 Option 1 (Yes/No), bottom

  • submit button clicks - 'option1-submit'

AFT5 Option 1 (Yes/No), overlay

  • impressions (overlay only) - 'option1-impression-overlay'
  • disable button clicks (overlay only) - 'option1-overlay-close'
  • submit button clicks - 'option1-submit-overlay'

AFT5 Option 2 (4 tabs), bottom

  • submit button clicks - 'option2-submit-bottom'

AFT5 Option 2 (4 tabs), overlay

  • impressions (overlay only) - 'option2-impression-overlay'
  • disable button clicks (overlay only) - 'option2-overlay-close'
  • submit button clicks - 'option2-submit'

AFT5 Option 3 (Rating), bottom

  • submit button - 'option3-submit-bottom'

AFT5 Option 3 (Rating), overlay

  • impressions (overlay only) - 'option3-impression-overlay'
  • disable button (overlay only) - 'option3-overlay-close'
  • submit button - 'option3-submit-overlay'

CTA-1 (Edit), bottom

  • impressions - 'cta-edit-impression-bottom'
  • edit this page button clicks - 'cta-edit-button-click-bottom'
  • edit save attempts - NOT YET WORKING
  • edit save completed - NOT YET WORKING

CTA-1 (Edit), overlay

  • impressions - 'cta-edit-overlay-impression'
  • disable button (overlay only) - 'cta-edit-overlay-close'
  • edit this page button clicks- 'cta-edit-button-click-overlay'
  • edit save attempts - NOT YET WORKING
  • edit save completed - NOT YET WORKING

CTA-2 (Learn more), bottom

  • impressions - 'cta-learn-more-bottom-impression'
  • learn how to edit button clicks - 'cta-learn-more-button-click-bottom'

CTA-2 (Learn more), overlay

  • impressions - 'cta-learn-more-overlay-impression'
  • disable button (overlay only) - 'cta-learn-more-overlay-close'
  • learn how to edit button clicks- 'cta-learn-more-button-click-overlay'

reha wrote:

Tweaks to naming in r106267: Added "optionX-" to the front of all the cta events, and to the feedback links

General style is now [option identifier]-([cta identifier]-)[event]-[bottom/overlay]

Copying here for the record some important tweaks that Reha and I just
discussed over IRC:

  • vertical-link events should be prefixed with the option identifier
  • all CTA events should be prefixed with the option and the placement

identifier

The general naming scheme we'll be using is the following:

'ext.articleFeedbackv5@' + version + '-' + key

with version as the version number set in the configuration and key the event
identifier:

  • [option id]-[event]-[placement id]
  • [option id]-[cta id]-[event]-[placement id]

reha wrote:

This should be fixed now. Complete list of tracking ids:

option1-submit-overlay
option2-submit-overlay
option3-submit-overlay
option1-submit-bottom
option2-submit-bottom
option3-submit-bottom
option1-impression-overlay
option2-impression-overlay
option3-impression-overlay
option1-cta_edit-impression-overlay
option2-cta_edit-impression-overlay
option3-cta_edit-impression-overlay
option1-cta_learn_more-impression-overlay
option2-cta_learn_more-impression-overlay
option3-cta_learn_more-impression-overlay
option1-cta_edit-impression-bottom
option2-cta_edit-impression-bottom
option3-cta_edit-impression-bottom
option1-cta_learn_more-impression-bottom
option2-cta_learn_more-impression-bottom
option3-cta_learn_more-impression-bottom
option1-close-overlay
option2-close-overlay
option3-close-overlay
option1-cta_edit-close-overlay
option2-cta_edit-close-overlay
option3-cta_edit-close-overlay
option1-cta_learn_more-close-overlay
option2-cta_learn_more-close-overlay
option3-cta_learn_more-close-overlay
option1-cta_learn_more-button_click-overlay
option2-cta_learn_more-button_click-overlay
option3-cta_learn_more-button_click-overlay
option1-cta_learn_more-button_click-bottom
option2-cta_learn_more-button_click-bottom
option3-cta_learn_more-button_click-bottom
option1-cta_edit-button_click-overlay
option2-cta_edit-button_click-overlay
option3-cta_edit-button_click-overlay
option1-cta_edit-button_click-bottom
option2-cta_edit-button_click-bottom
option3-cta_edit-button_click-bottom
option1-cta_edit-edit_attempt-overlay
option2-cta_edit-edit_attempt-overlay
option3-cta_edit-edit_attempt-overlay
option1-cta_edit-edit_attempt-bottom
option2-cta_edit-edit_attempt-bottom
option3-cta_edit-edit_attempt-bottom
option1-cta_edit-edit_success-overlay
option2-cta_edit-edit_success-overlay
option3-cta_edit-edit_success-overlay
option1-cta_edit-edit_success-bottom
option2-cta_edit-edit_success-bottom
option3-cta_edit-edit_success-bottom

*NB: Resolved in code; not yet pushed.

reha wrote:

Added:

option1-impression-bottom
option2-impression-bottom
option3-impression-bottom

Added at random 1% sampling:

option1-init
option2-init
option3-init

Both in r108063.

It appears that the following events are not being captured at all in clicktracking.log, unless their name is different from the one specified here and at: http://meta.wikimedia.org/wiki/Research:Article_feedback/Clicktracking#Full_list_of_events

option1-cta_edit-button_click-overlay
option2-cta_edit-button_click-overlay
option3-cta_edit-button_click-overlay
option1-cta_edit-button_click-bottom
option2-cta_edit-button_click-bottom
option3-cta_edit-button_click-bottom

Counting these events is critical to measure the click-through and completion rate for the edit CTA.

reha wrote:

Found and fixed this bug in r111179; tested locally and on prototype.

I noticed while I was working on it that although the Learn More CTA's button clicks are tracked, the Edit CTA's learn how to edit link is not. Dario's draft list (comment #3) includes it, but his priority revision (comment #7) does not. Is this correct? Are we ready to start tracking it now.

Thanks for the fix, Reha.

"Learn how to edit" is not as critical as the edit button clicks, but it would be nice to have if it's simple to enable it.

Merged to both 1.18wmf1 and 1.19wmf1 in r111506

Pushed to site in 1.18wmf1 config

Added clicktracking requirements for phase 2
http://meta.wikimedia.org/wiki/Research:Article_feedback/Clicktracking#Phase_2_.28placement.29

New bucket definitions:

  • option1A: trigger A + widget in the overlay (bottom widget is still displayed)
  • option1E: trigger E + widget in the overlay (bottom widget is still displayed)
  • option1X: no trigger + widget at the bottom of the page only

List of events

option1A-init (1%)
option1E-init (1%)
option1X-init (1%)

option1A-impression-overlay
option1E-impression-overlay

option1A-impression-bottom
option1E-impression-bottom
option1X-impression-bottom

option1A-submit-overlay
option1E-submit-overlay

option1A-submit-bottom
option1E-submit-bottom
option1X-submit-bottom

option1A-cta_edit-impression-overlay
option1E-cta_edit-impression-overlay

option1A-cta_learn_more-impression-overlay
option1E-cta_learn_more-impression-overlay

option1A-cta_edit-impression-bottom
option1E-cta_edit-impression-bottom
option1X-cta_edit-impression-bottom

option1A-cta_learn_more-impression-bottom
option1E-cta_learn_more-impression-bottom
option1X-cta_learn_more-impression-bottom

option1A-close-overlay
option1E-close-overlay

option1A-cta_edit-close-overlay
option1E-cta_edit-close-overlay

option1A-cta_learn_more-close-overlay
option1E-cta_learn_more-close-overlay

option1A-cta_learn_more-button_click-overlay
option1E-cta_learn_more-button_click-overlay

option1A-cta_learn_more-button_click-bottom
option1E-cta_learn_more-button_click-bottom
option1X-cta_learn_more-button_click-bottom

option1A-cta_edit-button_click-overlay
option1E-cta_edit-button_click-overlay

option1A-cta_edit-button_click-bottom
option1E-cta_edit-button_click-bottom
option1X-cta_edit-button_click-bottom

option1A-cta_edit-edit_attempt-overlay
option1E-cta_edit-edit_attempt-overlay

option1A-cta_edit-edit_attempt-bottom
option1E-cta_edit-edit_attempt-bottom
option1X-cta_edit-edit_attempt-bottom

option1A-cta_edit-edit_success-overlay
option1E-cta_edit-edit_success-overlay

option1A-cta_edit-edit_success-bottom
option1E-cta_edit-edit_success-bottom
option1X-cta_edit-edit_success-bottom

option1A-triggerA-click-overlay
option1E-triggerE-click-overlay

option1A-edit_tab_link-click
option1E-edit_tab_link-click
option1X-edit_tab_link-click

option1A-section_edit_link-click
option1E-section_edit_link-click
option1X-section_edit_link-click

option1A-edit_tab_link-edit_attempt
option1E-edit_tab_link-edit_attempt
option1X-edit_tab_link-edit_attempt

option1A-section_edit_link-edit_attempt
option1E-section_edit_link-edit_attempt
option1X-section_edit_link-edit_attempt

option1A-edit_tab_link-edit_success
option1E-edit_tab_link-edit_success
option1X-edit_tab_link-edit_success

option1A-section_edit_link-edit_success
option1E-section_edit_link-edit_success
option1X-section_edit_link-edit_success

All events used in Phase 1 should be disabled

yoni wrote:

reopening by Derio's request.
Reha: implement the rest of the requirements.

Added the following six events to the specs for Phase 2:

/* clicks on the x button */
option1A-disable_button_click
option1E-disable_button_click

/* impressions of the flyover */
option1A-disable_flyover-impression
option1E-disable_flyover-impression

/* clicks on the go to preferences button from the flyover */
option1A-disable_gotoprefs_click
option1E-disable_gotoprefs_click

Unless it's trivial, I don't think we need to count successfully saved changes in user preferences: logging the above events should be sufficient to estimate how many registered users have the intention to disable AFT.

I updated the specs on Meta accordingly
http://meta.wikimedia.org/wiki/Research:Article_feedback/Clicktracking#Phase_2_.28placement.29

reha wrote:

Phase 2 clicktracking (including the disable flyover) checked in -- r114189

Phase 3 clicktracking specs added:

http://meta.wikimedia.org/wiki/Research:Article_feedback/Clicktracking#Stage_3_.28Impact_on_engagement.29

I prepared these specs based on a number of (partly arbitary) assumptions on the implementation of the Option4 fallback for non-editable pages. All these assumptions will need to be reviewed before we start adding the tracking codes.

reha wrote:

Phase 3 clicktracking committed and submitted to gerrit for review:

https://gerrit.wikimedia.org/r/5141
https://gerrit.wikimedia.org/r/5142

reha wrote:

Discussed this with Dario -- it makes more sense to keep clicktracking in just one place and only open a bug if something's gone wrong.

From here on out, clicktracking *requirements* go here:

http://meta.wikimedia.org/wiki/Research:Article_feedback/Clicktracking

Closing.