Page MenuHomePhabricator

Post Deployment Data QA - Collapsible Sidebar Instrumentation
Closed, ResolvedPublic

Description

This will be started once the instrumentation is deployed on production i.e. all target wikis (listed here https://www.mediawiki.org/wiki/Reading/Web/Desktop_Improvements#Test_Wikis)

All scenarios and QA status will be reflected in this document : https://docs.google.com/spreadsheets/d/1GtlAas8wBD4-J03nqwZlFaBJ0TYbhv-ORWheho4LCQs/edit#gid=0

Event Timeline

I have done some preliminary checks on the data and posted results here : https://github.com/wikimedia-research/Collapsible-Sidebar-Analysis/blob/master/Post%20Deployment%20Data%20QA%20for%20Collapsible%20Sidebar%20Instrumentation.ipynb

Per @Jdlrobson on Slack, the instrumentation has not been deployed on frwiki, hewiki, fawiki.

I will keep updating this notebook with data-QA results over the next few weeks.

Checked the data in Schema:DesktopWebUIActionsTracking and everything looks good. Events from test wikis are in the schema as expected.
Results posted in this notebook.

Summary:

  • Ideally, since this instrumentation is deployed to test wikis that have the collapsible sidebar option, we would expect events for new version to be higher than legacy version. However, skinVersion 1 is higher than version 2, probably due to the late rollout of Fr wiki (which is the largest test wiki) .
  • We can see a rise in the clicks to sidebar button starting July 27, when the new skin version was rolled out to larger test wikis Fa and He and then stabilizes.
  • There is a rise on Aug 5 when the feature was rolled out to Fr wiki. We expect this rise to stabilize over time. This also indicates that Eventlogging is able to capture the surge in events correctly.
  • The Collapsible Sidebar is 'open' ie 'uncollapsed' by default and not persistent for Anonymous (logged out) users. This explains why the # of events for isSidebarCollapsed = False is greater than isSidebarCollapsed=True
  • Donate link Clicks by skinVersion
    • Looks like the new Vector skinVersion is driving more traffic (mostly from Anonymous i.e. logged out users) toward the Donate link on Fa, Eu, He wikis ! !

@MNeisler : can you please review this notebook and the results whenever possible and let me know if we need any additional tests ? I will close this task after your review. Thanks a lot for your help !

@Mayakp.wiki

I finished my review of the post-deployment QA checks in your notebook. Everything looked good to me and I don't have any major changes or questions. Thanks so much for the QA support on this!

I summarized just a few additional notes/observations from my review below:

  • Number of events: I separated the charts to review clicks and inits separately to see the trends for each more clearly. Events look as expected and increases correlate with the deployment dates to each wiki.
  • Number of events by skin version: I reran the numbers through the end of August and confirmed that the number of events for skin version 2 are higher than skin version 1 on test wikis as expected. The numbers for version 2 were initially lower when you first ran the query in early August due to the late rollout of French Wikipedia as you indicated in the report.
  • Sidebar clicks: I ran a check to make sure that the number of events did not change when you add the condition, event.action = 'click' , to the query. I confirmed no init events were recorded for these sidebar events as expected.
  • Donate clicks: I ran the same check as above and confirmed no init events were recorded for these events as expected. Really interesting that there are more donate click events recorded with the new skin Version. As you noted, it looks like this was initially driven by anonymous users on Persian and Hebrew wikis. You can see a sharp increase in events on July 28th right after deployment to those wikis, which then stabilizes around July 31st. I'll explore this more in the sidebar analysis [T245735] to confirm if this trend continued or was just caused by this one spike.

@ovasileva - Going ahead and marking this as resolved but let us know if you have any additional questions.