Post-deployment data QA
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ppelberg
	Mar 7 2020, 2:00 AM

Description

T245794 will make the V1.0 of the new Replying workflow available to contributors in production.

This task is about making sure the events we instrumented as part of T243364 are ending up in the database in the ways we expect.

Timing

This task can now be worked considering T245794 is resolved. Read: the feature is now available on our partner wikis as a Beta Feature.

Done

Document whether the events defined in the instrumentation spec [1] are landing the database in the ways we expect

Google Doc (no sign in required): https://docs.google.com/spreadsheets/d/1iMWKmT7nHimjaa7OItGINoTVGtnaIy86I3ICj0RN5KY/edit#gid=1016912708

Related Objects
Search...

Status	Assigned	Task
Open	None	T233443 [Epic] Reply Tool
Resolved	ppelberg	T244500 Instrument new replying workflow
Resolved	Mayakp.wiki	T247152 Post-deployment data QA

Event Timeline

ppelberg created this task.Mar 7 2020, 2:00 AM

ppelberg updated the task description. (Show Details)

ppelberg moved this task from Needs Prioritization to Required after release on the OWC2020 (OWC2020 Replying 1.0) board.Mar 9 2020, 6:45 PM

Mayakp.wiki edited projects, added Product-Analytics (Kanban); removed Product-Analytics.Mar 10 2020, 5:31 PM

ppelberg moved this task from Needs Discussion / Investigation to Upcoming on the Editing-team (Q3 2019-2020 Kanban Board) board.Mar 11 2020, 4:26 PM

Meeting notes: 11-March
Notes from Maya's and my conversations today:

We are planning to use the same queries to complete this task as we did for T243364 with the following changes:
- Whether a person has turned on DiscussionTools in Beta Features

JTannerWMF edited projects, added Editing-team (Tracking); removed Editing-team (Q3 2019-2020 Kanban Board).Mar 11 2020, 8:36 PM

JTannerWMF moved this task from Backlog to Analytics on the Editing-team (Tracking) board.

In T245794#5965664, @ppelberg wrote:

...As discussed during today's stand-up, v1.0 of the new Replying feature should be deployed:

As a Beta Feature

This upcoming Monday, 16-March

To the following Wikipedias: Arabic, Dutch, French and Hungarian

...the task description has been updated to include this information.

ppelberg set Due Date to Mar 27 2020, 7:00 AM.Mar 12 2020, 8:56 PM

@ppelberg I understand this may be delayed due to delayed deployments - can you update the due date & other information to reflect that?

Mayakp.wiki moved this task from Triage to Needs Investigation on the Product-Analytics board.Mar 17 2020, 5:32 PM

In T247152#5976608, @kzimmerman wrote:

@ppelberg I understand this may be delayed due to delayed deployments - can you update the due date & other information to reflect that?

Absolutely – I've made the following changes (below) to this task. Thank you for the ping, Kate.

Task changes:

REMOVED: task due date
ADDED: deployment timing to task description

Thank you! I'll move it to our backlog for now, then.

ppelberg mentioned this in T247138: Workflow engagement check: Replying v1.0.Mar 18 2020, 7:36 PM

ppelberg edited projects, added Editing-team (Q3 2019-2020 Kanban Board); removed Editing-team (Tracking).Mar 26 2020, 4:09 PM

Update: 26-March
Deployment is scheduled for this upcoming Tuesday, 31-March, assuming no critical issues surface in QA which will finish Monday, 30-March.

Mayakp.wiki moved this task from Backlog to Upcoming Quarter on the Product-Analytics board.Mar 26 2020, 5:03 PM

Updating task description to reflect yesterday's deployment and resolution of T245794, the task that had been blocking work on this one.

JTannerWMF edited projects, added Editing-team (Tracking); removed Editing-team (Q3 2019-2020 Kanban Board).Apr 2 2020, 12:29 AM

Mayakp.wiki edited projects, added Product-Analytics (Kanban); removed Product-Analytics.Apr 4 2020, 12:06 AM

Mayakp.wiki moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.Apr 9 2020, 10:51 PM

Increase Sampling Rate of DiscussionTools edits :

While doing the data QA we have observed that we are getting very few ‘discussion tools’ related events in the EditAttemptStep schema. This is possibly because the Replying V1.0 feature is deployed on 4 small wikis - nl, hu, ar, fr as an opt-in feature and the schema is sampled at 1/16 rate.
These are all the events on each wiki from Mar 31 to Apr 10.

Screen Shot 2020-04-13 at 5.06.49 PM.png (280×628 px, 23 KB)

My request is to oversample all DiscussionTools events to 1/5.

**Note: Given the impact, I believe that this should not cause a huge spike in eventlogging. However if we need to increase or decrease the sampling we can do so by changing the config override T250086

Mayakp.wiki mentioned this in T250086: Add config override for instrumentation sampling rate.Apr 14 2020, 12:22 AM

Next steps
Next steps from the conversations Maya and I had today:

@Mayakp.wiki to provide update about how the deployment of T250086 has affected the number of events being logged in the DB.

QC Document link Data looks correct for the most part.

Outputs and Screenshots are provided in a separate tab for reference.
Note: I did expect a big rise in the number of events after the sampling rate was changed to 1/5. However, that doesn't seem to be the case. With that said, it does seem sufficient to do our analysis. I will monitor the events even after this task is closed.

	F31755706: Screen Shot 2020-04-13 at 5.06.49 PM.png
	Apr 14 2020, 12:08 AM

Post-deployment data QAClosed, ResolvedPublicActions