Wed, May 5
@jrobson - Yes, this should be included as part of the UniversalLanguageSelector Schema if possible.
I drafted a list of events that need to be tracked in order to calculate the metrics outlined in the measurement plan. Please see the "Events to be tracked" column in the instrumentation spec document.
Tue, May 4
Fri, Apr 30
Wed, Apr 28
I completed a QA of the events logged from the new schema (mediawiki_pref_diff).
Tue, Apr 27
Mon, Apr 26
Thanks @DLynch. I agree that Option #2 (Sending a VEFU event) seems like the best method. In addition to not requiring any schema modifications, I think that VEFU is a better fit for this type of event and the analysis will still be pretty straightforward (I'll just need to do a join to EditAttemptStep on session id)
@ppelberg - I've reviewed the Reply Tool project page and made some edits [i] to clarify the results.
The deployment for this change is scheduled for this Thursday, April 29th. I will plan to QA following deployment after data has been logged and available to query in SearchSatisfaction.
Thanks @MNeisler -- it looks like the token field will be coming through as an empty string per a recent change in the WikimediaEvents extension. So as it is, token will not be a reliable field to track unique user clicks (it was left in for backwards compatibility).
Fri, Apr 23
@MNeisler: to be doubly sure, would it be more accurate to say, "...we observed a 79.5% decrease in the revert rate for comments Junior Contributors made..." vs. "...we observed a 79.5% decrease in the revert rate for comments made..."?
Here is the updated report.
Wed, Apr 21
Does this test written in such a way that we are getting the kind of (qualitative feedback) that we are not getting in T274215?
@MNeisler, for each of the "Scenarios" in this sheet [i] whose "Priority" has been marked as "HIGH" or "MEDIUM" can you please do the below?
Fri, Apr 16
I confirmed that events for this schema are logged and stored in eventlogging data as event.mediawiki_pref_diff.
Wed, Apr 14
@cjming - Thanks for the ping! If possible, it would be useful to have a user identifier to track the number of users. This will help us understand the frequency of usage across users (i.e. Is it just a few users clicking it frequently or a number of users clicking it once?)
Tue, Apr 13
Pending availability of data in the new desktop preference schema, I reviewed data in mediawiki [user properties table] to determine the percent of logged-in users at all of the pliot wikis (looking at both registered users and active users) that have their vector skin preference changed to legacy as of 13 April 2021. See summary below:
Apr 6 2021
Apr 5 2021
The AB impact analysis report is still in progress but here is an initial look at the number of search sessions initiated by wiki and search group by users in the AB test.
Apr 1 2021
Per discussions with @ovasileva, we are aiming to complete this in time to include in the Q3 Tuning Sessions. Data will be calculated using the new desktop opt-in and opt-out schema, which is currently in development T261842.
Mar 26 2021
Here is the draft report on the KPIs and guardrails for the Reply Tool AB Test for review.
Mar 25 2021
The product analytics team will review and prioritize this request during our next board review meeting on Tuesday, 30 March 2021.
Mar 24 2021
@ovasileva - Just checking in on the status of this task. Any questions or further analysis needed?
Mar 23 2021
This is correct - the PrefUpdate instrumentation is designed to only send out events when user initiated preference changes occur . I think there may have been confusion regarding this fact. From my understanding, the PrefUpdate schema/instrumentation isn't currently meeting our needs because it isn't sending the before state of a preference change (e.g. we are interested in knowing that a user went from latest Vector to Monobook) which is info that the PrefUpdate instrumentation doesn't currently send. cc @MNeisler if there are any corrections or additional needs that I missed here.
Mar 22 2021
A question: can we include a metric that helps us know the percentage of Junior and Senior Contributors who use the New Discussion Tool to add at least one topic and then turn off the feature?
Thinking: knowing if a relatively large number of people are turning off the tool after using it will serve as a helpful leading indicator for issues people may be having with the tool we are not aware of.
Mar 18 2021
Mar 16 2021
Mar 15 2021
Non AB Test Events
I noticed that from March 2nd to today, we have not recorded any Vue.Js search sessions initiated by logged-in users on wikis not included in the AB test. Vue.Js search events on non-pilot wikis were only recorded from Feb 25th through > March 2nd. @phuedx @ovasileva Is that expected?
AB Test Events
Mar 12 2021
Here is a quick look at the overall edit completion rate for replies on talk pages by editor type. Data is based on edit attempts by users in the AB test recorded in EditAttemptStep from 12 February 2021 through 10 March 2021. Note: This data currently reflects logged-in users across all experience levels. I will review edit completion rate by experience level when I complete the preliminary analysis on KPIs.
Mar 11 2021
I've reviewed and added my proposed quantitative metrics for the Hypotheses and Guardrails to the Talk pages project/Notifications/Measurement plan.
Mar 10 2021
FYI - I rechecked the Search Vue.JS AB test events and the search sessions initiated look much more balanced between the two test groups. Here are the search sessions initiated for each wiki and search group recorded since the deployment of the fix.
@ppelberg - I updated the task description's Workflow engagement metrics section with suggested metrics.
I ran a check of the data logged following the deployment of the AB test to all of the pilot wikis.
Mar 9 2021
Mar 8 2021
Mar 5 2021
Moving this to upcoming quarter for now. Timing will depend on the implementation of instrumentation in (T263505).
Mar 3 2021
Mar 2 2021
@ovasileva Here are my initial findings on the usage of language settings. Let me know if this helps answer the team's questions or you need any additional info/breakdowns.
Feb 24 2021
I completed QA of the new instrumentation changes and confirmed that the events are logging as expected and we can now differentiate between edits to existing sections vs the creation of new section using current VE/Wikieditor workflows.