We would like to study the impact of AFTv5 to answer these key research questions, using new data:
- Editor Conversions: how many readers who post feedback go on to become editors?
- New Editor Productivity: how many of these new editors edit productively? (with same reversion rates as other new editors)
Previous research showed that, when the CTA is present, a number of readers go on to make productive edits, , as outlined in this report:
We would now like to collect more data for at least one week to verify if this is still the case, now that AFT5 has been deployed to 10% (with a new feedback form and more sophisticated monitoring tools). To that end, we would like to conduct the same Metrics Stage 3 test that we ran from 4/27 to 5/07, when clicks from CTA 1 (Edit this page) were gathered from the clicktracking logs.
For this new study 6, we have created a task card and checklist on Trello to track our progress:
From a development standpoint, here are some of the main action items that are needed for this test:
a) change the bucketing parameters to show the CTA1 at 100% for all readers who have not edited before (but not to existing editors)
b) re-enable clicktracking to collect fresh data, tracking these metrics for non-editors:
- total impressions of the CTA1 (Edit this page)
- total clicks on the CTA1 (causing the Edit form to appear)
- total clicks on the Edit Save button
c) re-enable similar clicktracking for the Page Edit (and Section Edit) buttons, so we can compare the activity of users who edit through the AFT5 CTA1 with users who edit through conventional means.
d) re-enable the special tag that was associated with edits in Metrics Stage 3, so that we can track the number of saved edits which were reverted, and determine the productivity of these edits, compared with other new editors who click on the regular Edit button.
e) test thoroughly that we are collecting the proper data and tagging edits correctly on prototype before deployment.
Dario will provide more details in comments below on what exact codes and tags we want to use to track this properly.
See also section 1 of this Etherpad page for more details on our overall pre-rampup data collection plan:
For more info on our CTAs and how AFT5 works, check this feature requirements page:
See also related bugs from earlier efforts on this topic: