Change Details

== Background During the reading list experiment, we created desktop and mobile instrumentation to track usage of the feature. We need to transition them to permanent instrumentation in order to track usage trends as a beta feature. The experimentation platform team has created some [[ https://wikitech.wikimedia.org/wiki/Test_Kitchen/Conduct_an_experiment#Convert_experiment_to_baseline_instrument | documentation ]] around how to do so, although some of the strings still say "xLab" when it's possible that this needs to be updated to the newest name: test kitchen. NOTE: this will encompass switching the custom stream we created to the web stream as we can now select custom contextual attributes via the UI with the instrument workflow Remaining work as I (Steph) understand it: - confirm with Jennifer whether the redundant metrics identified in T418289 are sufficient, given discrepancies in sampling/contextual attributes - follow the above documentation, substituting xlab for test kitchen, on relevant metrics - switch the custom stream to use web base stream, and decommission the now-unused custom stream - document instrument [[ https://wikitech.wikimedia.org/wiki/Test_Kitchen/Instrument_list | here ]] === Design requirements //Add design requirements or link to design files.// --- == Requirements === Acceptance criteria [] All events are logged as documented under "Engagement metrics" [[ https://docs.google.com/spreadsheets/d/1zbz52jUbSGteSUvNmZBlKaGXWJUf9pRnbARz1wtKP9M/edit?gid=0#gid=0 | here ]]. This does not include the grayed out sections. [] all events are tracked via the ReadingList instrument and not pulled from other instruments, even with redundancy we want to keep all the tracking together under one instrument. [] use `page-visited` instead of `page_load` for init events, as per documentation. [] `article_count` is removed from `action_context` [] instrumentation is for all logged in users using reading lists [] create instrument in [[ https://test-kitchen.wikimedia.org/create-instrument | Test Kitchen UI ]] with the following info: - Machine-readable name: reading-list-engagement - Description: For tracking usage trends of ReadingList as a beta feature - OKR: WE3.3 - Schema Type: web base - Stream name: product_metrics.web_base - Sample unit: session - Sample rate: 1 (aka 100%) - start/end date: asap and next FY - Risk Level: low - wikis: all (this may need to be done programmatically and not through UI, TBD) [] instrument is documented [[ https://wikitech.wikimedia.org/wiki/Test_Kitchen/Instrument_list | here ]] === BDD - //For QA engineer to fill out.// === Test Steps - //For QA engineer to fill out.// === Communication criteria //Add if this needs an announcement or discussion.// === Rollback plan //Describe the rollback plan in production for this task if something goes wrong.// //This task was created by Version 1.0.0 of the Reader Experience team task template using [[ https://phabulous.toolforge.org/ | phabulous ]].//

== Background During the reading list experiment, we created desktop and mobile instrumentation to track usage of the feature. We need to transition them to permanent instrumentation in order to track usage trends as a beta feature. The experimentation platform team has created some [[ https://wikitech.wikimedia.org/wiki/Test_Kitchen/Conduct_an_experiment#Convert_experiment_to_baseline_instrument | documentation ]] around how to do so, although some of the strings still say "xLab" when it's possible that this needs to be updated to the newest name: test kitchen. NOTE: this will encompass switching the custom stream we created to the web stream as we can now select custom contextual attributes via the UI with the instrument workflow Remaining work as I (Steph) understand it: - confirm with Jennifer whether the redundant metrics identified in T418289 are sufficient, given discrepancies in sampling/contextual attributes - follow the above documentation, substituting xlab for test kitchen, on relevant metrics - switch the custom stream to use web base stream, and decommission the now-unused custom stream - document instrument [[ https://wikitech.wikimedia.org/wiki/Test_Kitchen/Instrument_list | here ]] === Design requirements //Add design requirements or link to design files.// --- == Requirements === Acceptance criteria [] All events are logged as documented under "Engagement metrics" [[ https://docs.google.com/spreadsheets/d/1zbz52jUbSGteSUvNmZBlKaGXWJUf9pRnbARz1wtKP9M/edit?gid=0#gid=0 | here ]]. This does not include the grayed out sections. [] all events are tracked via the ReadingList instrument and not pulled from other instruments, even with redundancy we want to keep all the tracking together under one instrument. [] use `page-visited` instead of `page_load` for init events, as per documentation. [] `article_count` is removed from `action_context` [] instrumentation is for all logged in users using reading lists [] create instrument in [[ https://test-kitchen.wikimedia.org/create-instrument | Test Kitchen UI ]] with the following info: - Machine-readable name: reading-list-engagement - Description: For tracking usage trends of ReadingList as a beta feature - OKR: WE3.3 - Schema Type: web base - Stream name: product_metrics.web_base - Sample unit: session - Sample rate: 1 (aka 100%) - start/end date: asap and next FY - Risk Level: low - wikis: all [] instrument is documented [[ https://wikitech.wikimedia.org/wiki/Test_Kitchen/Instrument_list | here ]] === BDD - //For QA engineer to fill out.// === Test Steps - //For QA engineer to fill out.// === Communication criteria //Add if this needs an announcement or discussion.// === Rollback plan //Describe the rollback plan in production for this task if something goes wrong.// //This task was created by Version 1.0.0 of the Reader Experience team task template using [[ https://phabulous.toolforge.org/ | phabulous ]].//

== Background During the reading list experiment, we created desktop and mobile instrumentation to track usage of the feature. We need to transition them to permanent instrumentation in order to track usage trends as a beta feature. The experimentation platform team has created some [[ https://wikitech.wikimedia.org/wiki/Test_Kitchen/Conduct_an_experiment#Convert_experiment_to_baseline_instrument | documentation ]] around how to do so, although some of the strings still say "xLab" when it's possible that this needs to be updated to the newest name: test kitchen. NOTE: this will encompass switching the custom stream we created to the web stream as we can now select custom contextual attributes via the UI with the instrument workflow Remaining work as I (Steph) understand it: - confirm with Jennifer whether the redundant metrics identified in T418289 are sufficient, given discrepancies in sampling/contextual attributes - follow the above documentation, substituting xlab for test kitchen, on relevant metrics - switch the custom stream to use web base stream, and decommission the now-unused custom stream - document instrument [[ https://wikitech.wikimedia.org/wiki/Test_Kitchen/Instrument_list | here ]] === Design requirements //Add design requirements or link to design files.// --- == Requirements === Acceptance criteria [] All events are logged as documented under "Engagement metrics" [[ https://docs.google.com/spreadsheets/d/1zbz52jUbSGteSUvNmZBlKaGXWJUf9pRnbARz1wtKP9M/edit?gid=0#gid=0 | here ]]. This does not include the grayed out sections. [] all events are tracked via the ReadingList instrument and not pulled from other instruments, even with redundancy we want to keep all the tracking together under one instrument. [] use `page-visited` instead of `page_load` for init events, as per documentation. [] `article_count` is removed from `action_context` [] instrumentation is for all logged in users using reading lists [] create instrument in [[ https://test-kitchen.wikimedia.org/create-instrument | Test Kitchen UI ]] with the following info: - Machine-readable name: reading-list-engagement - Description: For tracking usage trends of ReadingList as a beta feature - OKR: WE3.3 - Schema Type: web base - Stream name: product_metrics.web_base - Sample unit: session - Sample rate: 1 (aka 100%) - start/end date: asap and next FY - Risk Level: low - wikis: all (this may need to be done programmatically and not through UI, TBD) [] instrument is documented [[ https://wikitech.wikimedia.org/wiki/Test_Kitchen/Instrument_list | here ]] === BDD - //For QA engineer to fill out.// === Test Steps - //For QA engineer to fill out.// === Communication criteria //Add if this needs an announcement or discussion.// === Rollback plan //Describe the rollback plan in production for this task if something goes wrong.// //This task was created by Version 1.0.0 of the Reader Experience team task template using [[ https://phabulous.toolforge.org/ | phabulous ]].//