Change Details

**Objective**: To analyze the quality of the data collected with a new Metrics Platform-based instrument (`mediawiki_web_ui_actions`) by comparing that data to the data collected with the existing non-Metrics Platform-based instrument (`desktop_web_ui_actions`) and verify whether the migration to Metrics Platform can proceed or if there are any issues that need to be resolved. ## Prerequisites for data QA - [X] Mapping of old instrumentation to new instrumentation: **[[ https://docs.google.com/document/d/14sFvCL-WZT7W8E8hpY16b152al2Mj69tcF9Ihy1UqJE/edit?usp=sharing | LINK TO DOCUMENT ]]** - This is primarily driven by engineer performing the migration - Analyst reviews map for completeness and correctness, and contributes/assists as needed - [X] Specific QA needs have been identified and are agreed upon ([[ https://phabricator.wikimedia.org/T353029#9540370 | SEE BELOW ]]) - [] Analyst decides in collaboration with engineers/PM whether to QA the whole instrument, or if there are key parts that should be QAed and the rest can be assumed to be okay. - [X] Document the parts and the relevant queries: - (1) overall counts by action and sub-action (if applicable) - **if relevant** (2) counts by specific identifiers (e.g. by session) - **NOTE**: these queries will almost always be limited by time in some way - [X] New instrument has been deployed and activated ([[ https://phabricator.wikimedia.org/T351298 | Link to Phab task ]]) - [X] Engineer has verified that events are flowing in and that the instrument is not producing schema validation errors. - [X] Verify that events are flowing in: - [X] EventGate Grafana dashboard - [X] Kafka by Topic Grafana dashboard - [X] EventStreams - [X] New instrument doesn't have any schema validation errors - [] Prioritization agreement between analyst & PM of the QA work in the context of other needs/requests (e.g. PM may need to wait longer for some analysis so that the analyst can do the QA work) - [X] Documentation of old and new table names and date of deployment for analyst's reference: | Instrument | Table name | Stream deployed (if applicable) | |------------|------------|---------------------------------| | DesktopWebUIActionsTracking | DesktopWebUIActionsTracking | 2023-12-05 | | MediaWikiWebUIActions | mediawiki_web_ui_actions | 2024-01-08 | ## Data QA checklist If more than one instrument is being migrated, these steps need to be completed for each one. - [x] Count the daily number of schema validation errors for - [x] Old instrument - [x] New instrument - [x] Compare counts of events by action and sub-action (as defined in the mapping from prerequisites) - [x] **if relevant** Compare counts by specific identifier (as defined in the prerequisites) - [] Upload QA notebooks to Gitlab, making sure to follow [[ https://foundation.wikimedia.org/wiki/Legal:Data_publication_guidelines | data publication guidelines ]] - [x] Document any issues (or notable observations found) on this ticket - [] Resolve this ticket **NOTE**: If any issues were identified that require fixing the new instrument, data QA of the fixed instrument will need to be filed as a new Phab task. Some of the checked prerequisites will carry over. ------ See [[ https://docs.google.com/document/d/1sCD4I0zHn5BrL6TFB-Hh_YwZDafFKoJjZUrYfyQSQIM/edit | Metrics Platform Instrument Migration Data QA Process Description ]] for more details.

**Objective**: To analyze the quality of the data collected with a new Metrics Platform-based instrument (`mediawiki_web_ui_actions`) by comparing that data to the data collected with the existing non-Metrics Platform-based instrument (`desktop_web_ui_actions`) and verify whether the migration to Metrics Platform can proceed or if there are any issues that need to be resolved. ## Prerequisites for data QA - [X] Mapping of old instrumentation to new instrumentation: **[[ https://docs.google.com/document/d/14sFvCL-WZT7W8E8hpY16b152al2Mj69tcF9Ihy1UqJE/edit?usp=sharing | LINK TO DOCUMENT ]]** - This is primarily driven by engineer performing the migration - Analyst reviews map for completeness and correctness, and contributes/assists as needed - [X] Specific QA needs have been identified and are agreed upon ([[ https://phabricator.wikimedia.org/T353029#9540370 | SEE BELOW ]]) - [] Analyst decides in collaboration with engineers/PM whether to QA the whole instrument, or if there are key parts that should be QAed and the rest can be assumed to be okay. - [X] Document the parts and the relevant queries: - (1) overall counts by action and sub-action (if applicable) - **if relevant** (2) counts by specific identifiers (e.g. by session) - **NOTE**: these queries will almost always be limited by time in some way - [X] New instrument has been deployed and activated ([[ https://phabricator.wikimedia.org/T351298 | Link to Phab task ]]) - [X] Engineer has verified that events are flowing in and that the instrument is not producing schema validation errors. - [X] Verify that events are flowing in: - [X] EventGate Grafana dashboard - [X] Kafka by Topic Grafana dashboard - [X] EventStreams - [X] New instrument doesn't have any schema validation errors - [] Prioritization agreement between analyst & PM of the QA work in the context of other needs/requests (e.g. PM may need to wait longer for some analysis so that the analyst can do the QA work) - [X] Documentation of old and new table names and date of deployment for analyst's reference: | Instrument | Table name | Stream deployed (if applicable) | |------------|------------|---------------------------------| | DesktopWebUIActionsTracking | DesktopWebUIActionsTracking | 2023-12-05 | | MediaWikiWebUIActions | mediawiki_web_ui_actions | 2024-01-08 | ## Data QA checklist If more than one instrument is being migrated, these steps need to be completed for each one. - [x] Count the daily number of schema validation errors for - [x] Old instrument - [x] New instrument - [x] Compare counts of events by action and sub-action (as defined in the mapping from prerequisites) - [x] **if relevant** Compare counts by specific identifier (as defined in the prerequisites) - [] Upload QA notebooks to Gitlab, making sure to follow [[ https://foundation.wikimedia.org/wiki/Legal:Data_publication_guidelines | data publication guidelines ]] - [x] Document any issues (or notable observations found) on this ticket - [] Resolve this ticket **NOTE**: If any issues were identified that require fixing the new instrument, data QA of the fixed instrument will need to be filed as a new Phab task. Some of the checked prerequisites will carry over. ------ See [[ https://docs.google.com/document/d/1sCD4I0zHn5BrL6TFB-Hh_YwZDafFKoJjZUrYfyQSQIM/edit | Metrics Platform Instrument Migration Data QA Process Description ]] for more details. ==== QA summary [[ https://docs.google.com/spreadsheets/d/1cR2R2K54tzkHiIdUoQta53YbQapVm3YVN5KIemgmjzA/edit?usp=sharing | the tab of desktopwebuiactionstracking in the google sheet ]]

**Objective**: To analyze the quality of the data collected with a new Metrics Platform-based instrument (`mediawiki_web_ui_actions`) by comparing that data to the data collected with the existing non-Metrics Platform-based instrument (`desktop_web_ui_actions`) and verify whether the migration to Metrics Platform can proceed or if there are any issues that need to be resolved. ## Prerequisites for data QA - [X] Mapping of old instrumentation to new instrumentation: **[[ https://docs.google.com/document/d/14sFvCL-WZT7W8E8hpY16b152al2Mj69tcF9Ihy1UqJE/edit?usp=sharing | LINK TO DOCUMENT ]]** - This is primarily driven by engineer performing the migration - Analyst reviews map for completeness and correctness, and contributes/assists as needed - [X] Specific QA needs have been identified and are agreed upon ([[ https://phabricator.wikimedia.org/T353029#9540370 | SEE BELOW ]]) - [] Analyst decides in collaboration with engineers/PM whether to QA the whole instrument, or if there are key parts that should be QAed and the rest can be assumed to be okay. - [X] Document the parts and the relevant queries: - (1) overall counts by action and sub-action (if applicable) - **if relevant** (2) counts by specific identifiers (e.g. by session) - **NOTE**: these queries will almost always be limited by time in some way - [X] New instrument has been deployed and activated ([[ https://phabricator.wikimedia.org/T351298 | Link to Phab task ]]) - [X] Engineer has verified that events are flowing in and that the instrument is not producing schema validation errors. - [X] Verify that events are flowing in: - [X] EventGate Grafana dashboard - [X] Kafka by Topic Grafana dashboard - [X] EventStreams - [X] New instrument doesn't have any schema validation errors - [] Prioritization agreement between analyst & PM of the QA work in the context of other needs/requests (e.g. PM may need to wait longer for some analysis so that the analyst can do the QA work) - [X] Documentation of old and new table names and date of deployment for analyst's reference: | Instrument | Table name | Stream deployed (if applicable) | |------------|------------|---------------------------------| | DesktopWebUIActionsTracking | DesktopWebUIActionsTracking | 2023-12-05 | | MediaWikiWebUIActions | mediawiki_web_ui_actions | 2024-01-08 | ## Data QA checklist If more than one instrument is being migrated, these steps need to be completed for each one. - [x] Count the daily number of schema validation errors for - [x] Old instrument - [x] New instrument - [x] Compare counts of events by action and sub-action (as defined in the mapping from prerequisites) - [x] **if relevant** Compare counts by specific identifier (as defined in the prerequisites) - [] Upload QA notebooks to Gitlab, making sure to follow [[ https://foundation.wikimedia.org/wiki/Legal:Data_publication_guidelines | data publication guidelines ]] - [x] Document any issues (or notable observations found) on this ticket - [] Resolve this ticket **NOTE**: If any issues were identified that require fixing the new instrument, data QA of the fixed instrument will need to be filed as a new Phab task. Some of the checked prerequisites will carry over. ------ See [[ https://docs.google.com/document/d/1sCD4I0zHn5BrL6TFB-Hh_YwZDafFKoJjZUrYfyQSQIM/edit | Metrics Platform Instrument Migration Data QA Process Description ]] for more details. ==== QA summary [[ https://docs.google.com/spreadsheets/d/1cR2R2K54tzkHiIdUoQta53YbQapVm3YVN5KIemgmjzA/edit?usp=sharing | the tab of desktopwebuiactionstracking in the google sheet ]]