Page MenuHomePhabricator

Data validation iOS edit data in EditAttemptStep
Closed, ResolvedPublic

Description

Data should now be sending from latest beta release.

Event Timeline

LGoto triaged this task as Medium priority.May 23 2023, 6:21 PM
LGoto moved this task from Needs Triage to Product Backlog on the Wikipedia-iOS-App-Backlog board.

These changes will be deployed as part of the 7.3.0 release but will be verified by analytics post-release

Working with beta user data validating we are seeing events however for the events where action = saveSuccess we should also be seeing a value for revision_id, currently not seeing that field populated. Since we only have 2 of these events I can't say for sure there is an issue but this needs to be included with event so keeping an eye on this.

mpopov moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.
mpopov subscribed.

Moving into Doing since Shay has started on this.

Raising a flag since we now have 2 more events where action = saveSuccess we should also be seeing a value for revision_id, currently not seeing that field populated. I checked in case this was being sent with another action and not seeing any revision_id values on any rows of our data.

Hi @SNowick_WMF, we were not sending this field. I opened a PR to fix it, https://github.com/wikimedia/wikipedia-ios/pull/4544, after we merge it and release a version I'll assign it back to you for validation. Thanks for catching this!

Tsevener subscribed.

Fix is in 7.3.1 (Testflight beta).

We are seeing values for event.revision_id but only for a very small number of saveSuccess events. Looking for a pattern now to see if there are any clues, not seeing anything yet. Events by app versions: 7.3.1.0 (3), 7.3.1.2286 (8), 7.3.1.2297 (3),7.3.1.2318 (1). Everything else as expected.

Daily saveSuccess events with/without revision_id

Update: I did a pass and pulled edit data/counts from mediawiki_history (and verified on Turnilo) that show much higher counts of edit events [with value ios app edit in revision_tags] - also noticed the variance of page_ns for edits on mediawiki_historycompared to the page_ns tracked in editattemptstep that only show edits for 0/article edit (98% of edits on mediawiki_history are for page_ns = 0 so this isn't the main issue). Will need to look into where we are missing event counts.

@SNowick_WMF I have a Draft PR here with some updates on the schema, so we're tracking talk page and article description edits. I'll move this ticket to waiting until we decide on how to log events for Wikidata article description edits (cc @Tsevener)

@Mazevedo I just remembered - another area we need to add this is in our new undo and rollback functionality on the Diffs screen.

PR updated with diff events, we're leaving the wikidata edit logs as-is to match Android https://github.com/wikimedia/wikipedia-ios/pull/4567

Additional edit events (talk pages, diff undo/rollback, article description) are now being sent in TestFlight 7.4.0 (2459).

Note I pulled baseline daily edit count data to prep for post-release data comparison - Data. Will re-run queries in 2 weeks when we have accumulated data.

Verifying we are seeing the increased edit counts due to additional events being added, unique editor counts remain around the same (Data). It's difficult to determine if we are actually missing data or just getting limited counts because of user data sharing opt-out.

The source of truth is mediawiki_history which has edit/unique editor counts that are ~130-150% higher for both values, making the iOS data in editattemptstep unreliable when used as intended - to compare Mobile Web edits to iOS in app edits.

Closing this integration as resolved for now because I don't think the missing data is an issue for our engineers but a limitation on what we can track within the app. Will need to communicate the lack of accurate counts in editattemptstep for other analysts when making iOS edit comparisons to Mobile Web.