I reviewed data logged in wikifunctions_ui to confirm events to track the editing of existing implementation and existing tests appear as expected following the patch.

Here are the count of events logged from 30 November 2023 through today (4 Dec 2023)

• No attempts to edit an existing test completed yet.
• Edis to existing implementations
  • 20 edit attempts
  • 20 publish events

So far data appears ok but I'm going to check again later this week once we have more data logged to confirm.

@DMartin-WMF

There still appears to be a discrepancy between the number of publish and attempt start events logged in wikifunctions_ui, especially for edits to existing tests.

See a summary of findings below and let me know if any additional data would be helpful in investigating the issue.

Note: The data below is limited to attempts and publish events that were recorded in December 2023 following the deployment of the fix.

Edits to Existing Tests:

• 5 edit attempt events
• 12 publish events

Daily edit existing test attempt and publish events in December 2023

Also, I reviewed events by session (using the performer.session_id field) and all of the sessions with publish test events do not have an associated attempt start within the same session. These attempts starts still appear to be missing from the data.

Edits to Existing implementations:
The total numbers appear closer to what's expected but still just slightly off.

There are just a couple days early after the patch was deployed (December 2nd and 3rd) where the number of publish events is higher than attempt starts (see table below). Sessions with published events on these days do not have a recorded attempt start in the same session. However, since it is only a couple days after the fix was merged, I'm wondering if the attempt starts for these sessions occurred before the fix which is why they are not logged. That's difficult to confirm thought.

After December 3rd, the daily total number of attempt starts is higher or the same as the number of publish events for all edits to existing implementations. I also confirmed that all sessions with a publish implementation event have an associated attempt start in the same session.

Daily edit existing implementation attempt and publish events in December 2023

Update per the planning meeting discussion today: The patch has not yet been deployed to production which is why the discrepancy still exists. I plan to QA on Thursday and Friday to confirm data appears as expected.

I just rechecked events today but we have not received sufficient test or implementation edit events to QA. Will plan to QA after the holidays.

Note: The daily rate of attempt start and publish events can be monitored on the wikifunctions dashboard on the "Edit existing tests" and "Edit existing implementation" charts on the interactions tab.

All attempt start events for both editing existing tests and implementation workflows now appear to be logging as expected following the deployment of the patch in T350066.

The total daily number of attempt starts is higher than the total number of published events since 14 December 2023. In addition, I confirmed that all sessions (as indicated by performer.session_id) with a publish implementation event have an associated attempt start in the same session.

Data collected via

-- collects data on edits to existing tests
WITH attempts AS (
SELECT
date_format(dt, 'yyyy-MM-dd') AS attempt_time,
performer.session_id AS attempts_session,
COUNT(1) AS attempt_events
FROM event.wikifunctions_ui
WHERE
name = 'wf.ui.defaultView.load' 
AND (custom_data)['edit'].value = 'true'  
AND (custom_data)['zobjecttype'].value = 'Z20'  -- substitute 'Z14' for implementation events
AND  (custom_data)['isnewzobject'].value = 'false'
AND ((Year = 2023 AND MONTH = 12 and Day >= 15) OR 
(Year = 2024 and MONTH = 01 AND Day = 01))
GROUP BY 
date_format(dt, 'yyyy-MM-dd'),
performer.session_id
),

publishes AS (
SELECT
date_format(dt, 'yyyy-MM-dd') AS publish_time,
performer.session_id AS publish_session,
COUNT(1) AS publish_events
FROM event.wikifunctions_ui
WHERE
name = 'wf.ui.editTester.publish'  -- substitute 'wf.ui.editImplementation.publish' for implementation events
AND  (custom_data)['isnewzobject'].value = 'false'
AND ((Year = 2023 AND MONTH = 12 and Day >= 15) OR 
(Year = 2024 and MONTH = 01 AND Day = 01))
GROUP BY 
date_format(dt, 'yyyy-MM-dd'),
performer.session_id
)

Superset Chart Update
Now that we are capturing all attempt start events, I was able to update the charts related to editing existing tests and implementations on the "Interaction metrics" tab of the dashboard. I've added a note to these charts to indicate that complete data is only available from 18 Dec 2023.

I'm closing this task as this as the issue now appears resolved but please let me know if you have any questions.
cc @DMartin-WMF