The analysis can be found in this notebook. For simplicity, I used the most recent 90 days of data, with known test accounts removed. I used HomepageModule as the authoritative source of data, because if a user blocks EventLogging we of course won't have any data at all.

The event funnel is somewhat different on mobile than on desktop, on the former it requires the user to click through from the mobile Homepage to get the overlay for the task fetch process to start. Therefore, I analyzed desktop and mobile separately. In both cases, the funnel starts with all sessions that had an impression of an activated SE module. This means that a user can have multiple sessions and therefore also be part of multiple paths in the funnel. Doing it this way simplifies the analysis a lot, but also ensures that we get a realistic picture based on the underlying data.

Desktop:

For desktop sessions, we aggregate on whether there was an se-fetch-tasks event, meaning tasks completed loading, and then whether there was an se-task-impression event, meaning we logged that the user saw a task. Here's how that breaks down across wikis:

We can see from the table that there's a varying amount of sessions where tasks completed fetching but no task impression event was logged. On the low end is Czech with 1.1%, on the high end is Korean with 3.9%. There's also a handful of sessions that had task impression events but did not log that task fetching completed.

Mobile:

Examining the data for mobile sessions, I split the analysis to first get a sense to what degree users click through to see the Newcomer Task overlay. Here's how that breaks down across wikis:

We see a fair amount sessions where users click through to see the overlay, but a much lower proportion in Korean than in the other three wikis.

Next, we do a similar breakdown of sessions as for desktop, but look only at sessions where the user did click through to see the overlay. The funnel then looks as follows:

From the table, we get the sense that the problem of task impressions not being logged provided there was an se-fetch-tasks event is much smaller on mobile than on desktop, we find 0.1–0.2% of sessions meet those criteria. We can also see that on mobile there are no task impression events that weren't preceded by a task fetch event.

Summary:

We have a varying degree of sessions where tasks completed loading, but no task impression event was logged. This issue appears to be isolated to the desktop side, as on mobile this rarely happens.

We fire a fetch event on page load and whenever the configuration (task types and topics) changes, and an impression when there's a non-empty search result and it can be successfully displayed. So one possibility is that the search returns empty (this seems like an unlikely thing to happen before the topic matching deployment; with topic filters it's not that surprising), or loading the first search result errors out (which of course shouldn't happen; could be due to network errors, but that should be rare). If that's the case, sessions which do not have a se-task-impression should have a se-task-pseudo-impression.

We discussed this in the board refinement meeting. At this point we're not going to prioritize this as I have other things on my plate. We did note two data points we'd like to look into:

1. For users who have an se-fetch-tasks event, but not an se-task-impression event, do we have other events logged suggesting that they were still on the page?
2. How quickly are tasks loading? In other words, what's the time difference between the se-fetch-tasks and first se-task-impression events for users who have both?

Once we know more, it might also be worthwhile doing some testing on slower network speeds.

@nettrom_WMF @MMiller_WMF Should we decline this or otherwise remove it from our current sprint board?

@kostajh : I think we can decline this. The second question seems to have been answered by the work we've done with improving performance. As for the first one, I think that's for @MMiller_WMF to decide. I think it would be quick to get an estimate of what that proportion is, but I'm also not sure it's something we're particularly concerned about.