I've completed the initial investigation into the effects of how load times affect Homepage engagements.

To keep this task constrained, I used a very simple approach to data gathering: I used the week prior to the bug going into effect (2024-06-11 through 2024-06-17) and compared that to the first week of the bug being present (2024-06-18 through 2024-06-24). I used data from all wikis except testwiki, and split the data up by platform (desktop and mobile web). Given this simple method, I have not attempted to determine if there's a *significant* change between the weeks, but instead try to interpret the findings based on the relative differences.

Summary of findings

1. Homepage visits: no substantial difference (-0.4%), but opposing trends between desktop and mobile web. Desktop decreases (-0.9%) whereas mobile web increases (+0.9%). Visits on desktop are roughly 2x that of mobile web, btw.

2. Homepage module impressions: meaningful decrease (-3.8%). The trends are similar on both platforms (desktop -4.4%, mobile web -2.5%), but the mobile web effect might be dampened by the slight increase in visits.

3. Module interactions: substantial decrease (-12.8%). The effect is larger on desktop (-14.2%), where most of our interactions happen, but still substantial on mobile web (-9.3%).

4. Clicks on Suggested Edits tasks: substantial decrease (-7.1%). Here the effect is larger on mobile web (-13.7%) than on desktop (-5.6%).

5. Completed Structured Task edits: overall no difference (-0.7%) but opposing trends between desktop and mobile. On mobile web we see a substantial decrease (-17.8%) while on desktop there's an increase (+4.5%).

Detailed findings

Homepage visits

We do not find a substantial difference in the number of Homepage visits. Overall, the pre/post difference is a decrease from 148.3k to 147.8k visits (-0.5k or -0.4%). We also do not find a substantial difference when splitting this up by platform, but we find opposing trends. On desktop there's a decrease from 106.7k to 105.8k visits (-0.9k or -0.9%), while on mobile web there's an increase from 42.6k to 43.0k (+0.4k or +0.9%).

Homepage module impressions

We find a meaningful decrease in the number of Homepage module impressions. Overall, we find a decrease pre/post from 604.5k to 581.4k (-23.1k or -3.8%). Split by platform, the decrease is larger on desktop than on mobile web. Desktop goes from 426.1k pre-bug to 407.6k post-bug (-18.6k or -4.4%), while mobile web goes from 178.3k to 173.9k (-4.5k or -2.5%).

Homepage module interactions

We find a substantial decrease in the number of interactions with Homepage modules. Overall the number of interactions decrease from 322.1k to 280.9k (-41.2k or -12.8%). Split by platform we see a larger relative decrease on desktop (227.6k to 195.3k, -32.4k or -14.2%) than on mobile web (94.4k to 85.6k, -8.8k or -9.3%).

Clicks on Suggested Edits tasks

We find a substantial decrease in the number of clicks on Suggested Edits tasks. Overall we find a reduction in clicks from 10.5k to 9.7k (-0.7k or -7.1%). Split by platform we find a larger relative difference on mobile web (1.9k to 1.7k, -0.3k or -13.7%) compared to desktop (8.6k to 8.1k, -0.5k or -5.6%).

Completed Structured Task edits

We find a substantial decrease in the number of completed Structured Task edits but on the mobile web platform. On desktop we see an increase, and overall the trend is flat (5.8k pre/post, -0.0k, -0.7%). On mobile web there's a decrease from 1.4k to 1.1k (-0.2k or -17.8%), on desktop there's an increase from 4.5k to 4.7k (+0.2k or +4.5%).