We've released some improvements to the Impact module empty state logic and loading (example: T324285 & T322832), is it possible that some of those edge cases were enough to cause this discrepancy? Are there other edge cases we aren't considering?

I doubt that those two tasks are related.

Is there any chance we are showing the "new design" alert on mobile?

Yes, we are, including for brand new users, which seems wrong per T323619: NewImpact: Introduce new design to existing newcomers via GuidedTour and Drawer. You can check the behavior with this snippet in your browser console: new mw.Api().saveOptions( { 'growthexperiments-tour-newimpact-discovery': 0 } ); On desktop, no "updated design" prompt appears, but on mobile, we show the "updated design" dialog after the user taps on the module.

Is there a difference in performance / loading speed?

We no longer render the summary via the server side. So yes, the rendering is slower but there is also a skeleton to compensate. That said, I believe the suggested edits module is what would drive activation, not the impact module, so I am not sure that performance of the impact module rendering is relevant here.

In Grafana, you can see the mount time for the overlay summary: it's steady at ~200ms.

From meeting just now: I'd suggest comparing specifically the funnel of users who tap on empty impact module, then press "See suggested edits". If there is a marked decrease in users who click on "See suggested edits" in the new impact module, that could help explain the lower activation rate.

On desktop, no "updated design" prompt appears, but on mobile, we show the "updated design" dialog after the user taps on the module.

OK, I added: T330692: Impact Module: don't show "Updated design" info drawer to new users

From meeting just now: I'd suggest comparing specifically the funnel of users who tap on empty impact module, then press "See suggested edits". If there is a marked decrease in users who click on "See suggested edits" in the new impact module, that could help explain the lower activation rate.

Agreed, I added this and a few other questions to T327581: New Impact Module: experiment analysis

I think it's possible that this difference in activation relates to a few differences, and can't be explained by just one easy to identify problem, so I've also added a design related task: T330695: New Impact module's empty state on mobile: Design iteration

Wrt performance, we load the user impact for the mobile preview so that could slow down display of the suggested edits module as well. OTOH we are talking about users with 0 edits here, the impact lookup should not take a perceptible amount of time.

The new module uses Vue and it's needed to show the mobile overview, right? So maybe loading Vue makes a performance difference?

Wrt errors, T324930: NewImpact: Cannot read properties of undefined (reading 'days') is somewhat common (40-50/day) but the user impact should be barely noticeable. There's about 10/day of HTTP error with a similarly unhelpful stack trace, not sure about that. (Maybe related to userimpact API calls? Can't see how it would happen.) There's a similar amount of mustbeloggedin-generic errors (session timeout?) which I can't imagine to be possibly related. Other JS errors are either clearly not impact module related, or very low frequency.

I agree funnel or user pathway style representation of the data would be useful. E.g. it would be interesting to see if the first click users make on the overview page is affected (maybe the impact module looks more appealing now, so more users click it instead of suggested edits, but then it doesn't contain anything that can be interacted with so they bounce?).

Wrt performance, we load the user impact for the mobile preview so that could slow down display of the suggested edits module as well. OTOH we are talking about users with 0 edits here, the impact lookup should not take a perceptible amount of time.

Good to know, I didn't realize the data loaded would be different for brand new accounts (with zero edits). Far more newcomers on mobile end up on the newcomer homepage than navigate to the actual empty impact module view, so perhaps it makes sense to look at that closely.

When I analyze the page load with Lighthouse (for an account with zero edits) it seems like the homepage with the new impact module regularly has a higher "Time to Interactive" and "Largest Contentful Paint".

Newcomer homepage page load comparison:

with "Old" impact module	with new-impact module enabled

These values change slightly each time I run the report, but it does seem like the "Time to Interactive" and "Largest Contentful Paint" metrics are different enough that we might want to look into this further?

Huh, those are pretty high values. Is that on a production website? We switched one of the panels to Vue which cannot be rendered on server side so it's not surprising some of the paint happens later, but I would have expected way smaller values both with and without Vue.

Huh, those are pretty high values. Is that on a production website?

Yes, those values are from Chrome Lighthouse DevTools on French Wikipedia using the same account, and using new-impact=1 to opt into seeing the new impact module. The screenshots I added are for "Mobile" stats, not the "Desktop" stats. My understanding is that for mobile stats, Lighthouse tries to simulate imperfect mobile conditions: a slower connection on a slightly underpowered device. So it's not too surprising to me that the values are higher, but it is surprising that the numbers are so different between new-impact and the "old impact" module when the empty states are nearly the same. But it sounds like there might not be an easy performance solution here if the issue relates to Vue and lack of server side rendering?

@KStoller-WMF can you please share the full URL you used with Lighthouse?

In T330614#8654049, @kostajh wrote:

@KStoller-WMF can you please share the full URL you used with Lighthouse?

@kostajh
Homepage with "old" impact module: https://fr.m.wikipedia.org/wiki/Sp%C3%A9cial:Accueil_de_l%E2%80%99espace_personnel
Homepage with new-impact module enabled: https://fr.m.wikipedia.org/w/index.php?title=Sp%C3%A9cial:Accueil_de_l%E2%80%99espace_personnel&new-impact=1#

old impact	new impact

This is what I see. (@KStoller-WMF the files you posted in T330614#8651465 are restricted, can you make them public please?)

The definition of Time to Interactive for Lighthouse is:

TTI measures how long it takes a page to become fully interactive. A page is considered fully interactive when:

The page displays useful content, which is measured by the First Contentful Paint,
Event handlers are registered for most visible page elements, and
The page responds to user interactions within 50 milliseconds.

Whereas for us, I think the more useful metric is how long it takes the suggested edits module to be interactive, since that is the main call to action for the user:

Once the user has tapped on suggested edits module, we see that the TTI is 381 ms according to our instrumentation:

But we don't currently instrument on the client-side the loading of all modules to show in Special:Homepage on mobile. We should probably do that: time to interactive for each mobile summary block, and time for all modules to render.

In T330614#8655888, @kostajh wrote:

But we don't currently instrument on the client-side the loading of all modules to show in Special:Homepage on mobile. We should probably do that: time to interactive for each mobile summary block, and time for all modules to render.

We do instrument the mobile summary rendering time for the new impact module: https://grafana.wikimedia.org/d/vGq7hbnMz/special-homepage-and-suggested-edits?orgId=1&from=now-7d&to=now&viewPanel=170&var-platform=All&var-UserImpactHandlerCache=All&var-UserImpactHandlerPingLimiter=NoData&var-impactrendermode=All

That shows that in most cases, the impact module's summary state (whether it has data or not) is within 50ms-350ms.