@nettrom_WMF - as discussed in our team meeting this week, can you look a little further into if we’re losing suggested activations or non-suggested activations? In other words, we want to see if the percentage of users who click through to suggested edits from the empty state is different between the two groups.

@Tgr - does that cover your question?

Yes, thanks.

If we don't trust click logging (activation stats are robust against ad blockers etc, click stats aren't), we could also just split users by whether their first edit is a suggested one, I think.

We've examined this phenomenon in several different ways looking for patterns, and not found any specific pattern that provides clues as to why this is happening. This analysis started out by looking at the three breakdowns suggested in the task, as well as by wiki since it was trivial to split the data that way too. Here's what we found:

• Wiki: Bangla Wikipedia is different from the other three in that users who get the new impact module are slightly more likely to activate.
• Geolocation: no particular pattern by country within each wiki.
• Android & iOS: users with either OS show the same drop in activation, and there are no differences by OS version.
• Browsers: no difference between the major browsers, and various versions also show the same pattern.

We've also investigated whether newcomers activate through suggested edits or not, and did not find a significant difference between experiment groups. One thing we did learn was that newcomers who activate by making a constructive article edit tend to do so on their first edit!

Lastly, we investigated whether there was a temporal pattern associated with the data center switchover in late February/early March. We did not find any suggestion that activation was affected by that either.

As far as this task is concerned, we've exhausted the obvious places to look for patterns and will shift our attention to other tasks. Moving this to "Needs Sign-off" on the Product Analytics board for @KStoller-WMF to review and comment in case I've missed something, otherwise this task can be resolved.

I created a user account "KHarlan Test New Impact‬", with no contributions, and used ge.utils.setUserVariant('oldimpact') to switch to the non-Vue interface, and ge.utils.setUserVariant('control') to switch to the Vue interface. I used Firefox 114, and used the Network inspector to disable HTTP caching. Then I reloaded the page and recorded results.

Note, in the results, the terms load and DOMContentLoaded have the following meanings:

load is the more important event from the end-user perspective.

Some observations

• Old impact module on desktop makes more HTTP requests
  • Follow-up: identify what these additional requests are about
• Desktop, old impact: Loads faster and transfers less data than Vue, as expected.
  • Follow-up: It is surprising that the Vue implementation is more than double the load time for the non-Vue implementation. Figure out where/what/why
• Mobile: Vue implementation transfers less data and has significantly faster load time than non-Vue implementation
  • Follow-up: Understand why this is the case

I've taken a first look at this by investigating two proportions. First, the proportion of newcomers with constructive article activation. Second, the proportion of those activating through Suggested Edits. While that doesn't specifically answer if users are able to find Suggested Edits and/or activate it, it should give us an indication of whether we are running into issues.

I grabbed data from our four pilot wikis to get an initial sense, and then we can follow up further as needed. The table below shows the numbers and proportions. Generally, the trend is: users with the old impact module are more likely to activate, and more likely to do so through Suggested Edits. Czech desktop registrations are different when it comes to the activation proportions, and Spanish mobile registrations are different when it comes to Suggested Edits. Note that none of these differences in activation proportions are statistically significant (that goes for overall by platform as well, and I didn't check the Suggested Edits proportions).

@KStoller-WMF : What would the next step be here? Wait for T337314 to have been deployed for two weeks and then check the numbers again?

In T334411#8904606, @nettrom_WMF wrote:

I've taken a first look at this by investigating two proportions. First, the proportion of newcomers with constructive article activation. Second, the proportion of those activating through Suggested Edits. While that doesn't specifically answer if users are able to find Suggested Edits and/or activate it, it should give us an indication of whether we are running into issues.

Thanks @nettrom_WMF!

@KStoller-WMF : What would the next step be here? Wait for T337314 to have been deployed for two weeks and then check the numbers again?

My 2c is that T337314: [BUG] New impact empty module is loading without CTA and with mobile headers is unlikely to have any significant effect in what we are seeing here. IIRC last time we looked at mobile-details mode, it was in use by ~1% of mobile views to Special:Homepage.