@ppelberg
I've completed investigating the decrease in edit completion rate we observed for logged-in users in the Mobile Web section editing dead end experiment. See key findings and results below.

Key takeways:

• Logged-in users who click on the Edit Full Page Button are 55.6% more likely to complete their edit compared to those in the treatment group who don't click the new button. This is partly because users who click this button are likely high-intent users, who clicked that section editor with the goal of making an edit (vs accidental or curiosity clicks).
  • However, even when we isolate to users who started to type, we still observe an increase in edit completion rate for users who were shown the Edit Full Page Button, indicating the feature is an effective tool for logged-in users navigating the editing process.
• The overall 5.8% decrease in edit completion rate is mostly due to logged-in users in the treatment group who abandon their edits before starting to type. This indicates that the new Edit Full Page button might be slightly distracting or confusing to some newcomers or deter low-intent editors.
• Overall, the new Edit Full Page button is highly effective at helping high-intent users complete their edit, but also appears to produce some slight friction to the edit process that causes a drop-off for some newcomers before they start to type.

Results

Compare the abandonment rates for those who engaged with the new with the new Edit full page button (T409990) to those that did not.

Treatment Group Section Edit Completion Rate by Engagement with the Edit Full Page Button
(defined as users that open the section editor (action = init) and save their edit (action = saveSuccess))

For the treatment group, logged-in users who directly engaged with the Edit Full Page Button had a significantly higher edit completion rate (+55.6%) than those who didn't. We saw a similar trend for logged-out users as well.

This indicates that the decrease in overall edit completion rate for logged-out users was not due to engagement with the Full Page Button.

I next explored the abandonment rate of users at specific points in the editing workflow to help isolate where users in the treatment group are abandoning their edits. I included splits by test group and whether the user engaged with the edit Full Page Button.

Review the proportion of users that abandoned edits prior to starting to type (action = 'firstChange')

Proportion of Logged-In users who initiated a section edit (action = init) and started to type (action = firstChange)

A higher proportion of logged-in users in the treatment group abandoned their edit before making their first change than in the control group. There was a -19.5% decrease in users who started to type after opening the section editor. This includes both logged-in users who clicked and did not click the new Edit Full page button.

Review the proportion of users that abandoned edits after to starting to type (action = 'firstChange')

Proportion of users that started typing (action = firstChange) and saved their edit (action = SaveSuccess)

Logged-in users in the treatment group who started to type were 8.5% more likely to complete their edit compared to the control group.
Among logged-in users who started to type, clicking the switch resulted in a very high completion rate (96.83%) compared to users in the treatment group who did not click on the new button and started to type.

Resolving this task @MNeisler and I will incorporate these findings into the path forward we will propose for this intervention in T409112.