Page MenuHomePhabricator

Investigate proportion of edit sessions that would see fallback at a given load time
Closed, ResolvedPublic

Description

This task involves the work of understanding the proportion of edit sessions that would see the fallback experience T239676 implements if presented at a range of load times.

Knowing this will inform the amount of time we define as needing to have elapsed between someone tapping edit and the fallback experience (T239676) appearing.

Decision(s) to be made

  • 1. What – if any – adjustments will we make to the amount of loading time that will need to have elapsed before someone is shown the prompt T239676 introduces?
    • For now, we're going to leave the current 3 second loading threshold as-is for now and reconsider when we prioritize work on T349231.

Research questions

  1. What proportion of edit sessions that involve mobile VE being loaded by default would see the fallback experience (T239676) if it were presented at range of load times?

Requirements

The below is very much a first pass and requires @MNeisler to review and refine before we consider it "scoped."
Using data from the most recent mobile VE A/B test, show the number and proportion of distinct edit sessions that load mobile VE by default and would see the fallback experience (T239676) were it be shown after the following elapsed loading times:

  1. 0.5 seconds
  2. 1.0 second
  3. 1.50 seconds
  4. 1.75 seconds
  5. 2.0 seconds
  6. 2.25 seconds
  7. 2.50 seconds
  8. 2.75 seconds
  9. 3.00 seconds
  10. 3.25 seconds
  11. 3.50 seconds
  12. 3.75 seconds
  13. 4.0 seconds

Note: For this analysis, loading time is defined as the amount of time that elapses between when someone presses edit (init) and the editor is ready (ready).

Event Timeline

MNeisler triaged this task as Medium priority.
MNeisler edited projects, added Product-Analytics (Kanban); removed Product-Analytics.
MNeisler moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.

@ppelberg - Please see the analysis results summarized below and let me know if you have any questions or if there is any additional info that would be helpful in understanding when to show the fallback prompt.

Methodology
I reviewed the data collected during the Mobile VE as Default AB Test from 1 Nov 2019 through 26 September 2022. For this analysis, loading time is defined as the amount of time that elapses between when someone presses edit (init) and the editor is ready (ready).

Number and proportion of sessions that loaded mobile VE by default and would see the fallback experience were it be shown after the following elapsed loading times:

Elapsed Loading TimeNumber of SessionsProportion of Sessions
0.5 seconds101296194.5%
1.0 second23616674.8%
1.50 seconds44447352.8%
1.75 seconds47756844.5%
2.0 seconds40419337.7%
2.25 seconds34594432.3%
2.50 seconds29695227.7%
2.75 seconds25160623.5%
3.00 seconds21425320.0%
3.25 seconds18397617.2%
3.50 seconds15899814.8%
3.75 seconds13902413.0%
4.0 seconds12190411.4%

How to read numbers in table above:
About 20% of editing sessions would be shown the fallback experience if it were shown after 3 seconds and 11% would see the fallback experience if shown after 4 seconds.

Comparison of mobile VE load times to wikitext editor load times: (Further details are available in the AB test analysis report but highlighting some details here as it I think helps provide context to the distribution of load time)

Percentiles Summary:

load_time_distribution_comparison.png (545×928 px, 37 KB)

VE as default:
25th: 0.98 seconds (75% of sessions occur after this elapsed time)
50th (median): 1.6 seconds (50% of sessions occur before and after this elapsed load time)
75th: 2.7 seconds (25% of sessions occur after this time)

Source as default:
25th: 0.21 seconds (75% of sessions occur after this elapsed time)
50th (median): 0.38 seconds (50% of sessions occur before and after this elapsed load time)
75th: 0.67 seconds (25% of sessions occur after this time)

@ppelberg - Please see the analysis results summarized below and let me know if you have any questions or if there is any additional info that would be helpful in understanding when to show the fallback prompt.

This is wonderful – thank you, @MNeisler. In short, we're going to leave the current 3 second loading threshold as-is for now and reconsider when we prioritize work on T349231.


Context for the above...

The Editing Team discussed these results offline today in service of deciding, " 1. What – if any – adjustments will we make to the amount of loading time that will need to have elapsed before someone is shown the prompt T239676 introduces?"

We came to think that another piece of information of that would be useful to making this decision is knowing how much time needs to have elapsed for people to start thinking something like, "This is taking too long...".

To know how much time needs to have elapsed between someone tapping edit and the "This is taking too long..." thought emerging in their minds, we thought it would be useful to look at the distribution of mobile abandonment times. Although, the current instrumentation is *not* set up to answer this question. Thus why we've filed T349231 and prioritized it as part of T221185.

ppelberg updated the task description. (Show Details)