How do we want to compare the current search performance to the new one? This task is to make sure we have the proper user journeys defined, implementing them in the Performance test repo, and setting up a dashboard.
This task does not cover any behavioral regression tests such as "the new UI is confusing."
User journeys
Legacy Vector:
For anon cached and uncached, and logged-in uncached scenarios:
- Visit the Barack Obama page.
- Tap search.
- Type "b".
- Wait for a response.
- Type "ananb" one character at a time.
- Wait for a response.
- Type "<backspace>a" one character at a time.
- Wait for a response.
- Tap the first result.
- Wait for the page to load.
Latest Vector:
For anon cached and uncached, and logged-in uncached scenarios:
- Visit the Barack Obama page.
- Tap search.
- Wait for the form to load.
- Type "b".
- Wait for a response.
- Type "ananb" one character at a time.
- Wait for a response.
- Type "<backspace>a" one character at a time.
- Wait for a response.
- Tap the first result.
- Wait for the page to load.
Questions
- Confirm with @ovasileva and @alexhollender that these journeys make sense.
- Is the Barack Obama page a good test for the tests wikis? The Performance team has also used the Sweden and Facebook pages. Is there a universal page that works on the test wikis (French Wikipedia and Wiktionary, Hebrew Wikipedia, Portuguese Wikiversity, Basque Wikipedia, and Persian Wikipedia)? Answer: there is no universal page. The Obama page is usually a good example of a large page. See T251544#6103760.
- Do we want to also test Special:BlankPage? Answer: no, but we may want to test an empty page in the user namespace. See T251544#6103760.
- Are parameters possible or do these break caching (see T215088#4993170)? E.g., safemode=1, useskin=vector&useskinversion=2, and banner=null. If not, how do we set the skin version and enable features?
- Do we need to test loading the largest dependencies (e.g., Vue) without using them (see T250336#6075053)?
Dashboards
For Legacy and Latest modes, mimic at least the metrics described in the reference dashboard:
- TTFB
- FirstVisualChange
- Heading
- LargestImage
- LastVisualChange
- SpeedIndex
- PerceptualSpeedIndex
- Fully Loaded
- ...
Both reports should be comparable on the same dashboard.
Acceptance criteria
- Legacy and latest reports are comparable on the test wikis.
- Chores are updated to include the new dashboards
- The wiki (either a link off of the Vue.js page or one of the Performance pages) is updated with recommendations for future Vue.js projects
- Performance team signs off on tests and dashboards