Page MenuHomePhabricator

Add user journey performance tests for Vector's Legacy and Vue.js search
Open, MediumPublic5 Estimated Story Points

Description

How do we want to compare the current search performance to the new one? This task is to make sure we have the proper user journeys defined, implementing them in the Performance test repo, and setting up a dashboard.

This task does not cover any behavioral regression tests such as "the new UI is confusing."

User journeys

Legacy Vector:

For anon cached and uncached, and logged-in uncached scenarios:

  1. Visit the Barack Obama page.
  2. Tap search.
  3. Type "b".
  4. Wait for a response.
  5. Type "ananb" one character at a time.
  6. Wait for a response.
  7. Type "<backspace>a" one character at a time.
  8. Wait for a response.
  9. Tap the first result.
  10. Wait for the page to load.

Latest Vector:

For anon cached and uncached, and logged-in uncached scenarios:

  1. Visit the Barack Obama page.
  2. Tap search.
  3. Wait for the form to load.
  4. Type "b".
  5. Wait for a response.
  6. Type "ananb" one character at a time.
  7. Wait for a response.
  8. Type "<backspace>a" one character at a time.
  9. Wait for a response.
  10. Tap the first result.
  11. Wait for the page to load.

Questions

  • Confirm with @ovasileva and @alexhollender that these journeys make sense.
  • Is the Barack Obama page a good test for the tests wikis? The Performance team has also used the Sweden and Facebook pages. Is there a universal page that works on the test wikis (French Wikipedia and Wiktionary, Hebrew Wikipedia, Portuguese Wikiversity, Basque Wikipedia, and Persian Wikipedia)? Answer: there is no universal page. The Obama page is usually a good example of a large page. See T251544#6103760.
  • Do we want to also test Special:BlankPage? Answer: no, but we may want to test an empty page in the user namespace. See T251544#6103760.
  • Are parameters possible or do these break caching (see T215088#4993170)? E.g., safemode=1, useskin=vector&useskinversion=2, and banner=null. If not, how do we set the skin version and enable features?
  • Do we need to test loading the largest dependencies (e.g., Vue) without using them (see T250336#6075053)?

Dashboards

For Legacy and Latest modes, mimic at least the metrics described in the reference dashboard:

  • TTFB
  • FirstVisualChange
  • Heading
  • LargestImage
  • LastVisualChange
  • SpeedIndex
  • PerceptualSpeedIndex
  • Fully Loaded
  • ...

Both reports should be comparable on the same dashboard.

Acceptance criteria

  • Legacy and latest reports are comparable on the test wikis.
  • Chores are updated to include the new dashboards
  • The wiki (either a link off of the Vue.js page or one of the Performance pages) is updated with recommendations for future Vue.js projects
  • Performance team signs off on tests and dashboards

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Niedzielski updated the task description. (Show Details)Apr 30 2020, 5:18 PM
Gilles added a comment.May 4 2020, 9:07 AM

The Barack Obama page is an extreme one in terms of size and amount of subresources, but due to that it might highlight some issues that otherwise wouldn't be clear, such as the bandwidth competing between the article's images and the JS you're loading for this feature to work. On that note, you have to figure out bandwidth conditions that you want to use during your tests.

And in terms of interacting with the page, some users don't want for the page to be fully loaded to do so. That's also something where the combination of a large article + constrained bandwidth would be useful to see the range of times that the feature actually activates based on when the low priority JS got a chance to load (sometimes after the user interaction).

We don't have a universal test page across wiki for perf tests.

I would advise against Special:BlankPage, as it's uncached. Making an empty article would be better to test this feature, if you put it in the User namespace it's highly likely that nobody will touch it.

I think that testing the loading on its own is useful to determine the cost. It's all about tradeoffs and to make an educated decision about loading this code on every page or not and how, you need to know what the blanket cost is going to be for the majority of pageviews where the feature will remain unused.

Niedzielski updated the task description. (Show Details)May 4 2020, 3:23 PM

@alexhollender, this is especially pertinent to our recent performance training. Can I ask you to review these journeys with @ovasileva while the learnings are fresh?

ovasileva added a comment.EditedJun 8 2020, 9:43 AM

User journeys look good, but I will also review with @alexhollender. One note: I would say that maybe something more global might make more sense for testing on test wikis (https://en.wikipedia.org/wiki/Moon for example - compare https://eu.wikipedia.org/wiki/Ilargia to https://eu.wikipedia.org/wiki/Barack_Obama)

Change 620062 had a related patch set uploaded (by Nray; owner: Nray):
[performance/synthetic-monitoring-tests@master] POC: Add user journey performance tests for Legacy Vector Search

https://gerrit.wikimedia.org/r/620062

nray added a comment.Aug 13 2020, 4:27 PM

@Peter First, great work with Sitespeed.io! I found it incredibly powerful.

Do you have time to look over my POC patch https://gerrit.wikimedia.org/r/620062 that attempts to implement the user journey for legacy Vector search as specified in this ticket? I'd love your valuable feedback. I'm sure some things in my patch could be improved :)

ovasileva triaged this task as Medium priority.Aug 24 2020, 5:08 PM
ovasileva set the point value for this task to 5.Aug 25 2020, 4:20 PM

Change 620062 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] POC: Add user journey performance tests for Legacy Vector Search

https://gerrit.wikimedia.org/r/620062

Change 624824 had a related patch set uploaded (by Nray; owner: Nray):
[mediawiki/core@master] Add client-side performance metrics for legacy search

https://gerrit.wikimedia.org/r/624824

Restricted Application added a subscriber: Huji. · View Herald TranscriptSep 4 2020, 7:09 PM

Change 624825 had a related patch set uploaded (by Nray; owner: Nray):
[performance/synthetic-monitoring-tests@master] Revise legacyObama search

https://gerrit.wikimedia.org/r/624825

Change 624826 had a related patch set uploaded (by Nray; owner: Nray):
[performance/synthetic-monitoring-tests@master] WIP: Add Vue search script

https://gerrit.wikimedia.org/r/624826

Change 624888 had a related patch set uploaded (by Nray; owner: Nray):
[mediawiki/skins/Vector@master] Add client-side performance metrics for searchLoader

https://gerrit.wikimedia.org/r/624888

nray added a comment.Sep 4 2020, 8:30 PM

Update: I have two more metrics to add when the TypeadheadSearch component is complete, but I think those should be pretty easy and the above patches can be reviewed now.

Change 628240 had a related patch set uploaded (by Nray; owner: Nick Ray):
[wvui@master] WIP: Add client-side performance metrics for vue search

https://gerrit.wikimedia.org/r/628240

nray reassigned this task from nray to Peter.Sep 23 2020, 3:46 PM

What's the expectation/next step for @Peter and the perf team? Reviewing the patches linked here that aren't [WIP] nor [search]? Do you need anything else?

nray added a comment.Oct 9 2020, 8:12 PM

@Gilles, these are patches I was hoping the performance team could review:

Add client-side performance metrics for legacy search
Add client-side performance metrics for searchLoader
Revise legacyObama search (Note: Peter already +2d this, I just had a small revision)
WIP: Add Vue search script (Note: This is currently not ready for review as it is blocked on our release of the Vue search component, but it will eventually need review)

If your team reviews those, I think our team will be in good shape to review the rest of the patches.

Change 624824 merged by jenkins-bot:
[mediawiki/core@master] Add client-side performance metrics for legacy search

https://gerrit.wikimedia.org/r/624824

Change 624888 merged by jenkins-bot:
[mediawiki/skins/Vector@master] Add client-side performance metrics for searchLoader

https://gerrit.wikimedia.org/r/624888

nray claimed this task.Oct 15 2020, 3:54 PM
nray added a comment.Oct 15 2020, 4:06 PM

\o/ All the patches that were merge candidates have now been merged (thank you Performance team!). The remaining work on this is blocked on the Vue search component work:

  1. Emit fetch-start and fetch-end events (from TypeaheadSearch component)
  2. Add metrics in App.vue that integrates with events from step 1 (pending)
  3. WIP: Add Vue search script (small patch in the synthetic performance repo)

Change 624825 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Revise legacyObama search

https://gerrit.wikimedia.org/r/624825

Now that the typeahead search component has been merged, the next patch up for review (to be reviewed by the Web team) is:

https://gerrit.wikimedia.org/r/c/wvui/+/628240

nray removed nray as the assignee of this task.Oct 28 2020, 4:00 PM

Change 628240 merged by jenkins-bot:
[wvui@master] [search] Emit fetch-start and fetch-end events

https://gerrit.wikimedia.org/r/628240

  1. WIP: Add Vue search script (small patch in the synthetic performance repo)

AIUI this is the only patch that's left. I'm being bold and moving this back to Doing.

nray claimed this task.Nov 2 2020, 5:00 PM
nray added a comment.Nov 2 2020, 6:36 PM

Final piece remaining on this depends on the integration bit (T264355) so I have moved this to blocked

Change 644631 had a related patch set uploaded (by Nray; owner: Nray):
[mediawiki/skins/Vector@master] [search] Add Vue search performance instrumentation

https://gerrit.wikimedia.org/r/644631

nray added a comment.EditedDec 1 2020, 10:51 PM

The above patch is dependent on the integration patch and part of @phuedx 's patch "Instrument Vue.js-based search widget" . It's not ready for merge yet, but if anyone from the web team has bandwidth feel free to review :)

nray removed nray as the assignee of this task.Dec 2 2020, 6:04 PM

Change 644631 merged by jenkins-bot:
[mediawiki/skins/Vector@master] [search] Add Vue search performance instrumentation

https://gerrit.wikimedia.org/r/644631

nray removed Edtadros as the assignee of this task.Dec 9 2020, 6:14 PM
nray added a subscriber: Edtadros.
nray added a comment.EditedDec 9 2020, 6:35 PM

Now that the Vector/WVUI performance metric recording code has been merged, the next (and hopefully last) patch is the one in the synthetic performance repo that adds a test to collect these Vue search metrics:

https://gerrit.wikimedia.org/r/c/performance/synthetic-monitoring-tests/+/624826

However, it needs a url to run the test against. Since we have not launched the new search anywhere yet, this is moved back to the blocked column.

Change 624826 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Add Wvui search script and point legacy/Wvui search tests to beta

https://gerrit.wikimedia.org/r/624826

nray added a comment.EditedTue, Feb 23, 5:21 PM

Today I met with @Peter to talk about the steps remaining for this ticket. Peter was very helpful and now we have a really cool dashboard thanks to him!

https://grafana.wikimedia.org/d/GivPpdsGk/vue-vs-legacy-search?orgId=1

Note that this dashboard is currently collecting metrics for legacy and Wvui search on the beta cluster. I will update it to point to production when we have deployed there. It is also worth noting that these are synthetic performance tests [1] meaning that they are running automated tests rather than collecting metrics from real users (although the same code could in theory be used to test real users if desired)

[1] https://wikitech.wikimedia.org/wiki/Performance/Synthetic_testing