Page MenuHomePhabricator
Create Task
Maniphest T251544

Add user journey performance tests for Vector's Legacy and Vue.js search
Open, MediumPublic5 Estimated Story Points
Actions

Description

How do we want to compare the current search performance to the new one? This task is to make sure we have the proper user journeys defined, implementing them in the Performance test repo, and setting up a dashboard.

This task does not cover any behavioral regression tests such as "the new UI is confusing."

User journeys

Legacy Vector:

For anon cached and uncached, and logged-in uncached scenarios:

  1. Visit the Barack Obama page.
  2. Tap search.
  3. Type "b".
  4. Wait for a response.
  5. Type "ananb" one character at a time.
  6. Wait for a response.
  7. Type "<backspace>a" one character at a time.
  8. Wait for a response.
  9. Tap the first result.
  10. Wait for the page to load.

Latest Vector:

For anon cached and uncached, and logged-in uncached scenarios:

  1. Visit the Barack Obama page.
  2. Tap search.
  3. Wait for the form to load.
  4. Type "b".
  5. Wait for a response.
  6. Type "ananb" one character at a time.
  7. Wait for a response.
  8. Type "<backspace>a" one character at a time.
  9. Wait for a response.
  10. Tap the first result.
  11. Wait for the page to load.

Questions

  • Confirm with @ovasileva and @alexhollender that these journeys make sense.
  • Is the Barack Obama page a good test for the tests wikis? The Performance team has also used the Sweden and Facebook pages. Is there a universal page that works on the test wikis (French Wikipedia and Wiktionary, Hebrew Wikipedia, Portuguese Wikiversity, Basque Wikipedia, and Persian Wikipedia)? Answer: there is no universal page. The Obama page is usually a good example of a large page. See T251544#6103760.
  • Do we want to also test Special:BlankPage? Answer: no, but we may want to test an empty page in the user namespace. See T251544#6103760.
  • Are parameters possible or do these break caching (see T215088#4993170)? E.g., safemode=1, useskin=vector&useskinversion=2, and banner=null. If not, how do we set the skin version and enable features?
  • Do we need to test loading the largest dependencies (e.g., Vue) without using them (see T250336#6075053)?

Dashboards

For Legacy and Latest modes, mimic at least the metrics described in the reference dashboard:

  • TTFB
  • FirstVisualChange
  • Heading
  • LargestImage
  • LastVisualChange
  • SpeedIndex
  • PerceptualSpeedIndex
  • Fully Loaded
  • ...

Both reports should be comparable on the same dashboard.

Acceptance criteria

  • Legacy and latest reports are comparable on the test wikis.
  • Chores are updated to include the new dashboards
  • The wiki (either a link off of the Vue.js page or one of the Performance pages) is updated with recommendations for future Vue.js projects
  • Performance team signs off on tests and dashboards

Details

ProjectBranchLines +/-Subject
performance/synthetic-monitoring-testsmaster+23 -5Add Wvui search script and point legacy/Wvui search tests to beta
mediawiki/skins/Vectormaster+63 -1[search] Add Vue search performance instrumentation
wvuimaster+105 -30[search] Emit `fetch-start` and `fetch-end` events
performance/synthetic-monitoring-testsmaster+41 -51Revise legacyObama search
mediawiki/coremaster+97 -7Add client-side performance metrics for legacy search
mediawiki/skins/Vectormaster+40 -4Add client-side performance metrics for searchLoader
performance/synthetic-monitoring-testsmaster+52 -0POC: Add user journey performance tests for Legacy Vector Search
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Niedzielski updated the task description. (Show Details)Apr 30 2020, 5:18 PM
Gilles added a comment.May 4 2020, 9:07 AM

The Barack Obama page is an extreme one in terms of size and amount of subresources, but due to that it might highlight some issues that otherwise wouldn't be clear, such as the bandwidth competing between the article's images and the JS you're loading for this feature to work. On that note, you have to figure out bandwidth conditions that you want to use during your tests.

And in terms of interacting with the page, some users don't want for the page to be fully loaded to do so. That's also something where the combination of a large article + constrained bandwidth would be useful to see the range of times that the feature actually activates based on when the low priority JS got a chance to load (sometimes after the user interaction).

We don't have a universal test page across wiki for perf tests.

I would advise against Special:BlankPage, as it's uncached. Making an empty article would be better to test this feature, if you put it in the User namespace it's highly likely that nobody will touch it.

I think that testing the loading on its own is useful to determine the cost. It's all about tradeoffs and to make an educated decision about loading this code on every page or not and how, you need to know what the blanket cost is going to be for the majority of pageviews where the feature will remain unused.

Niedzielski updated the task description. (Show Details)May 4 2020, 3:23 PM
Niedzielski edited projects, added Vue.js (Vue.js-Search); removed Vue.js.May 6 2020, 2:18 PM
Niedzielski moved this task from Backlog to Analysis on the Vue.js (Vue.js-Search) board.May 6 2020, 2:42 PM
Niedzielski mentioned this in T249306: [Spike] How should we load Vue.js and dependencies for Vue.js search?.May 11 2020, 2:46 AM
Niedzielski moved this task from Analysis to Ready on the Vue.js (Vue.js-Search) board.
Krinkle moved this task from Limbo to Watching on the Performance-Team (Radar) board.May 21 2020, 1:34 AM
Niedzielski added a comment.Jun 7 2020, 7:07 PM

@alexhollender, this is especially pertinent to our recent performance training. Can I ask you to review these journeys with @ovasileva while the learnings are fresh?

ovasileva added a comment.EditedJun 8 2020, 9:43 AM

User journeys look good, but I will also review with @alexhollender. One note: I would say that maybe something more global might make more sense for testing on test wikis (https://en.wikipedia.org/wiki/Moon for example - compare https://eu.wikipedia.org/wiki/Ilargia to https://eu.wikipedia.org/wiki/Barack_Obama)

nray claimed this task.Aug 10 2020, 9:25 PM
nray added a project: Readers-Web-Backlog (Kanbanana-FY-2020-21).
nray moved this task from Needs Analysis to Doing on the Readers-Web-Backlog (Kanbanana-FY-2020-21) board.
nray moved this task from Ready to On Web Board on the Vue.js (Vue.js-Search) board.Aug 11 2020, 11:54 PM
gerritbot added a comment.Aug 13 2020, 4:18 PM

Change 620062 had a related patch set uploaded (by Nray; owner: Nray):
[performance/synthetic-monitoring-tests@master] POC: Add user journey performance tests for Legacy Vector Search

https://gerrit.wikimedia.org/r/620062

gerritbot added a project: Patch-For-Review.Aug 13 2020, 4:18 PM
nray added a comment.Aug 13 2020, 4:27 PM

@Peter First, great work with Sitespeed.io! I found it incredibly powerful.

Do you have time to look over my POC patch https://gerrit.wikimedia.org/r/620062 that attempts to implement the user journey for legacy Vector search as specified in this ticket? I'd love your valuable feedback. I'm sure some things in my patch could be improved :)

ovasileva triaged this task as Medium priority.Aug 24 2020, 5:08 PM
ovasileva set the point value for this task to 5.Aug 25 2020, 4:20 PM
gerritbot added a comment.Aug 27 2020, 11:38 AM

Change 620062 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] POC: Add user journey performance tests for Legacy Vector Search

https://gerrit.wikimedia.org/r/620062

Maintenance_bot removed a project: Patch-For-Review.Aug 27 2020, 12:10 PM
gerritbot added a comment.Sep 4 2020, 7:09 PM

Change 624824 had a related patch set uploaded (by Nray; owner: Nray):
[mediawiki/core@master] Add client-side performance metrics for legacy search

https://gerrit.wikimedia.org/r/624824

Restricted Application added a subscriber: Huji. · View Herald TranscriptSep 4 2020, 7:09 PM
gerritbot added a project: Patch-For-Review.Sep 4 2020, 7:09 PM

Change 624825 had a related patch set uploaded (by Nray; owner: Nray):
[performance/synthetic-monitoring-tests@master] Revise legacyObama search

https://gerrit.wikimedia.org/r/624825

gerritbot added a comment.Sep 4 2020, 7:10 PM

Change 624826 had a related patch set uploaded (by Nray; owner: Nray):
[performance/synthetic-monitoring-tests@master] WIP: Add Vue search script

https://gerrit.wikimedia.org/r/624826

gerritbot added a comment.Sep 4 2020, 8:21 PM

Change 624888 had a related patch set uploaded (by Nray; owner: Nray):
[mediawiki/skins/Vector@master] Add client-side performance metrics for searchLoader

https://gerrit.wikimedia.org/r/624888

nray added a comment.Sep 4 2020, 8:30 PM

Update: I have two more metrics to add when the TypeadheadSearch component is complete, but I think those should be pretty easy and the above patches can be reviewed now.

nray moved this task from Doing to Needs Code Review on the Readers-Web-Backlog (Kanbanana-FY-2020-21) board.Sep 4 2020, 8:30 PM
nray moved this task from Needs Code Review to Doing on the Readers-Web-Backlog (Kanbanana-FY-2020-21) board.Sep 14 2020, 3:50 PM
nray moved this task from Doing to Needs Code Review on the Readers-Web-Backlog (Kanbanana-FY-2020-21) board.Sep 16 2020, 1:13 AM
gerritbot added a comment.Sep 18 2020, 12:28 AM

Change 628240 had a related patch set uploaded (by Nray; owner: Nick Ray):
[wvui@master] WIP: Add client-side performance metrics for vue search

https://gerrit.wikimedia.org/r/628240

WMDE-leszek added a subscriber: WMDE-leszek.Sep 22 2020, 5:26 AM
nray reassigned this task from nray to Peter.Sep 23 2020, 3:46 PM
phuedx moved this task from Needs Code Review to Needs More Work on the Readers-Web-Backlog (Kanbanana-FY-2020-21) board.Oct 5 2020, 11:18 AM

I've left some feedback on https://gerrit.wikimedia.org/r/628240.

nray moved this task from Needs More Work to Needs Code Review on the Readers-Web-Backlog (Kanbanana-FY-2020-21) board.Oct 6 2020, 5:07 PM
Gilles added a comment.Oct 9 2020, 12:26 PM

What's the expectation/next step for @Peter and the perf team? Reviewing the patches linked here that aren't [WIP] nor [search]? Do you need anything else?

nray added a comment.Oct 9 2020, 8:12 PM

@Gilles, these are patches I was hoping the performance team could review:

Add client-side performance metrics for legacy search
Add client-side performance metrics for searchLoader
Revise legacyObama search (Note: Peter already +2d this, I just had a small revision)
WIP: Add Vue search script (Note: This is currently not ready for review as it is blocked on our release of the Vue search component, but it will eventually need review)

If your team reviews those, I think our team will be in good shape to review the rest of the patches.

gerritbot added a comment.Oct 15 2020, 8:47 AM

Change 624824 merged by jenkins-bot:
[mediawiki/core@master] Add client-side performance metrics for legacy search

https://gerrit.wikimedia.org/r/624824

gerritbot added a comment.Oct 15 2020, 8:47 AM

Change 624888 merged by jenkins-bot:
[mediawiki/skins/Vector@master] Add client-side performance metrics for searchLoader

https://gerrit.wikimedia.org/r/624888

ReleaseTaggerBot added a project: MW-1.36-notes (1.36.0-wmf.14; 2020-10-20).Oct 15 2020, 9:00 AM
nray claimed this task.Oct 15 2020, 3:54 PM
nray added a comment.Oct 15 2020, 4:06 PM

\o/ All the patches that were merge candidates have now been merged (thank you Performance team!). The remaining work on this is blocked on the Vue search component work:

  1. Emit fetch-start and fetch-end events (from TypeaheadSearch component)
  2. Add metrics in App.vue that integrates with events from step 1 (pending)
  3. WIP: Add Vue search script (small patch in the synthetic performance repo)
nray moved this task from Needs Code Review to Blocked on Others on the Readers-Web-Backlog (Kanbanana-FY-2020-21) board.Oct 15 2020, 4:06 PM
gerritbot added a comment.Oct 21 2020, 9:30 AM

Change 624825 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Revise legacyObama search

https://gerrit.wikimedia.org/r/624825

nray moved this task from Blocked on Others to Doing on the Readers-Web-Backlog (Kanbanana-FY-2020-21) board.Oct 26 2020, 6:44 PM
nray moved this task from Doing to Needs Code Review on the Readers-Web-Backlog (Kanbanana-FY-2020-21) board.Oct 27 2020, 12:20 AM

Now that the typeahead search component has been merged, the next patch up for review (to be reviewed by the Web team) is:

https://gerrit.wikimedia.org/r/c/wvui/+/628240

nray removed nray as the assignee of this task.Oct 28 2020, 4:00 PM
gerritbot added a comment.Oct 30 2020, 11:51 AM

Change 628240 merged by jenkins-bot:
[wvui@master] [search] Emit fetch-start and fetch-end events

https://gerrit.wikimedia.org/r/628240

phuedx moved this task from Needs Code Review to Doing on the Readers-Web-Backlog (Kanbanana-FY-2020-21) board.Oct 30 2020, 11:57 AM
In T251544#6547319, @nray wrote:
  1. WIP: Add Vue search script (small patch in the synthetic performance repo)

AIUI this is the only patch that's left. I'm being bold and moving this back to Doing.

Jdlrobson added a parent task: T257865: Merge feat/search to master for Vue.js search deployment.Oct 30 2020, 5:45 PM
nray claimed this task.Nov 2 2020, 5:00 PM
nray moved this task from Doing to Blocked on Others on the Readers-Web-Backlog (Kanbanana-FY-2020-21) board.Nov 2 2020, 6:20 PM
nray added a comment.Nov 2 2020, 6:36 PM

Final piece remaining on this depends on the integration bit (T264355) so I have moved this to blocked

nray moved this task from Blocked on Others to Doing on the Readers-Web-Backlog (Kanbanana-FY-2020-21) board.Nov 24 2020, 12:51 AM
gerritbot added a comment.Dec 1 2020, 10:44 PM

Change 644631 had a related patch set uploaded (by Nray; owner: Nray):
[mediawiki/skins/Vector@master] [search] Add Vue search performance instrumentation

https://gerrit.wikimedia.org/r/644631

nray added a comment.EditedDec 1 2020, 10:51 PM

The above patch is dependent on the integration patch and part of @phuedx 's patch "Instrument Vue.js-based search widget" . It's not ready for merge yet, but if anyone from the web team has bandwidth feel free to review :)

nray moved this task from Doing to Needs Code Review on the Readers-Web-Backlog (Kanbanana-FY-2020-21) board.Dec 1 2020, 10:54 PM
Jdlrobson mentioned this in T249298: [Epic] Establish initial project scaffolding for Vue.js search iterations.Dec 2 2020, 5:50 PM
nray removed nray as the assignee of this task.Dec 2 2020, 6:04 PM
Jdlrobson moved this task from Needs Code Review to Needs More Work on the Readers-Web-Backlog (Kanbanana-FY-2020-21) board.Dec 7 2020, 10:54 PM

https://gerrit.wikimedia.org/r/c/mediawiki/skins/Vector/+/643355 has a -1.

nray moved this task from Needs More Work to Blocked on Others on the Readers-Web-Backlog (Kanbanana-FY-2020-21) board.Dec 7 2020, 11:15 PM
Jdlrobson moved this task from Blocked on Others to Needs Code Review on the Readers-Web-Backlog (Kanbanana-FY-2020-21) board.Dec 8 2020, 10:29 PM
Jdlrobson assigned this task to Edtadros.Dec 8 2020, 11:43 PM
Jdlrobson moved this task from Needs Code Review to Needs QA on the Readers-Web-Backlog (Kanbanana-FY-2020-21) board.
gerritbot added a comment.Dec 9 2020, 12:06 AM

Change 644631 merged by jenkins-bot:
[mediawiki/skins/Vector@master] [search] Add Vue search performance instrumentation

https://gerrit.wikimedia.org/r/644631

ReleaseTaggerBot edited projects, added MW-1.36-notes (1.36.0-wmf.22; 2020-12-15); removed MW-1.36-notes (1.36.0-wmf.14; 2020-10-20).Dec 9 2020, 1:00 AM
ovasileva moved this task from Needs QA to Blocked on Others on the Readers-Web-Backlog (Kanbanana-FY-2020-21) board.Dec 9 2020, 6:11 PM
nray removed Edtadros as the assignee of this task.Dec 9 2020, 6:14 PM
nray added a subscriber: Edtadros.
nray added a comment.EditedDec 9 2020, 6:35 PM

Now that the Vector/WVUI performance metric recording code has been merged, the next (and hopefully last) patch is the one in the synthetic performance repo that adds a test to collect these Vue search metrics:

https://gerrit.wikimedia.org/r/c/performance/synthetic-monitoring-tests/+/624826

However, it needs a url to run the test against. Since we have not launched the new search anywhere yet, this is moved back to the blocked column.

Jdlrobson assigned this task to Peter.Dec 9 2020, 8:37 PM

Assigning Peter since he's assigned to https://gerrit.wikimedia.org/r/c/performance/synthetic-monitoring-tests/+/624826

Jdlrobson moved this task from Blocked on Others to Needs Code Review on the Readers-Web-Backlog (Kanbanana-FY-2020-21) board.Fri, Feb 12, 1:50 AM
ovasileva moved this task from Needs Code Review to Blocked on Others on the Readers-Web-Backlog (Kanbanana-FY-2020-21) board.Mon, Feb 15, 6:13 PM
nray claimed this task.Mon, Feb 22, 6:03 PM
nray moved this task from Blocked on Others to Doing on the Readers-Web-Backlog (Kanbanana-FY-2020-21) board.
nray moved this task from Doing to Blocked on Others on the Readers-Web-Backlog (Kanbanana-FY-2020-21) board.Tue, Feb 23, 1:28 AM
gerritbot added a comment.Tue, Feb 23, 8:05 AM

Change 624826 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Add Wvui search script and point legacy/Wvui search tests to beta

https://gerrit.wikimedia.org/r/624826

Maintenance_bot removed a project: Patch-For-Review.Tue, Feb 23, 8:10 AM
nray added a comment.EditedTue, Feb 23, 5:21 PM

Today I met with @Peter to talk about the steps remaining for this ticket. Peter was very helpful and now we have a really cool dashboard thanks to him!

https://grafana.wikimedia.org/d/GivPpdsGk/vue-vs-legacy-search?orgId=1

Note that this dashboard is currently collecting metrics for legacy and Wvui search on the beta cluster. I will update it to point to production when we have deployed there. It is also worth noting that these are synthetic performance tests [1] meaning that they are running automated tests rather than collecting metrics from real users (although the same code could in theory be used to test real users if desired)

[1] https://wikitech.wikimedia.org/wiki/Performance/Synthetic_testing

nray moved this task from Blocked on Others to Doing on the Readers-Web-Backlog (Kanbanana-FY-2020-21) board.Wed, Feb 24, 4:50 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL