Page MenuHomePhabricator

Significant performance degradation in new (split) notifications
Closed, ResolvedPublic

Description

Ever since the new (split) notifications have been deployed to the wikis, I'm experiencing *significant* degradation in the time between I click the notification icon and any visible change on the user interface. Even on WMF's fast office network, I just waited *six seconds* for anything to appear. I am observing this at least on Meta, English Wikipedia, and Hebrew Wikipedia.

This makes the feature as a whole feel like a significant regression. :(

Event Timeline

Ijon created this task.Sep 28 2015, 7:23 PM
Ijon updated the task description. (Show Details)
Ijon raised the priority of this task from to Needs Triage.
Ijon added a project: Notifications.
Ijon added a subscriber: Ijon.
Restricted Application added a project: Collaboration-Team-Triage. · View Herald TranscriptSep 28 2015, 7:23 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

TLDR: numbers show that the flyout has gotten slower, but also that the 6 seconds you experienced is unusually slow (in the 5% of slowest times)

https://grafana.wikimedia.org/#/dashboard/db/echoflyout (ignore the units, milliseconds should be seconds) shows that the flyout has gotten slower, but also that 6 seconds is near the 95th percentile value. The median time was 400-600ms and is now 700-900ms, and the 75th percentile time went from 1-1.5 to 2-2.5 seconds. The 95th percentile was already bad at 3-4 seconds (see T56486: Echo flyout takes too long to open) but is now worse at 4-8 seconds.

We are deploying https://gerrit.wikimedia.org/r/241758 today which will give us more information about which parts of loading the flyout are slow and how slow they are.

Ijon added a comment.Sep 30 2015, 1:51 AM

Great, thanks!

(was I really the first one to file a bug about it in the current iteration? o_O)

Catrope set Security to None.
Catrope added a subscriber: Mattflaschen-WMF.

@Mattflaschen suggests we separate our metrics for the two different flyouts.

Catrope triaged this task as Normal priority.Sep 30 2015, 11:30 PM

Judging from the charts[0] (& confirmed by the date of this ticket), the performance degradation started on 24-25 September.
24 September is when 1.26wmf24 was deployed[1]. In it was code that was merged between 15 & 22 September, the dates of 1.26wmf23 & 1.26wmf24 branch cuts.
Looking at the code that was merged around that time, the one that only loads ext.echo.ui (incl. ooui widgets) if the user clicks the echo badge[2] is a very likely candidate: opening the widget now comes with a lot of additional resources.

The sudden performance was intentional: instead of loading all of those things on page-load, it was deferred until (and if) we actually need them. But then when we do need them, it'll take some time to load.

Maybe, in ext.echo.init.js we could check if the user has unread alerts/messages and (if yes) start loading (async) all the modules?

0: https://grafana.wikimedia.org/dashboard/db/echoflyout
1: https://wikitech.wikimedia.org/wiki/Deployments/Archive/2015/09
2: https://gerrit.wikimedia.org/r/#/c/238389/

Change 271503 had a related patch set uploaded (by Matthias Mullie):
Preload flyout modules if user has unread messages

https://gerrit.wikimedia.org/r/271503

We could probably lighten the impact of loading oojs-ui as well by loading only the parts of it that we need, now that T113681: Split oojs-ui module into smaller ones is done.

Change 271945 had a related patch set uploaded (by Catrope):
Use more granular oojs-ui dependencies

https://gerrit.wikimedia.org/r/271945

Change 271945 merged by jenkins-bot:
Use more granular oojs-ui dependencies

https://gerrit.wikimedia.org/r/271945

Catrope removed matthiasmullie as the assignee of this task.Apr 19 2016, 4:45 PM
Catrope added a subscriber: matthiasmullie.
Restricted Application added a subscriber: TerraCodes. · View Herald TranscriptApr 19 2016, 4:45 PM

Change 271503 abandoned by Matthias Mullie:
Preload flyout modules if user has unread messages

https://gerrit.wikimedia.org/r/271503

jmatazzoni closed this task as Resolved.Oct 11 2017, 4:46 PM