Page MenuHomePhabricator

Instrument the "time to suggested edits" metric
Open, Needs TriagePublic

Description

Proposed by @Gilles from the Performance Team at T240201#6132287, the idea is to collect client-side metrics for when the user first sees Suggested Edits completely loaded and interactive. This will be useful so we can measure the impact of our changes.

We will do something like:

mw.track( 'timing.MediaWiki.GrowthExperiments.timeToSuggestedEdits', mw.now() - initialLoad )

While we are looking at instrumentation for performance we could also consider:

  • Instrument server-side timing for Special:Homepage
  • client-side timing for all modules having loaded
  • Instrument various steps (search, API execution, AQS, RESTBase)

Event Timeline

kostajh created this task.Jul 7 2020, 8:53 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 7 2020, 8:53 PM

Change 610756 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] Instrument server-side render execution of Special:Homepage

https://gerrit.wikimedia.org/r/610756

Change 610758 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] SuggestedEdits: Instrument time to complete loading the module

https://gerrit.wikimedia.org/r/610758

Change 610770 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] Instrument GrowthTasksApi methods

https://gerrit.wikimedia.org/r/610770

client-side timing for all modules having loaded

I think we could probably set this one to the side for now, I'm not sure it is too useful while we are mainly focused on a single, heavier module (Suggested Edits).

Change 610756 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Instrument server-side render execution of Special:Homepage

https://gerrit.wikimedia.org/r/610756

Change 610758 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] SuggestedEdits: Instrument time to complete loading the module

https://gerrit.wikimedia.org/r/610758

Change 610770 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Instrument GrowthTasksApi methods

https://gerrit.wikimedia.org/r/610770

kostajh added a subscriber: Etonkovidova.

@Etonkovidova I can post links to Grafana once these patches are in production.

Client-side metrics can be viewed in Grafana under growthExperiments.specialHomepage., there are sections for suggested edits time to complete loading and also for the various API requests we make (under growthExperiments.specialHomepage.growthTasksApi)

@kostajh -- the links you posted rewrite me just to grafana.wikimedia.org. Do you have to permission them or something?

@kostajh - checked grafana today. The all options are there and the load times, so far, do not look extreme. I also try run Lighthouse reports which showed relatively high overall ranking for performance (88%) which is kind of puzzling especially on betalabs where all filters+all difficulty levels gives timeout:

Fetching task suggestions failed: http {xhr: {…}, textStatus: "timeout", exception: "timeout"

Questions:

  • The data in grafana is scarce - the data collection has only been started in wmf.41?
  • it's only aggregate data not by wikis or anything?

@kostajh you might want to create an actual Grafana dashboard for these metrics, as those "explore" links only work when already logged into Grafana.

@kostajh -- my issue was that I was not logged in. I do have a login, but since some Grafana dashboards are accessible publicly, I thought I was logged in, just not permissioned on your graphs. Now that I have them open, though, I think I would need to be walked through how to read them. I also see very little data on them. Perhaps you can show them at a team meeting next week.

Just responding quickly to one point: there will be very little data until the train reaches group2, all the data right now is coming from test wiki.

Re-checked today - there are more data now. The above links are not actual dashboards? There is grafana Special:Hmepoage/Suggested Edits dashboard.

Moving for PM review - note my question about whether the data is completely aggregated in my comment T257371#6306872.

I will add some charts to the dashboard after I've played around with Grafana a bit more.

kostajh removed kostajh as the assignee of this task.Jul 28 2020, 1:21 PM

From our meeting a week or two ago:

  • We want to segment the performance metrics by wiki -- maybe not for everything, but probably for "time to suggested edits" as well as server-side rendering time
  • we'd like to be able to see quickly a percentage of users who fall into the different buckets (0 - 2s, 2s-4s, 4s+), this should be possible with some Grafana skills
  • add a timeshift to show this week versus previous week

Unassigning myself in case someone else wants to pick it up.