Page MenuHomePhabricator
Create Task
Maniphest T104396

Investigate distribution of browser test run time
Closed, DeclinedPublic
Actions

Description

I'd like to know how long it takes the majority of our browser tests to run.

I'd also like to know which tests suites take the longest (and how far from median they are) to identify high reward areas to improve.

Related Objects

Event Timeline

greg created this task.Jun 30 2015, 10:37 PM
greg raised the priority of this task from to Needs Triage.
greg updated the task description. (Show Details)
greg added a project: Browser-Tests-Infrastructure.
greg subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 30 2015, 10:37 PM
hashar subscribed.Aug 25 2015, 3:40 PM

One of the slowdown are the network roundtrips to SauceLabs. In some cases there are thousands of them and that is tracked at T92613

zeljkofilipin claimed this task.Aug 25 2015, 3:49 PM
zeljkofilipin triaged this task as Medium priority.
zeljkofilipin set Security to None.
zeljkofilipin moved this task from Inbox to In Progress on the Browser-Tests-Infrastructure board.
zeljkofilipin subscribed.
hashar added a subscriber: • dduvall.Aug 26 2015, 7:06 AM

I am speculating we have timing in Raita?

zeljkofilipin added a comment.Sep 2 2015, 4:44 PM

Not sure how to get build times from raita:

http://raita.wmflabs.org/#builds/latest

zeljkofilipin added a comment.Sep 2 2015, 4:46 PM

All browser tests in Jenkins, 59 jobs at the moment:

https://integration.wikimedia.org/ci/view/BrowserTests/view/-All/

zeljkofilipin added a comment.EditedSep 2 2015, 4:46 PM

Build time trend for one job:

https://integration.wikimedia.org/ci/job/browsertests-CentralAuth-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/buildTimeTrend

API docs

https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API

Example for the job

https://integration.wikimedia.org/ci/job/browsertests-CentralAuth-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/api/

https://integration.wikimedia.org/ci/job/browsertests-CentralAuth-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/api/json?pretty=true

zeljkofilipin added a comment.Sep 2 2015, 5:22 PM

I have created a simple spreadsheet:

https://docs.google.com/spreadsheets/d/1xrJxTVYPx1uCVKXC5Qrdc1a0LLctxYm_PIFG65yjw2A/edit?usp=sharing

The data is copy/pasted from jenkins, that is why I limited it to 5 jobs for now. If it provides the data you need, I can automate the process.

For now it only has 5 jobs and lists only average time for each job. I will add another dashboard with more stats.

zeljkofilipin added a comment.Sep 2 2015, 5:42 PM

Added more stats. Is something like this chart what you need. Something else? Something completely different?

zeljkofilipin added a comment.Sep 3 2015, 10:06 AM

http://stackoverflow.com/questions/13799652/jenkins-get-build-time-trend-values-using-remote-access-api

zeljkofilipin added a comment.EditedSep 3 2015, 10:10 AM

https://integration.wikimedia.org/ci/job/browsertests-CentralAuth-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/api/json?tree=builds[duration]&pretty=true

Numbers are build durations in ms.

{
  "builds" : [
    {
      "duration" : 151344
    },
    {
      "duration" : 138994
    },
    {
      "duration" : 135699
    },
    {
      "duration" : 122756
    },
    {
      "duration" : 148412
    },
    {
      "duration" : 144184
    },
    {
      "duration" : 155243
    },
    {
      "duration" : 168169
    },
    {
      "duration" : 154493
    },
    {
      "duration" : 161775
    },
    {
      "duration" : 153852
    },
    {
      "duration" : 138690
    },
    {
      "duration" : 153293
    },
    {
      "duration" : 152862
    },
    {
      "duration" : 199134
    },
    {
      "duration" : 217630
    },
    {
      "duration" : 170512
    },
    {
      "duration" : 164296
    },
    {
      "duration" : 141154
    },
    {
      "duration" : 148764
    },
    {
      "duration" : 143064
    },
    {
      "duration" : 147358
    },
    {
      "duration" : 159911
    },
    {
      "duration" : 194167
    },
    {
      "duration" : 181029
    },
    {
      "duration" : 144981
    },
    {
      "duration" : 206367
    },
    {
      "duration" : 153216
    },
    {
      "duration" : 147530
    },
    {
      "duration" : 160792
    },
    {
      "duration" : 212025
    }
  ]
}
hashar added a comment.Sep 3 2015, 2:24 PM

The duration is stored in Raita Elastic Search. So we could build a dashboard there to represent durations.

I have no idea how Raita / ES work but on integration-raita I managed to fetch something using:

curl -XGET 'http://localhost:9200/raita/build/_search?pretty=1&q=number:636'

That yields fields matching build #636 such as:

{
    "_id": "M2eQNEXzQryuHzINL1Ggow",
    "_index": "raita",
    "_score": 1.0,
    "_source": {
        "browser": {
            "name": "firefox",
            "platform": "linux",
            "version": null
        },
        "environment": {
            "name": "beta",
            "url": "http://commons.wikimedia.beta.wmflabs.org/wiki/"
        },
        "number": "732",
        "project": {
            "branch": "origin/master",
            "commit": "c5f8640aa4f026ef45b74498f543958816b70f80",
            "name": "browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce",
            "repo": "https://gerrit.wikimedia.org/r/mediawiki/extensions/UploadWizard"
        },
        "result": {
            "duration": 1490184335796,
            "status": "failed"
        },
        "url": "https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/732/"
    },
    "_type": "build"
}

Not sure what 1490184335796 represent has a duration though. The job took roughly 25 minutes.

zeljkofilipin added a comment.Sep 3 2015, 2:55 PM

I wrote a simple script. The script gets data from Jenkins json api.

Example output (job duration median in minutes, job name):

$ ruby stats.rb 
       2.6 browsertests-CentralAuth-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce
       0.2 browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-linux-android-sauce
       0.2 browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-os_x_10.10-iphone-sauce
       0.6 browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce
       0.9 browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce
       0.9 browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce
       0.2 browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce
       0.6 browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-chrome-sauce
       0.9 browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-firefox-sauce
       0.2 browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-10-sauce
       0.1 browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce
       1.3 browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce
       2.6 browsertests-ContentTranslation-language-screenshot-os_x_10.10-firefox
       8.4 browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce
       1.4 browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce-T92613
       6.5 browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce
       7.8 browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce
      29.9 browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce
      29.8 browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce
      29.1 browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce
      29.6 browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce
       0.5 browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce
      14.1 browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce
       0.8 browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce
       0.9 browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce
       0.8 browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce-T99656
       0.9 browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce
      27.4 browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce
      36.6 browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce
       7.7 browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce
      13.4 browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce
      27.2 browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce
      58.8 browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce
      34.1 browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce
       5.4 browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-8-sauce
      39.1 browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce
      32.8 browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce
      32.9 browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce
       2.9 browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce
       0.7 browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce
       0.9 browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce
       0.8 browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce
      20.0 browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce
      26.3 browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce
      13.3 browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce
      13.3 browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce-T99661
      27.7 browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce
      10.3 browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce
      12.3 browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce
       1.4 browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce
      32.6 browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox
      75.7 browsertests-VisualEditor-production-linux-firefox-sauce
       0.6 browsertests-Wikidata-PerformanceTests-linux-firefox-sauce
       9.8 browsertests-Wikidata-SmokeTests-linux-firefox-sauce
     123.3 browsertests-Wikidata-WikidataTests-linux-chrome-sauce
       1.6 browsertests-Wikidata-WikidataTests-linux-firefox-sauce
       2.6 browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce
       2.5 browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce-T99660
       0.4 browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs

Long term we probably need data from raita.

greg added a comment.Sep 3 2015, 3:58 PM
In T104396#1598398, @zeljkofilipin wrote:

Added more stats. Is something like this chart what you need. Something else? Something completely different?

As said in our 1:1, this is good for now, let's look back at this when we create a "QA Health Dashboard" or similar.

Sidenote: some of the quick ones could be converted to @integration browser tests :)

• dduvall added a comment.Sep 3 2015, 9:19 PM
In T104396#1602304, @hashar wrote:

The duration is stored in Raita Elastic Search. So we could build a dashboard there to represent durations.

See T102536: Display date/time and timing info for Raita builds

Not sure what 1490184335796 represent has a duration though. The job took roughly 25 minutes.

I think Cucumber's instrumentation is done in nanoseconds for some bonkers reason. So (1490184335796 / 1000000000) / 60 = 24.8364055966 minutes which seems about right considering it wouldn't include setup/teardown overhead.

hashar added a comment.Sep 4 2015, 8:10 AM

Awesome @dduvall nanoseconds totally make sense :-} *giggles*. I am subscribing to T102536 right now!

zeljkofilipin removed zeljkofilipin as the assignee of this task.Sep 4 2015, 12:51 PM
zeljkofilipin moved this task from In Progress to Next on the Browser-Tests-Infrastructure board.
zeljkofilipin moved this task from Next to Ruby on the Browser-Tests-Infrastructure board.Sep 30 2015, 1:24 PM
zeljkofilipin moved this task from Ruby to CI on the Browser-Tests-Infrastructure board.Sep 30 2015, 4:08 PM
greg added a project: Release-Engineering-Team (Backlog).May 20 2017, 4:38 PM
zeljkofilipin lowered the priority of this task from Medium to Low.May 29 2017, 10:29 AM
zeljkofilipin added a comment.May 29 2017, 11:08 AM

@greg can this be resolved? Is this something that you still want done?

zeljkofilipin closed this task as Declined.Nov 7 2017, 3:16 PM

Unlikely to ever be resolved because of T139740: Port Selenium tests from Ruby to Node.js.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL