Page MenuHomePhabricator

Investigate distribution of browser test run time
Closed, DeclinedPublic

Description

I'd like to know how long it takes the majority of our browser tests to run.

I'd also like to know which tests suites take the longest (and how far from median they are) to identify high reward areas to improve.

Event Timeline

greg created this task.Jun 30 2015, 10:37 PM
greg raised the priority of this task from to Needs Triage.
greg updated the task description. (Show Details)
greg added a subscriber: greg.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 30 2015, 10:37 PM

One of the slowdown are the network roundtrips to SauceLabs. In some cases there are thousands of them and that is tracked at T92613

zeljkofilipin triaged this task as Medium priority.
zeljkofilipin set Security to None.
zeljkofilipin moved this task from Inbox to In Progress on the Browser-Tests-Infrastructure board.
zeljkofilipin added a subscriber: zeljkofilipin.

I am speculating we have timing in Raita?

Not sure how to get build times from raita:

http://raita.wmflabs.org/#builds/latest

I have created a simple spreadsheet:

https://docs.google.com/spreadsheets/d/1xrJxTVYPx1uCVKXC5Qrdc1a0LLctxYm_PIFG65yjw2A/edit?usp=sharing

The data is copy/pasted from jenkins, that is why I limited it to 5 jobs for now. If it provides the data you need, I can automate the process.

For now it only has 5 jobs and lists only average time for each job. I will add another dashboard with more stats.

Added more stats. Is something like this chart what you need. Something else? Something completely different?

zeljkofilipin added a comment.EditedSep 3 2015, 10:10 AM

https://integration.wikimedia.org/ci/job/browsertests-CentralAuth-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/api/json?tree=builds[duration]&pretty=true

Numbers are build durations in ms.

{
  "builds" : [
    {
      "duration" : 151344
    },
    {
      "duration" : 138994
    },
    {
      "duration" : 135699
    },
    {
      "duration" : 122756
    },
    {
      "duration" : 148412
    },
    {
      "duration" : 144184
    },
    {
      "duration" : 155243
    },
    {
      "duration" : 168169
    },
    {
      "duration" : 154493
    },
    {
      "duration" : 161775
    },
    {
      "duration" : 153852
    },
    {
      "duration" : 138690
    },
    {
      "duration" : 153293
    },
    {
      "duration" : 152862
    },
    {
      "duration" : 199134
    },
    {
      "duration" : 217630
    },
    {
      "duration" : 170512
    },
    {
      "duration" : 164296
    },
    {
      "duration" : 141154
    },
    {
      "duration" : 148764
    },
    {
      "duration" : 143064
    },
    {
      "duration" : 147358
    },
    {
      "duration" : 159911
    },
    {
      "duration" : 194167
    },
    {
      "duration" : 181029
    },
    {
      "duration" : 144981
    },
    {
      "duration" : 206367
    },
    {
      "duration" : 153216
    },
    {
      "duration" : 147530
    },
    {
      "duration" : 160792
    },
    {
      "duration" : 212025
    }
  ]
}

The duration is stored in Raita Elastic Search. So we could build a dashboard there to represent durations.

I have no idea how Raita / ES work but on integration-raita I managed to fetch something using:

curl -XGET 'http://localhost:9200/raita/build/_search?pretty=1&q=number:636'

That yields fields matching build #636 such as:

{
    "_id": "M2eQNEXzQryuHzINL1Ggow",
    "_index": "raita",
    "_score": 1.0,
    "_source": {
        "browser": {
            "name": "firefox",
            "platform": "linux",
            "version": null
        },
        "environment": {
            "name": "beta",
            "url": "http://commons.wikimedia.beta.wmflabs.org/wiki/"
        },
        "number": "732",
        "project": {
            "branch": "origin/master",
            "commit": "c5f8640aa4f026ef45b74498f543958816b70f80",
            "name": "browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce",
            "repo": "https://gerrit.wikimedia.org/r/mediawiki/extensions/UploadWizard"
        },
        "result": {
            "duration": 1490184335796,
            "status": "failed"
        },
        "url": "https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/732/"
    },
    "_type": "build"
}

Not sure what 1490184335796 represent has a duration though. The job took roughly 25 minutes.

I wrote a simple script. The script gets data from Jenkins json api.

Example output (job duration median in minutes, job name):

$ ruby stats.rb 
       2.6 browsertests-CentralAuth-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce
       0.2 browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-linux-android-sauce
       0.2 browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-os_x_10.10-iphone-sauce
       0.6 browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce
       0.9 browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce
       0.9 browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce
       0.2 browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce
       0.6 browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-chrome-sauce
       0.9 browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-firefox-sauce
       0.2 browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-10-sauce
       0.1 browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce
       1.3 browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce
       2.6 browsertests-ContentTranslation-language-screenshot-os_x_10.10-firefox
       8.4 browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce
       1.4 browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce-T92613
       6.5 browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce
       7.8 browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce
      29.9 browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce
      29.8 browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce
      29.1 browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce
      29.6 browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce
       0.5 browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce
      14.1 browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce
       0.8 browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce
       0.9 browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce
       0.8 browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce-T99656
       0.9 browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce
      27.4 browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce
      36.6 browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce
       7.7 browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce
      13.4 browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce
      27.2 browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce
      58.8 browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce
      34.1 browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce
       5.4 browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-8-sauce
      39.1 browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce
      32.8 browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce
      32.9 browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce
       2.9 browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce
       0.7 browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce
       0.9 browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce
       0.8 browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce
      20.0 browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce
      26.3 browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce
      13.3 browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce
      13.3 browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce-T99661
      27.7 browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce
      10.3 browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce
      12.3 browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce
       1.4 browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce
      32.6 browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox
      75.7 browsertests-VisualEditor-production-linux-firefox-sauce
       0.6 browsertests-Wikidata-PerformanceTests-linux-firefox-sauce
       9.8 browsertests-Wikidata-SmokeTests-linux-firefox-sauce
     123.3 browsertests-Wikidata-WikidataTests-linux-chrome-sauce
       1.6 browsertests-Wikidata-WikidataTests-linux-firefox-sauce
       2.6 browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce
       2.5 browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce-T99660
       0.4 browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs

Long term we probably need data from raita.

greg added a comment.Sep 3 2015, 3:58 PM

Added more stats. Is something like this chart what you need. Something else? Something completely different?

As said in our 1:1, this is good for now, let's look back at this when we create a "QA Health Dashboard" or similar.

Sidenote: some of the quick ones could be converted to @integration browser tests :)

The duration is stored in Raita Elastic Search. So we could build a dashboard there to represent durations.

See T102536: Display date/time and timing info for Raita builds

Not sure what 1490184335796 represent has a duration though. The job took roughly 25 minutes.

I think Cucumber's instrumentation is done in nanoseconds for some bonkers reason. So (1490184335796 / 1000000000) / 60 = 24.8364055966 minutes which seems about right considering it wouldn't include setup/teardown overhead.

Awesome @dduvall nanoseconds totally make sense :-} *giggles*. I am subscribing to T102536 right now!

zeljkofilipin removed zeljkofilipin as the assignee of this task.Sep 4 2015, 12:51 PM
zeljkofilipin moved this task from In Progress to Next on the Browser-Tests-Infrastructure board.
zeljkofilipin lowered the priority of this task from Medium to Low.May 29 2017, 10:29 AM

@greg can this be resolved? Is this something that you still want done?

zeljkofilipin closed this task as Declined.Nov 7 2017, 3:16 PM

Unlikely to ever be resolved because of T139740: Port Selenium tests from Ruby to Node.js.