Page MenuHomePhabricator
Create Task
Maniphest T409392

Measure CI overhead using xvfb/ffmpeg vs --headlesss
Open, Needs TriagePublic5 Estimated Story PointsSpike
Actions

Description

In WebdriverIO documentation headless is recommended (meaning using the browsers own headless switch). The drawback is that you can't get a video of the run. We uses XVFB as screen and use ffmpeg to record a video.

Since some time ago, Chromium --headless uses the same code path for both headless and non headless (before it used its own binary for headless). That is good since it should then run through the same code paths. You can also configure Chrome's tracelog to continues take screenshots by adding the trace category disabled-by-default-devtools.screenshot. The user can then drag and drop the trace log into Chrome and see the screenshots.

Should we follow best practices and also use --headless? Let try to measure what kind of overhead ffmpeg/xvfb adds to the test runs so we can take a decision.

Today we have these graphs for CI.

Acceptance Criteria:

  • Measure and document the run time (will --headless be faster?).
  • For both tests run with at least 1 and 10 simultaneously running tests.
  • Create follow up task where the decision can be documented and we can discuss if we should use headless or not

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
selenium: Test to see the xvfb/ffmpeg overheadmediawiki/coremaster+1 -1
selenium: Test many ffmpegs at the same timemediawiki/coremaster+1 -1
selenium: Test removing ffmpeg overheadmediawiki/coremaster+7 -7
Customize query in gerrit

Related Objects
Search...

Event Timeline

Peter created this task.Nov 6 2025, 7:43 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 6 2025, 7:43 AM
Peter moved this task from Backlog to Ready to Refine on the Test-Platform board.Nov 6 2025, 7:45 AM
Peter mentioned this in T409150: Setup/teardown xvfb in NodeJS to remove dependency of Docker start script.Nov 6 2025, 7:47 AM
Peter updated the task description. (Show Details)Nov 11 2025, 2:16 PM
Peter added a project: Browser Test Platform.Tue, Nov 18, 9:42 AM
Peter moved this task from Backlog to Prioritized on the Browser Test Platform board.Tue, Nov 18, 10:05 AM
Peter updated the task description. (Show Details)Tue, Nov 18, 4:44 PM
gerritbot added a comment.Wed, Nov 19, 1:43 PM

Change #1207155 had a related patch set uploaded (by Phedenskog; author: Phedenskog):

[mediawiki/core@master] selenium: Test removing ffmpeg overhead

https://gerrit.wikimedia.org/r/1207155

gerritbot added a project: Patch-For-Review.Wed, Nov 19, 1:43 PM
gerritbot added a comment.Wed, Nov 19, 4:03 PM

Change #1207196 had a related patch set uploaded (by Phedenskog; author: Phedenskog):

[mediawiki/core@master] selenium: Test many ffmpegs at the same time

https://gerrit.wikimedia.org/r/1207196

gerritbot added a comment.Thu, Nov 20, 4:26 AM

Change #1207449 had a related patch set uploaded (by Phedenskog; author: Phedenskog):

[mediawiki/core@master] selenium: Test to see the xvfb/ffmpeg overhead

https://gerrit.wikimedia.org/r/1207449

zeljkofilipin added a project: Spike.Thu, Nov 20, 2:52 PM
Restricted Application changed the subtype of this task from "Task" to "Spike". · View Herald TranscriptThu, Nov 20, 2:52 PM
zeljkofilipin updated the task description. (Show Details)Thu, Nov 20, 2:57 PM
Krinkle awarded a token.Mon, Nov 24, 8:43 PM
Krinkle subscribed.
Peter set the point value for this task to 5.Mon, Dec 1, 5:00 PM
Peter moved this task from Ready to Refine to Refined (Discussed and Estimated) Backlog on the Test-Platform board.
Peter moved this task from Refined (Discussed and Estimated) Backlog to Bashkent Two (Current Sprint) on the Test-Platform board.Mon, Dec 1, 5:11 PM
Peter edited projects, added Test-Platform (Bashkent Two (Current Sprint)); removed Test-Platform.
Peter added a comment.Mon, Dec 1, 5:24 PM

T410607 needs to be merged for this to make it easier to do this task.

Peter claimed this task.Mon, Dec 1, 6:32 PM
Peter updated the task description. (Show Details)Tue, Dec 2, 7:14 AM
Peter added a comment.Tue, Dec 2, 8:00 AM

I'm gonna do like this:
I'll prepare one of the bare metal servers we have for synthetic performance tests. Install mediawiki and quickstart and perf. Then I'll just make sure everything works on that machine and then when we merged T410607 we can stop the performance tests for a short while, run some tests and collect data using perf and see if I can evaluate the numbers we get.,

zeljkofilipin moved this task from Prioritized to Doing on the Browser Test Platform board.Tue, Dec 2, 1:29 PM
Peter moved this task from Backlog to In Progress on the Test-Platform (Bashkent Two (Current Sprint)) board.Tue, Dec 2, 2:01 PM
Peter added a comment.Wed, Dec 3, 6:42 AM

Dumping the log of the errors I get running the tests on the test machine:

[0-0] RUNNING in chrome - file:///tests/selenium/docs/Create_a_simple_test/specs/specialpages.js
[0-0] PASSED in chrome - file:///tests/selenium/docs/Create_a_simple_test/specs/specialpages.js
[0-1] RUNNING in chrome - file:///tests/selenium/docs/Page_object_pattern/specs/login.js
[0-1] Error in "User.should be able to log in without page object"
Error: expect(received).toBe(expected) // Object.is equality

Expected: "User-0.17611062292731283-Iñtërnâtiônàlizætiøn"
Received: null
    at Context.<anonymous> (file:///test/mediawiki/tests/selenium/docs/Page_object_pattern/specs/login.js:36:28)
[0-1] RETRYING in chrome - file:///tests/selenium/docs/Page_object_pattern/specs/login.js
[0-1] RUNNING in chrome - file:///tests/selenium/docs/Page_object_pattern/specs/login.js
[0-1] Error in "User.should be able to log in without page object"
Error: expect(received).toBe(expected) // Object.is equality

Expected: "User-0.1472929106489793-Iñtërnâtiônàlizætiøn"
Received: null
    at Context.<anonymous> (file:///test/mediawiki/tests/selenium/docs/Page_object_pattern/specs/login.js:36:28)
[0-1] FAILED in chrome - file:///tests/selenium/docs/Page_object_pattern/specs/login.js (1 retries)
[0-2] RUNNING in chrome - file:///tests/selenium/docs/Stack/specs/expect.js
[0-2] PASSED in chrome - file:///tests/selenium/docs/Stack/specs/expect.js
[0-3] RUNNING in chrome - file:///tests/selenium/docs/Stack/specs/mocha.js
[0-3] Edit link visible
[0-3] PASSED in chrome - file:///tests/selenium/docs/Stack/specs/mocha.js
[0-4] RUNNING in chrome - file:///tests/selenium/docs/Stack/specs/pageobject.js
[0-4] PASSED in chrome - file:///tests/selenium/docs/Stack/specs/pageobject.js
[0-5] RUNNING in chrome - file:///tests/selenium/docs/Use_MediaWiki_API/specs/api.js
[0-5] PASSED in chrome - file:///tests/selenium/docs/Use_MediaWiki_API/specs/api.js
[0-6] RUNNING in chrome - file:///tests/selenium/specs/page.js
[0-6] Error in "Page.should be previewable @daily"
Error: Cannot submit login form
    at async LoginPage.login (file:///test/mediawiki/tests/selenium/wdio-mediawiki/LoginPage.js:36:3)
    at async LoginPage.loginAdmin (file:///test/mediawiki/tests/selenium/wdio-mediawiki/LoginPage.js:50:3)
    at async Context.<anonymous> (file:///test/mediawiki/tests/selenium/specs/page.js:33:3)
[0-6] Error in "Page.should be re-creatable"
Error: Can't call setValue on element with selector "#wpTextbox1" because element wasn't found
    at async EditPage.edit (file:///test/mediawiki/tests/selenium/pageobjects/edit.page.js:57:3)
    at async Context.<anonymous> (file:///test/mediawiki/tests/selenium/specs/page.js:62:3)
[0-6] Error in "Page.should be deletable"
Error: Can't call setValue on element with selector "#wpReason input" because element wasn't found
    at async DeletePage.delete (file:///test/mediawiki/tests/selenium/pageobjects/delete.page.js:26:3)
    at async Context.<anonymous> (file:///test/mediawiki/tests/selenium/specs/page.js:98:3)
[0-6] Error in "Page.should be restorable"
Error: Cannot submit login form
    at async LoginPage.login (file:///test/mediawiki/tests/selenium/wdio-mediawiki/LoginPage.js:36:3)
    at async LoginPage.loginAdmin (file:///test/mediawiki/tests/selenium/wdio-mediawiki/LoginPage.js:50:3)
    at async Context.<anonymous> (file:///test/mediawiki/tests/selenium/specs/page.js:110:3)
[0-6] Error in "Page.should be protectable"
Error: Can't call setValue on element with selector "#mwProtect-reason input" because element wasn't found
    at async ProtectPage.protect (file:///test/mediawiki/tests/selenium/pageobjects/protect.page.js:30:3)
    at async Context.<anonymous> (file:///test/mediawiki/tests/selenium/specs/page.js:126:3)
[0-6] RETRYING in chrome - file:///tests/selenium/specs/page.js
[0-6] RUNNING in chrome - file:///tests/selenium/specs/page.js
[0-6] Error in "Page.should be previewable @daily"
Error: Cannot submit login form
    at async LoginPage.login (file:///test/mediawiki/tests/selenium/wdio-mediawiki/LoginPage.js:36:3)
    at async LoginPage.loginAdmin (file:///test/mediawiki/tests/selenium/wdio-mediawiki/LoginPage.js:50:3)
    at async Context.<anonymous> (file:///test/mediawiki/tests/selenium/specs/page.js:33:3)
[0-6] Error in "Page.should be creatable"
Error: Cannot submit login form
    at async LoginPage.login (file:///test/mediawiki/tests/selenium/wdio-mediawiki/LoginPage.js:36:3)
    at async LoginPage.loginAdmin (file:///test/mediawiki/tests/selenium/wdio-mediawiki/LoginPage.js:50:3)
    at async Context.<anonymous> (file:///test/mediawiki/tests/selenium/specs/page.js:45:3)
[0-6] Error in "Page.should be re-creatable"
Error: Can't call setValue on element with selector "#wpTextbox1" because element wasn't found
    at async EditPage.edit (file:///test/mediawiki/tests/selenium/pageobjects/edit.page.js:57:3)
    at async Context.<anonymous> (file:///test/mediawiki/tests/selenium/specs/page.js:62:3)
[0-6] Error in "Page.should be restorable"
Error: Cannot submit login form
    at async LoginPage.login (file:///test/mediawiki/tests/selenium/wdio-mediawiki/LoginPage.js:36:3)
    at async LoginPage.loginAdmin (file:///test/mediawiki/tests/selenium/wdio-mediawiki/LoginPage.js:50:3)
    at async Context.<anonymous> (file:///test/mediawiki/tests/selenium/specs/page.js:110:3)
[0-6] 2025-12-03T06:37:48.466Z ERROR webdriver: WebDriverError: element not interactable
[0-6]   (Session info: chrome=143.0.7499.40) when running "element/f.90C4A4E3465E4FF6BA5A0DDDDB81B42D.d.7F2E55B43F49EE943AACA939707640AA.e.66/click" with method "POST"
[0-6] Error in "Page.should be protectable"
Error: Cannot submit login form
    at async LoginPage.login (file:///test/mediawiki/tests/selenium/wdio-mediawiki/LoginPage.js:36:3)
    at async LoginPage.loginAdmin (file:///test/mediawiki/tests/selenium/wdio-mediawiki/LoginPage.js:50:3)
    at async Context.<anonymous> (file:///test/mediawiki/tests/selenium/specs/page.js:124:3)
Peter added a comment.Wed, Dec 3, 1:15 PM

Ok down to two type of errors now:

[0-1] Error in "User.should be able to log in without page object"
Error: expect(received).toBe(expected) // Object.is equality

Expected: "User-0.24708116128821733-Iñtërnâtiônàlizætiøn"
Received: "User-0.5685653779285673-Iñtërnâtiônàlizætiøn"

And

[0-6] Error in "Page.should be previewable @daily"
Error: Can't call setValue on element with selector "#wpTextbox1" because element wasn't found
Peter added a comment.EditedWed, Dec 3, 1:18 PM

Ok, the problem was that I used 127.0.0.1 and not localhost - I wonder though why our tests care? Let's check later and move on with the measuring.

There's one error left:

[chrome 143.0.7499.40 linux #0-8]
[chrome 143.0.7499.40 linux #0-8] 1) User temporary user should be able to create account
[chrome 143.0.7499.40 linux #0-8] expect(received).toBe(expected) // Object.is equality

Expected: "User-0.7420192976655529-Iñtërnâtiônàlizætiøn"
Received: "~2025-17"

And sometimes:

[0-6] 2025-12-03T13:35:03.495Z ERROR webdriver: WebDriverError: element click intercepted: Element <span class="oo-ui-labelElement-label" id="ooui-1" role="textbox" aria-readonly="true">...</span> is not clickable at point (361, 9). Other element would receive the click: <div class="vector-sticky-header-start">...</div>
[0-6]   (Session info: chrome=143.0.7499.40) when running "element/f.F0F70A2D71995E454DCDED6B8F168142.d.3990B08834F33D0865C8EBB73EBD6ED5.e.1152/click" with method "POST"
gerritbot added a comment.Wed, Dec 3, 2:46 PM

Change #1207155 abandoned by Phedenskog:

[mediawiki/core@master] selenium: Test removing ffmpeg overhead

Reason:

This was only for testing

https://gerrit.wikimedia.org/r/1207155

gerritbot added a comment.Wed, Dec 3, 2:46 PM

Change #1207196 abandoned by Phedenskog:

[mediawiki/core@master] selenium: Test many ffmpegs at the same time

Reason:

This was only for testing

https://gerrit.wikimedia.org/r/1207196

gerritbot added a comment.Wed, Dec 3, 2:46 PM

Change #1207449 abandoned by Phedenskog:

[mediawiki/core@master] selenium: Test to see the xvfb/ffmpeg overhead

Reason:

This was only for testing

https://gerrit.wikimedia.org/r/1207449

Maintenance_bot removed a project: Patch-For-Review.Wed, Dec 3, 3:34 PM
Peter added a comment.EditedThu, Dec 4, 12:02 AM

Skipping the following tests make the rest run without any errors:

it.skip( 'should be able to log in without page object',
it.skip( 'temporary user should not see signup form fields relevant to named users'
it.skip( 'temporary user should be able to create account', 
it.skip( 'should be protectable'
Peter moved this task from In Progress to Finishing Touches on the Test-Platform (Bashkent Two (Current Sprint)) board.Thu, Dec 4, 12:52 PM
Peter added a comment.Thu, Dec 4, 1:24 PM

I've been trying to test if we can reduce the speed of webdriverIO tests on CI by running tests truly headless and skipping the video. Let's look at the result. But first lets talk about how the test was conducted.

How the tests was done

I've been using one of the bare metal server we use for synthetic performance testing. I disabled the tests on the machine so nothing else than the OS was running. The server has 8 cores, running at max speed 4ghz using the performance governor. On the server I installed perf and use it to calculate CPU time spent and wall clock time (=how much CPU is used and how fast do the test run). I installed mediawiki-quickstart to have a local version of Mediawiki to test against. And then mediawiki, to run our core tests directly on the machine. There was a couple of tests that was flakey so I disabled those tests. It's important that we run the exact same tests and iterations when we compare.

I started with ten iteration but ended up running 50 iterations and looked like this: perf stat -r 50 -- npm run selenium-test
And then I modified the configuration between the tests.

With the tests I measured using XVFB /FFmpeg vs headless and different maxInstances settings (how many tests runs at the same time).

XVFB/FFMpeg vs --headless 1 maxInstance

Today in core we run with maxInstance test to 1 since with the upgrade to wdio 9 we had problem with multiple instances. However we want to run more at the same time.

XVFB/FFmpeg 1 instance

Performance counter stats for 'npm run selenium-test' (50 runs):

         59,398.42 msec task-clock:u              #    1.231 CPUs utilized            ( +-  0.10% )
                 0      context-switches:u        #    0.000 /sec
                 0      cpu-migrations:u          #    0.000 /sec
         1,855,105      page-faults:u             #   31.135 K/sec                    ( +-  0.31% )
   160,981,271,442      cycles:u                  #    2.702 GHz                      ( +-  0.07% )
   176,175,911,664      instructions:u            #    1.09  insn per cycle           ( +-  0.08% )
    32,762,818,402      branches:u                #  549.870 M/sec                    ( +-  0.07% )
       994,288,240      branch-misses:u           #    3.03% of all branches          ( +-  0.07% )

            48.267 +- 0.158 seconds time elapsed  ( +-  0.33% )

--headless 1 instance

Performance counter stats for 'npm run selenium-test' (50 runs):

      53,028.66 msec task-clock:u              #    1.172 CPUs utilized            ( +-  0.06% )
              0      context-switches:u        #    0.000 /sec
              0      cpu-migrations:u          #    0.000 /sec
      1,589,355      page-faults:u             #   29.858 K/sec                    ( +-  0.05% )
145,707,526,357      cycles:u                  #    2.737 GHz                      ( +-  0.07% )
154,862,977,084      instructions:u            #    1.06  insn per cycle           ( +-  0.06% )
 30,445,235,098      branches:u                #  571.942 M/sec                    ( +-  0.06% )
    942,474,642      branch-misses:u           #    3.08% of all branches          ( +-  0.06% )

         45.233 +- 0.213 seconds time elapsed  ( +-  0.47% )

Summary

Running one instance at a time the difference is:

  • Headless is ~6% faster (48.3 → 45.2 s)
  • Headless uses ~11% less CPU time

XVFB/FFMpeg vs --headless 4 maxInstance

Before we upgraded to wdio 9 we used 4 maxInstances. Lets see what that the difference would be

XVFB/FFmpeg 4 instances

Performance counter stats for 'npm run selenium-test' (50 runs):

         70,805.09 msec task-clock:u              #    2.836 CPUs utilized            ( +-  0.07% )
                 0      context-switches:u        #    0.000 /sec
                 0      cpu-migrations:u          #    0.000 /sec
         1,863,666      page-faults:u             #   26.431 K/sec                    ( +-  0.10% )
   199,697,640,484      cycles:u                  #    2.832 GHz                      ( +-  0.07% )
   180,696,157,493      instructions:u            #    0.91  insn per cycle           ( +-  0.07% )
    33,047,194,830      branches:u                #  468.693 M/sec                    ( +-  0.06% )
     1,100,075,988      branch-misses:u           #    3.34% of all branches          ( +-  0.08% )

            24.965 +- 0.486 seconds time elapsed  ( +-  1.95% )

--headless 4 instances

Performance counter stats for 'npm run selenium-test' (50 runs):

         62,360.63 msec task-clock:u              #    2.733 CPUs utilized            ( +-  0.07% )
                 0      context-switches:u        #    0.000 /sec
                 0      cpu-migrations:u          #    0.000 /sec
         1,582,559      page-faults:u             #   25.239 K/sec                    ( +-  0.06% )
   177,877,377,628      cycles:u                  #    2.837 GHz                      ( +-  0.08% )
   154,531,564,275      instructions:u            #    0.86  insn per cycle           ( +-  0.06% )
    30,269,405,014      branches:u                #  482.740 M/sec                    ( +-  0.06% )
     1,035,899,837      branch-misses:u           #    3.41% of all branches          ( +-  0.08% )

           22.8169 +- 0.0886 seconds time elapsed  ( +-  0.39% )

Summary

  • Headless is ~8.6% faster
  • Headless Uses ~12% less CPU time
Peter added a comment.Thu, Dec 4, 1:26 PM

That was the raw data. I'm gonna update the epic with all the numbers we have (I also measured other changes we done).

So how we run tests today with ffmpeg and Xvfb with maxInstance set to 1 and change that to use --headless and maxInstance 4, the core tests will be ~2.1× faster and the CPU cost for that will be 5% more CPU time.

Peter mentioned this in T408361: Reduce CI overhead for webdriver.io tests.Thu, Dec 4, 2:05 PM

I added a summary section to T408361 where we can see difference of different changes.

Peter mentioned this in T411784: Switch to use --headless as default in CI.Thu, Dec 4, 2:09 PM

Created follow up T411784

Peter moved this task from Finishing Touches to Done on the Test-Platform (Bashkent Two (Current Sprint)) board.Thu, Dec 4, 2:13 PM
Peter mentioned this in T411931: Fix flakey tests found when measuring xvfb/ffmpeg overhead.Sat, Dec 6, 1:14 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits