Running Chromium and FFmpeg in CI is a big source of runtime and potential flakiness. This epic aims to minimize the CPU/memory footprint and artifact cost of our browser and recording stack without sacrificing debuggability. The outcome: faster builds, lower ci cost/time, fewer moving parts, and a setup that is predictable and portable across Jenkins agents and other upcoming CI setup.

This epic is browser tool independent where reasonable: while our primary target is WebdriverIO on Chromium with FFmpeg, the patterns should translate to other setups too with modest tuning for FFmpeg and other tools that use Chromium (Cypress for now).

Goals:

• Reduce CI time attributable to test runtime, and video/screenshot handling.
• Make video capture smarter (maybe conditional recording, lower bitrate/size by default).
• Evaluate if headless would (--headless) could be a thing instead of using XVFB for some tests (faster, lower overhead etc)
• If possible: Instrument and expose metrics (browser launch time, test CPU-seconds) into our upcoming metrics stack.
• If we make changes, document them. Document our best practices for a good setup.
• It is not a goal to rewrite the current stack, the goal is to tune what we have.
• Reduce the longest CI job for core

Acceptance Criteria

• FFmpeg follow best practice for fast recording and low overhead
• --headless vs xvfb has been evaluated
• Chrome/Chromium setup has beed iterated for lowest CPU/memory footprint and best practices for browser testing

Summary (so far)

There's been some work, some already merged in core. Lets look at the improvements that has been made and the ones that has been measured. We have measured the difference in a lab environment (running tests on a bare metal server) using perf and comparing different configurations:

• T403357 we configured Chromium to follow best practices. That change made our core tests run 4.8% faster and using 2.3% less CPU time in our lab.
• T408328 we focused on configure FFmpeg to follow best practices (e.g the settings used in Browsertime). That change makes our core tests run 12-13% faster and use 26% less of CPU time.
• T408325 we want to change the default window size to be a little bit smaller but follow more moden screen sizes. When we record videos and move to 1280×720 that would make the tests ~6.8% faster and use something like 5.4% less CPU time. Using --headless there's almost no difference.
• T409392 explores running tests with true --headless and skip recording a video (recommended by webdriver.io). We can see that using headless would be something like 8% faster and use 12% less CPU time. However we could also see that if we skip the video and can increase our maxInstances to 4 (running four tests at the same time) the test time will be 2 times faster with a 5% increase in CPU time.