Page MenuHomePhabricator

Make MediaWiki Wdio tests less slow (Sept 2019)
Open, Needs TriagePublic

Description

Task to track a couple of things I want to try out (and revert/abandon if not compatible and faster):

Test runner:

  • Upgrade from Wdio v4 to v5. T213268
  • Use native async-await instead of the wdio-sync/Fibers hack.
  • Let Wdio use the devtools protocol instead of the webdriver protocol to connect with browsers locally and in CI. This is a new feature in Wdio 5 and should be significantly faster for test execution. It also removes the need for the chromedriver executable.

Test configuration:

  • Determine what overhead video recording is having on test execution.
  • If more than a little bit, consider disabling by default and opt-ing in via a "check <something>" command in Gerrit instead (which Quibble could detect via an environment variable like ZUUL_PIPELINE to decide to run something like npm run selenium-video, which we'd create as variant to npm run selenium).

Test execution:

  • Figure out why the API requests we run with MWBot from Wdio tests are slow (5 seconds is not unusual). This makes no sense to me because MediaWiki in CI has been hyper-optimised, as demonstrated by Fresnel and QUnit tasks running MediaWiki with super low latencies (almost everything in pure RAM, full MW page loads in Fresnel with Chrome take under 150ms, incl load.php etc.).
  • Talked with @kostajh and @Reedy about this, they suggested that maybe 1) the login session keeps getting lost between test cases and 2) login might be very slow because of too strong password crypto requirements by default which are intentionally CPU intensive.

Event Timeline

Krinkle created this task.Sep 27 2019, 3:07 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 27 2019, 3:07 AM

Change 539441 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] [WIP] build: Upgrade from webdriver v4 to v5

https://gerrit.wikimedia.org/r/539441

hashar added a subscriber: hashar.

Change 539691 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] selenium: Remove "RunJobs" wait from specialrecentchanges test

https://gerrit.wikimedia.org/r/539691

Change 539865 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] selenium: Use async-await for wdio-mediawiki/Api internals

https://gerrit.wikimedia.org/r/539865

Change 539866 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] selenium: Fix inefficient use of MWBot in specs/page.js

https://gerrit.wikimedia.org/r/539866

Change 539441 merged by jenkins-bot:
[mediawiki/core@master] selenium: Upgrade from webdriver v4 to v5

https://gerrit.wikimedia.org/r/539441

Change 539691 merged by jenkins-bot:
[mediawiki/core@master] selenium: Remove "RunJobs" wait from specialrecentchanges test

https://gerrit.wikimedia.org/r/539691

Change 539931 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[mediawiki/core@master] [DNM] package.json: Drop @wdio/sync

https://gerrit.wikimedia.org/r/539931

Change 539865 merged by jenkins-bot:
[mediawiki/core@master] selenium: Use async-await for wdio-mediawiki/Api internals

https://gerrit.wikimedia.org/r/539865

Change 539866 merged by jenkins-bot:
[mediawiki/core@master] selenium: Fix inefficient use of MWBot in specs/page.js

https://gerrit.wikimedia.org/r/539866

Change 539935 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/core@master] Selenium: Remove browser.setLocalStorage call

https://gerrit.wikimedia.org/r/539935

Change 539942 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/core@master] Selenium: Hide VE welcome dialog

https://gerrit.wikimedia.org/r/539942

Change 539935 abandoned by Kosta Harlan:
Selenium: Remove browser.setLocalStorage call

Reason:
in favor of https://gerrit.wikimedia.org/r/c/mediawiki/core/ /539942

https://gerrit.wikimedia.org/r/539935

Change 539942 merged by jenkins-bot:
[mediawiki/core@master] Selenium: Hide VE welcome and beta dialogs

https://gerrit.wikimedia.org/r/539942

Change 539965 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/core@master] WDIO: Use pageLoadStrategy=eager with Chromedriver 77

https://gerrit.wikimedia.org/r/539965

Change 540004 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] selenium: Fix more inefficient MWBot use and simplify wdio-mediawiki Api

https://gerrit.wikimedia.org/r/540004

Michael added a subscriber: Michael.Oct 1 2019, 9:27 AM

Change 539965 abandoned by Kosta Harlan:
WDIO: Use pageLoadStrategy=eager with Chromedriver 77

Reason:
This needs to wait for Chrome/Chromium 77 in Debian

https://gerrit.wikimedia.org/r/539965

Have folks considered using child_process.exec() to replace usages of bot.edit()? Seems like that would be considerably faster and accomplish the same goal of generating content for the tests to make assertions against.

Krinkle updated the task description. (Show Details)Oct 3 2019, 4:52 AM
Krinkle updated the task description. (Show Details)Oct 3 2019, 5:17 PM

Change 540004 merged by jenkins-bot:
[mediawiki/core@master] selenium: Fix more inefficient MWBot use and simplify wdio-mediawiki Api

https://gerrit.wikimedia.org/r/540004

Krinkle updated the task description. (Show Details)Oct 6 2019, 3:40 AM

Have folks considered using child_process.exec() to replace usages of bot.edit()? Seems like that would be considerably faster and accomplish the same goal of generating content for the tests to make assertions against.

@Krinkle I meant to write, "using child_process.exec() with php maintenance/edit.php to replace usages of bot.edit()".

Some preliminary tests locally didn't show a performance improvement so I abandoned exploring this.