Page MenuHomePhabricator

Run api-testing tests in parallel
Open, Needs TriagePublic

Description

See also T226869: Run browser tests in parallel

In theory it is simple as adding --parallel to the api-testing command definition in package.json. In practice, some of the setup code will need adjustment.

Event Timeline

Change 751961 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] api-testing: Run tests in parallel

https://gerrit.wikimedia.org/r/751961

Change 752141 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] DevelopmentSettings: Lift PasswordAttemptThrottle limits

https://gerrit.wikimedia.org/r/752141

Change 752148 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] api-testing: Combine two dependent tests into one

https://gerrit.wikimedia.org/r/752148

Change 752148 merged by jenkins-bot:

[mediawiki/core@master] api-testing: Combine two dependent tests into one

https://gerrit.wikimedia.org/r/752148

Change 752357 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/tools/api-testing@master] Run mocha tests in parallel

https://gerrit.wikimedia.org/r/752357

Change 752357 merged by jenkins-bot:

[mediawiki/tools/api-testing@master] Run mocha tests in parallel

https://gerrit.wikimedia.org/r/752357

Change 752994 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] api-testing: Run tests in parallel

https://gerrit.wikimedia.org/r/752994

Change 751961 abandoned by Kosta Harlan:

[mediawiki/core@master] api-testing: Run tests in parallel

Reason:

see I31966e96a996d30fb31f401abecff47a893b8f0c

https://gerrit.wikimedia.org/r/751961

Running the core api-testing tests in parallel works (see https://gerrit.wikimedia.org/r/c/mediawiki/core/+/752994), however due to T199393: Reproducible deadlock in User::addToDatabase() when api.php?action=createaccount is called simultaneously by several users, mw-error.log is not empty and so the assert-no-mediawiki-errors macro fails.

The code in api-testing is set to ignore these exceptions and retry the account creation process. But those errors are written to the logs anyway, see https://integration.wikimedia.org/ci/job/mediawiki-quibble-apitests-vendor-php72-docker/25346/console. Some options:

  1. disable this macro for any api-testing job (or just the core one)
  2. modify the macro to not through an error if the only exception is the "Deadlock found when trying to get lock". That's tricky, because the errors are written in multiple lines with stacktraces, but I guess not impossible to do with a scripting language. You'd need some code to find the start and end of the stacktrace containing "Deadlock found when trying to get lock", remove those lines from the log file, and then assert that the log file is empty at the end.
  3. ??

@hashar @Jdforrester-WMF any ideas?

The proper fix for this would be to fix the back-end to not instantly throw an error on deadlock but instead pause a little (rather than retrying at the browser test level)?

I'd be against dropping or modifying the clean-error-logs macro, as we put so much effort into cleaning up the code to get it to pass the first two times. Not running the macro for the browser tests feels very risky for missing things.

Change 753423 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] api-testing: Run tests in parallel

https://gerrit.wikimedia.org/r/753423

Change 752141 merged by jenkins-bot:

[mediawiki/core@master] DevelopmentSettings: Lift PasswordAttemptThrottle limits

https://gerrit.wikimedia.org/r/752141

Change 752994 merged by jenkins-bot:

[mediawiki/core@master] api-testing: Update to 1.5.0

https://gerrit.wikimedia.org/r/752994

Change 756967 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] DevelopmentSettings: Move wgPasswordAttemptThrottle to correct section

https://gerrit.wikimedia.org/r/756967

Change 756967 merged by jenkins-bot:

[mediawiki/core@master] DevelopmentSettings: Move wgPasswordAttemptThrottle to correct section

https://gerrit.wikimedia.org/r/756967

Change 771678 had a related patch set uploaded (by Krinkle; author: Aaron Schulz):

[mediawiki/core@master] phpunit: Fix slow testBotPasswordThrottled by lowering limits

https://gerrit.wikimedia.org/r/771678

Change 771678 merged by jenkins-bot:

[mediawiki/core@master] phpunit: Fix slow testBotPasswordThrottled by lowering limits

https://gerrit.wikimedia.org/r/771678