Page MenuHomePhabricator

Migrate Wikibase selenium tests to Quibble+Apache, enable concurrency
Open, Needs TriagePublic

Description

Apache is a multithreaded backend, which will make it possible to run tests concurrently, reducing runtime. Migrating the backend can be done in steps. The main issue we expect to run into are race conditions from test or production code that was not expecting a multithreaded backend, for example assuming a user exists after creating it in a client background thread.

  • Add wmf-quibble-apache-selenium-php72-docker to the "experimental" phase in CI config for the Wikibase repo.
  • Copy "extension-quibble" and modify to use apache jobs.
  • Migrate Wikibase repo to use the apache job as its default CI.
  • Fix any issues preventing Wikibase quibble+apache tests from passing.
  • Enable wdio concurrency (may move to a subtask).

Some of the issues can be reproduced locally using this command:

docker run \
    --volume `pwd`/src:/workspace/src \
    --volume `pwd`/cache:/cache \
    --volume `pwd`/ref:/srv/git:ro \
    --volume `pwd`/log:/workspace/log \
    --tmpfs /workspace/db:size=320M \
    --init --rm \
    -e ZUUL_PROJECT=mediawiki/extensions/Wikibase \
    docker-registry.wikimedia.org/releng/quibble-buster-php72-apache:1.1.0 \
    --run selenium,api-testing

Event Timeline

Just discovered that @Addshore has already migrated Wikibase CI to use the apache job, and it has been running successfully since 45196761454f28.

Just discovered that @Addshore has already migrated Wikibase CI to use the apache job, and it has been running successfully since 45196761454f28.

Correction: the tests are set to non-voting, and occasionally fail:

10:17:50 [0-0] AssertionError [ERR_ASSERTION] in "ImportPreview page.shows dismissible help banner"
10:18:01 AssertionError [ERR_ASSERTION]: the help banner is no longer visible on future visits
10:18:01     at Context.it (/workspace/src/extensions/FileImporter/tests/selenium/specs/helpbanner.js:36:3)
10:18:01 [0-0] FAILED in chrome - /tests/selenium/specs/helpbanner.js
18:55:38 [0-0] Error in "Pages in the Index namespace load the custom editing form in edit mode"
18:56:42 Error: Timeout of 60000ms exceeded. The execution in the test "Pages in the Index namespace load the custom editing form in edit mode" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/workspace/src/extensions/ProofreadPage/tests/selenium/specs/index.js)
18:56:42     at createTimeoutError (/workspace/src/extensions/ProofreadPage/node_modules/mocha/lib/errors.js:498:15)
18:56:42     at Test.Runnable._timeoutError (/workspace/src/extensions/ProofreadPage/node_modules/mocha/lib/runnable.js:431:10)
18:56:42     at Timeout.<anonymous> (/workspace/src/extensions/ProofreadPage/node_modules/mocha/lib/runnable.js:246:24)
18:56:42     at ontimeout (timers.js:436:11)
18:56:42     at tryOnTimeout (timers.js:300:5)
18:56:42     at listOnTimeout (timers.js:263:5)
18:56:42     at Timer.processTimers (timers.js:223:10)
18:56:42 [0-0] 2021-10-04T17:00:49.901Z ERROR @wdio/sync: Error: Protocol error (Runtime.evaluate): Session closed. Most likely the page has been closed.
19:00:49     at CDPSession.send (/workspace/src/extensions/ProofreadPage/node_modules/puppeteer-core/lib/cjs/puppeteer/common/Connection.js:195:35)
19:00:49     at ExecutionContext._evaluateInternal (/workspace/src/extensions/ProofreadPage/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:163:18)
19:00:49     at ExecutionContext.evaluate (/workspace/src/extensions/ProofreadPage/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:107:27)
19:00:49     at DevToolsDriver.checkPendingNavigations (/workspace/src/extensions/ProofreadPage/node_modules/devtools/build/devtoolsdriver.js:177:36)
19:00:49     at saveScreenshot (/workspace/src/extensions/ProofreadPage/node_modules/wdio-mediawiki/index.js:49:10)
19:00:49     at Object.afterTest (/workspace/src/extensions/ProofreadPage/node_modules/wdio-mediawiki/wdio-defaults.conf.js:132:3)
19:00:49 [0-0] Error in "AfterTest Hook"
19:00:49 Protocol error (Runtime.evaluate): Session closed. Most likely the page has been closed.
19:00:49 [0-0] 2021-10-04T17:01:42.700Z ERROR @wdio/local-runner: Failed launching test session: Error: Protocol error (Runtime.evaluate): Session closed. Most likely the page has been closed.
19:01:42     at CDPSession.send (/workspace/src/extensions/ProofreadPage/node_modules/puppeteer-core/lib/cjs/puppeteer/common/Connection.js:195:35)
19:01:42     at ExecutionContext._evaluateInternal (/workspace/src/extensions/ProofreadPage/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:163:18)
19:01:42     at ExecutionContext.evaluate (/workspace/src/extensions/ProofreadPage/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:107:27)
19:01:42     at DevToolsDriver.checkPendingNavigations (/workspace/src/extensions/ProofreadPage/node_modules/devtools/build/devtoolsdriver.js:177:36)
19:01:42     at Runner.endSession (/workspace/src/extensions/ProofreadPage/node_modules/@wdio/runner/build/index.js:326:30)
19:01:42     at Runner.run (/workspace/src/extensions/ProofreadPage/node_modules/@wdio/runner/build/index.js:152:24)
19:01:42 [0-0] FAILED in chrome - /tests/selenium/specs/index.js

More glitches which happen locally and I can see in a few CI failures,

[Chrome 90.0.4430.212 linux #0-5] 1) Termbox: AnonEditWarning "before each" hook for Termbox: AnonEditWarning         
[Chrome 90.0.4430.212 linux #0-5] element (".wb-ui-all-entered-languages-expandable__switch") still not existing after 10000ms                                                                                                              
[Chrome 90.0.4430.212 linux #0-5] Error: element (".wb-ui-all-entered-languages-expandable__switch") still not existing after 10000ms                                                                                                       
[Chrome 90.0.4430.212 linux #0-5]     at TermboxPage.waitForTermboxToLoad (/workspace/src/extensions/Wikibase/view/lib/wikibase-termbox/tests/selenium/pageobjects/Termbox.page.js:106:34)
[Chrome 90.0.4430.212 linux #0-5]     at TermboxPage.openItemPage (/workspace/src/extensions/Wikibase/view/lib/wikibase-termbox/tests/selenium/pageobjects/Termbox.page.js:100:8)                                                           
[Chrome 90.0.4430.212 linux #0-5]     at Context.beforeEach (/workspace/src/extensions/Wikibase/view/lib/wikibase-termbox/tests/selenium/specs/AnonEditWarning.spec.js:14:15)