Page MenuHomePhabricator

Karma tests using Firefox in VisualEditor/VisualEditor repo are failing
Closed, ResolvedPublic

Description

Karma tests using Firefox in VisualEditor/VisualEditor repo are failing. The same tests in Chrome finish successfully.

Example: https://gerrit.wikimedia.org/r/c/VisualEditor/VisualEditor/+/619033

Log: https://integration.wikimedia.org/ci/job/visualeditor-node10-browser-docker/1597/console

22:21:47 Running "karma:firefox" (karma) task
22:21:47 07 08 2020 20:21:47.251:WARN [filelist]: All files matched by "/src/node_modules/qunit/qunit/qunit.js" were excluded or matched by prior matchers.
22:21:47 07 08 2020 20:21:47.349:INFO [karma-server]: Karma v5.0.9 server started at http://0.0.0.0:9876/
22:21:47 07 08 2020 20:21:47.350:INFO [launcher]: Launching browsers FirefoxHeadless with concurrency unlimited
22:21:47 07 08 2020 20:21:47.351:INFO [launcher]: Starting browser FirefoxHeadless
22:22:47 07 08 2020 20:22:47.352:WARN [launcher]: FirefoxHeadless have not captured in 60000 ms, killing.
22:22:47 07 08 2020 20:22:47.449:INFO [launcher]: Trying to start FirefoxHeadless again (1/2).
22:23:47 07 08 2020 20:23:47.468:WARN [launcher]: FirefoxHeadless have not captured in 60000 ms, killing.
22:23:47 07 08 2020 20:23:47.573:INFO [launcher]: Trying to start FirefoxHeadless again (2/2).
22:24:47 07 08 2020 20:24:47.592:WARN [launcher]: FirefoxHeadless have not captured in 60000 ms, killing.
22:24:47 07 08 2020 20:24:47.717:ERROR [launcher]: FirefoxHeadless failed 2 times (timeout). Giving up.
22:24:47 Warning: Task "karma:firefox" failed. Use --force to continue.

This started happening today after 15:00 UTC, the last change that passed CI was https://gerrit.wikimedia.org/r/c/VisualEditor/VisualEditor/+/618778.

Event Timeline

matmarex created this task.Aug 7 2020, 8:29 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 7 2020, 8:29 PM
matmarex triaged this task as Unbreak Now! priority.Aug 7 2020, 8:33 PM

This is preventing us from merging changes, e.g. https://gerrit.wikimedia.org/r/c/VisualEditor/VisualEditor/+/610327

Krinkle added a subscriber: Krinkle.

Can you reproduce the issue locally on the same repo with fresh and running e.g. npm it?

Ran fine for me with fresh:

Running "karma:firefox" (karma) task
07 08 2020 21:46:05.332:WARN [filelist]: All files matched by "/ve/node_modules/qunit/qunit/qunit.js" were excluded or matched by prior matchers.
07 08 2020 21:46:05.412:INFO [karma-server]: Karma v5.0.9 server started at http://0.0.0.0:9876/
07 08 2020 21:46:05.412:INFO [launcher]: Launching browsers FirefoxHeadless with concurrency unlimited
07 08 2020 21:46:05.414:INFO [launcher]: Starting browser FirefoxHeadless
07 08 2020 21:46:07.793:INFO [Firefox 68.0 (Linux x86_64)]: Connected on socket wsnpt1qoY8bSQkiSAAAB with id 29299701
................................................................................
................................................................................
................................................................................
......................................................................
Firefox 68.0 (Linux x86_64): Executed 310 of 310 SUCCESS (12.555 secs / 12.457 secs)

Done.
I have no name!@c111d8ac46b7:/ve$

I think we had this issue before when we tried to go from FF60 to FF68: T240955

I've also found others complaining that 68 doesn't work: https://github.com/karma-runner/karma-firefox-launcher/issues/104

hashar added a subscriber: hashar.Aug 10 2020, 12:23 PM

CI wise, I have changed the job to use a new image on 2020-08-03 13:00.

- docker-registry.wikimedia.org/releng/node10-test-browser:0.6.0-s1
+ docker-registry.wikimedia.org/releng/node10-test-browser:0.6.2

Which roughly adds curl, fix some oddity with chromium, upgrade npm and also upgrade all packages.

Looking at the history of commits in CI I have found cd25b6e2bdddf6176b6b6907d6958c8cfd8c6ff8:

commit cd25b6e2bdddf6176b6b6907d6958c8cfd8c6ff8
Author: James D. Forrester <jforrester@wikimedia.org>
Date:   Thu Dec 19 16:32:10 2019 +0000

    jjb: Temporarily roll back node10-test-browser uses to 0.6.0-s1
    
    Apparently there are issues with the new Firefox version, at least
    for VisualEditor and OOjs.
    
    Affected jobs:
    - fresnel-node10-browser-docker
    - generic-node10-browser-docker
    - oojs-core-node10-browser-docker
    - oojs-router-node10-browser-docker
    - unicodejs-node10-browser-docker
    - visualeditor-node10-browser-docker
    
    Leaving node10-test users alone for now.
    
    Bug: T240955
    Change-Id: I3a1a30cbd5b8b815bec3f16852f7efe09756d4d8

So technically this is a dupe of T240955 . I will roll back the CI jobs to the previous image.

Change 619288 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: rollback node10-test-browser to 0.6.0-s1 [2]

https://gerrit.wikimedia.org/r/619288

Change 619288 merged by jenkins-bot:
[integration/config@master] jjb: rollback node10-test-browser to 0.6.0-s1 [2]

https://gerrit.wikimedia.org/r/619288

hashar closed this task as Resolved.Aug 10 2020, 12:33 PM
hashar claimed this task.

And indeed Firefox from 60 to 68 is upgraded:

- 60.8.0esr-1~deb9u1
+ 68.11.0esr-1~deb9u1
Restricted Application added a project: User-Ryasmeen. · View Herald TranscriptAug 10 2020, 12:33 PM