Karma tests using Firefox in VisualEditor/VisualEditor repo are failing
Closed, ResolvedPublic
Actions

Description

Karma tests using Firefox in VisualEditor/VisualEditor repo are failing. The same tests in Chrome finish successfully.

Example: https://gerrit.wikimedia.org/r/c/VisualEditor/VisualEditor/+/619033

Log: https://integration.wikimedia.org/ci/job/visualeditor-node10-browser-docker/1597/console

22:21:47 Running "karma:firefox" (karma) task
22:21:47 07 08 2020 20:21:47.251:WARN [filelist]: All files matched by "/src/node_modules/qunit/qunit/qunit.js" were excluded or matched by prior matchers.
22:21:47 07 08 2020 20:21:47.349:INFO [karma-server]: Karma v5.0.9 server started at http://0.0.0.0:9876/
22:21:47 07 08 2020 20:21:47.350:INFO [launcher]: Launching browsers FirefoxHeadless with concurrency unlimited
22:21:47 07 08 2020 20:21:47.351:INFO [launcher]: Starting browser FirefoxHeadless
22:22:47 07 08 2020 20:22:47.352:WARN [launcher]: FirefoxHeadless have not captured in 60000 ms, killing.
22:22:47 07 08 2020 20:22:47.449:INFO [launcher]: Trying to start FirefoxHeadless again (1/2).
22:23:47 07 08 2020 20:23:47.468:WARN [launcher]: FirefoxHeadless have not captured in 60000 ms, killing.
22:23:47 07 08 2020 20:23:47.573:INFO [launcher]: Trying to start FirefoxHeadless again (2/2).
22:24:47 07 08 2020 20:24:47.592:WARN [launcher]: FirefoxHeadless have not captured in 60000 ms, killing.
22:24:47 07 08 2020 20:24:47.717:ERROR [launcher]: FirefoxHeadless failed 2 times (timeout). Giving up.
22:24:47 Warning: Task "karma:firefox" failed. Use --force to continue.

This started happening today after 15:00 UTC, the last change that passed CI was https://gerrit.wikimedia.org/r/c/VisualEditor/VisualEditor/+/618778.

Details

	Subject	Repo	Branch	Lines +/-
	jjb: rollback node10-test-browser to 0.6.0-s1 [2]	integration/config	master	+1 -1

Customize query in gerrit

Related Objects

Mentioned In: T240955: Firefox CI tests keep failing in VE with Firefox 68
Mentioned Here: rCICFcd25b6e2bddd: jjb: Temporarily roll back node10-test-browser uses to 0.6.0-s1
T240955: Firefox CI tests keep failing in VE with Firefox 68

Event Timeline

matmarex created this task.Aug 7 2020, 8:29 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 7 2020, 8:29 PM

This is preventing us from merging changes, e.g. https://gerrit.wikimedia.org/r/c/VisualEditor/VisualEditor/+/610327

ppelberg subscribed.Aug 7 2020, 9:06 PM

Can you reproduce the issue locally on the same repo with fresh and running e.g. npm it?

Ran fine for me with fresh:

Running "karma:firefox" (karma) task
07 08 2020 21:46:05.332:WARN [filelist]: All files matched by "/ve/node_modules/qunit/qunit/qunit.js" were excluded or matched by prior matchers.
07 08 2020 21:46:05.412:INFO [karma-server]: Karma v5.0.9 server started at http://0.0.0.0:9876/
07 08 2020 21:46:05.412:INFO [launcher]: Launching browsers FirefoxHeadless with concurrency unlimited
07 08 2020 21:46:05.414:INFO [launcher]: Starting browser FirefoxHeadless
07 08 2020 21:46:07.793:INFO [Firefox 68.0 (Linux x86_64)]: Connected on socket wsnpt1qoY8bSQkiSAAAB with id 29299701
................................................................................
................................................................................
................................................................................
......................................................................
Firefox 68.0 (Linux x86_64): Executed 310 of 310 SUCCESS (12.555 secs / 12.457 secs)

Done.
I have no name!@c111d8ac46b7:/ve$

ppelberg added a project: Editing-team (Kanban Board).Aug 8 2020, 12:43 AM

ppelberg moved this task from Incoming to Ready to Be Worked On on the Editing-team (Kanban Board) board.

I think we had this issue before when we tried to go from FF60 to FF68: T240955

I've also found others complaining that 68 doesn't work: https://github.com/karma-runner/karma-firefox-launcher/issues/104

CI wise, I have changed the job to use a new image on 2020-08-03 13:00.

- docker-registry.wikimedia.org/releng/node10-test-browser:0.6.0-s1
+ docker-registry.wikimedia.org/releng/node10-test-browser:0.6.2

Which roughly adds curl, fix some oddity with chromium, upgrade npm and also upgrade all packages.

Looking at the history of commits in CI I have found cd25b6e2bdddf6176b6b6907d6958c8cfd8c6ff8:

commit cd25b6e2bdddf6176b6b6907d6958c8cfd8c6ff8
Author: James D. Forrester <jforrester@wikimedia.org>
Date:   Thu Dec 19 16:32:10 2019 +0000

    jjb: Temporarily roll back node10-test-browser uses to 0.6.0-s1
    
    Apparently there are issues with the new Firefox version, at least
    for VisualEditor and OOjs.
    
    Affected jobs:
    - fresnel-node10-browser-docker
    - generic-node10-browser-docker
    - oojs-core-node10-browser-docker
    - oojs-router-node10-browser-docker
    - unicodejs-node10-browser-docker
    - visualeditor-node10-browser-docker
    
    Leaving node10-test users alone for now.
    
    Bug: T240955
    Change-Id: I3a1a30cbd5b8b815bec3f16852f7efe09756d4d8

So technically this is a dupe of T240955 . I will roll back the CI jobs to the previous image.

Change 619288 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: rollback node10-test-browser to 0.6.0-s1 [2]

https://gerrit.wikimedia.org/r/619288

gerritbot added a project: Patch-For-Review.Aug 10 2020, 12:27 PM

Change 619288 merged by jenkins-bot:
[integration/config@master] jjb: rollback node10-test-browser to 0.6.0-s1 [2]

https://gerrit.wikimedia.org/r/619288

And indeed Firefox from 60 to 68 is upgraded:

- 60.8.0esr-1~deb9u1
+ 68.11.0esr-1~deb9u1

Restricted Application added a project: User-Ryasmeen. · View Herald TranscriptAug 10 2020, 12:33 PM

hashar mentioned this in T240955: Firefox CI tests keep failing in VE with Firefox 68.Aug 10 2020, 12:35 PM

Karma tests using Firefox in VisualEditor/VisualEditor repo are failingClosed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

Karma tests using Firefox in VisualEditor/VisualEditor repo are failing
Closed, ResolvedPublic
Actions