Page MenuHomePhabricator
Create Task
Maniphest T258528

Minerva browser tests failing without a proper summary
Closed, ResolvedPublic
Actions

Description

Minerva browser tests are failing but from the summary it wasn't clear that a single browser test was responsible:
https://integration.wikimedia.org/ci/job/selenium-daily-beta-Minerva/
There appear to be no logs (other than console output) in the failures or failed tests reported so it wasn't obvious why at first.

PS. Please remove me from the people sent email notifications;.

TODO

  • remove @Jdlrobson from e-mail notifications for selenium-daily-beta-Minerva Jenkins job (615503)

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Selenium: Remove Jon from MinervaNeue notificationsintegration/configmaster+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Jdlrobson created this task.Jul 21 2020, 8:03 PM
Restricted Application added subscribers: Masumrezarock100, Aklapper. · View Herald TranscriptJul 21 2020, 8:03 PM
Jdlrobson added a comment.Jul 21 2020, 8:05 PM

Okay, the logs are there.. seems like the failure is in "Opening and closing the reference drawer Opening a nested reference
12:48:02 [Chrome 71.0.3578.80 linux #0-5] waitUntil condition timed out after 10000ms" so hopefully T258096 will resolve this.

That wasn't clear - previously only failed test videos/screenshots were posted and it reported the failing test in the summary. Is there a reason this no longer happens?

Jdlrobson renamed this task from Minerva browser tests failing to Minerva browser tests failing without a proper summary.Jul 21 2020, 8:05 PM
Jdlrobson updated the task description. (Show Details)
zeljkofilipin triaged this task as High priority.Jul 22 2020, 11:51 AM
zeljkofilipin added a project: User-zeljkofilipin.
zeljkofilipin moved this task from Inbox to Test engineering on the Quality-and-Test-Engineering-Team board.
zeljkofilipin moved this task from Backlog 🪒 to Sprint 🏃‍➡️ on the User-zeljkofilipin board.
zeljkofilipin subscribed.
zeljkofilipin assigned this task to Edtadros.Jul 22 2020, 11:53 AM
zeljkofilipin added a parent task: T255447: Upgrade WebdriverIO to the latest version for MinervaNeue.
zeljkofilipin mentioned this in T255447: Upgrade WebdriverIO to the latest version for MinervaNeue.

Thanks @Jdlrobson for reporting this. It is already tracked in T255447: Upgrade WebdriverIO to the latest version for MinervaNeue but it makes sense to have it as a separate task.

zeljkofilipin updated the task description. (Show Details)Jul 22 2020, 11:56 AM
zeljkofilipin added a comment.Jul 22 2020, 1:38 PM
In T258528#6324039, @Jdlrobson wrote:

previously only failed test videos/screenshots were posted and it reported the failing test in the summary. Is there a reason this no longer happens?

In the last two months, the only changes to selenium tests were webdriverio updates from v4 to v5 and then from v5 to v6:

https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/skins/MinervaNeue/+log/master/tests/selenium

We didn't change any configuration. Maybe the defaults in webdriverio v5 or v6 have changed, so the output is different.

zeljkofilipin added a comment.Jul 22 2020, 1:47 PM
In T258528#6324039, @Jdlrobson wrote:

That wasn't clear - previously only failed test videos/screenshots were posted and it reported the failing test in the summary. Is there a reason this no longer happens?

I took a closer look at the Jenkins job output and I'm not sure what you're referring to. I took a look at an older failure (selenium-daily-beta-Minerva/498) and a recent one (selenium-daily-beta-Minerva/506). Both of them seem to have similar output.

selenium-daily-beta-Minerva/498:

...
------------------------------------------------------------------
[chrome 71.0.3578.80 Linux #0-4] Spec: /src/tests/selenium/specs/mainmenu_loggedin.js
[chrome 71.0.3578.80 Linux #0-4] Running: chrome (v71.0.3578.80) on Linux
[chrome 71.0.3578.80 Linux #0-4] Session ID: 905280344777f1edc4a7e5baeeb694c6
[chrome 71.0.3578.80 Linux #0-4]
[chrome 71.0.3578.80 Linux #0-4] Menus open correct page for anonymous users
[chrome 71.0.3578.80 Linux #0-4]    ? Check links in menu
[chrome 71.0.3578.80 Linux #0-4]    ✖ "before each" hook
[chrome 71.0.3578.80 Linux #0-4]
[chrome 71.0.3578.80 Linux #0-4] 1 failing (1m 0.1s)
[chrome 71.0.3578.80 Linux #0-4]
[chrome 71.0.3578.80 Linux #0-4] 1) Menus open correct page for anonymous users "before each" hook
[chrome 71.0.3578.80 Linux #0-4] Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/src/tests/selenium/specs/mainmenu_loggedin.js)
[chrome 71.0.3578.80 Linux #0-4] Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/src/tests/selenium/specs/mainmenu_loggedin.js)
[chrome 71.0.3578.80 Linux #0-4]     at Hook.Runnable._timeoutError (/src/node_modules/@wdio/mocha-framework/node_modules/mocha/lib/runnable.js:442:10)
[chrome 71.0.3578.80 Linux #0-4]     at Timeout.<anonymous> (/src/node_modules/@wdio/mocha-framework/node_modules/mocha/lib/runnable.js:252:24)
[chrome 71.0.3578.80 Linux #0-4]     at ontimeout (timers.js:436:11)
[chrome 71.0.3578.80 Linux #0-4]     at tryOnTimeout (timers.js:300:5)
[chrome 71.0.3578.80 Linux #0-4]     at listOnTimeout (timers.js:263:5)
[chrome 71.0.3578.80 Linux #0-4]     at Timer.processTimers (timers.js:223:10)
------------------------------------------------------------------
...
Spec Files:	 7 passed, 3 failed, 10 total (100% completed) in 00:06:50 
...

selenium-daily-beta-Minerva/506:

...
------------------------------------------------------------------
[Chrome 71.0.3578.80 linux #0-5] Spec: /src/tests/selenium/specs/references.js
[Chrome 71.0.3578.80 linux #0-5] Running: Chrome (v71.0.3578.80) on linux
[Chrome 71.0.3578.80 linux #0-5] Session ID: 74a66155-1878-4fdd-91b1-625d066ded28
[Chrome 71.0.3578.80 linux #0-5]
[Chrome 71.0.3578.80 linux #0-5] Opening and closing the reference drawer
[Chrome 71.0.3578.80 linux #0-5]    ✓ Opening a reference
[Chrome 71.0.3578.80 linux #0-5]    ✖ Opening a nested reference
[Chrome 71.0.3578.80 linux #0-5]
[Chrome 71.0.3578.80 linux #0-5] 1 passing (18.7s)
[Chrome 71.0.3578.80 linux #0-5] 1 failing
[Chrome 71.0.3578.80 linux #0-5]
[Chrome 71.0.3578.80 linux #0-5] 1) Opening and closing the reference drawer Opening a nested reference
[Chrome 71.0.3578.80 linux #0-5] waitUntil condition timed out after 10000ms
[Chrome 71.0.3578.80 linux #0-5] Error: waitUntil condition timed out after 10000ms
[Chrome 71.0.3578.80 linux #0-5]     at iShouldSeeDrawerWithText (/src/tests/selenium/features/step_definitions/reference_steps.js:25:10)
[Chrome 71.0.3578.80 linux #0-5]     at Context.it (/src/tests/selenium/specs/references.js:44:3)
------------------------------------------------------------------
...
Spec Files:	 9 passed, 1 failed, 10 total (100% completed) in 00:03:25 
...
gerritbot added a comment.Jul 22 2020, 2:40 PM

Change 615503 had a related patch set uploaded (by Edward Tadros; owner: Edward Tadros):
[integration/config@master] Selenium: Remove Jon from MinervaNeue notifications

https://gerrit.wikimedia.org/r/615503

gerritbot added a project: Patch-For-Review.Jul 22 2020, 2:40 PM
zeljkofilipin updated the task description. (Show Details)Jul 24 2020, 1:50 PM
gerritbot added a comment.Jul 24 2020, 1:51 PM

Change 615503 merged by jenkins-bot:
[integration/config@master] Selenium: Remove Jon from MinervaNeue notifications

https://gerrit.wikimedia.org/r/615503

Maintenance_bot removed a project: Patch-For-Review.Jul 24 2020, 2:10 PM
zeljkofilipin mentioned this in T256222: Check if any of selenium-daily-beta* Jenkins jobs started failing after webdriverio update.Jul 28 2020, 9:57 AM
Edtadros closed this task as Resolved.Jul 29 2020, 2:45 PM

Jon was removed and the failing test was skipped.

Jdlrobson added a comment.Jul 29 2020, 2:59 PM

I took a closer look at the Jenkins job output and I'm not sure what you're referring to.

Previously (as in a year ago) jenkins would list "Failed testd" on this page:
https://integration.wikimedia.org/ci/view/Reading-Web/job/selenium-daily-beta-Minerva/

Screen Shot 2020-07-29 at 7.58.30 AM.png (1×2 px, 172 KB)

Now it doesn't mention or list any of the failing tests and I have to jump into the logs to work it out which is time consuming and not very user friendly which I consider a big regression in debugging experience.

Screen Shot 2020-07-29 at 7.57.04 AM.png (902×1 px, 149 KB)

Jdlrobson reopened this task as Open.Jul 29 2020, 3:00 PM
zeljkofilipin claimed this task.Jul 29 2020, 6:06 PM
zeljkofilipin lowered the priority of this task from High to Medium.
zeljkofilipin moved this task from Sprint 🏃‍➡️ to Deep work 🌊 on the User-zeljkofilipin board.
zeljkofilipin added a subscriber: Edtadros.

Decreasing priority since the job is fixed.

zeljkofilipin removed zeljkofilipin as the assignee of this task.Jul 30 2020, 6:18 PM
zeljkofilipin moved this task from Deep work 🌊 to Backlog 🪒 on the User-zeljkofilipin board.
zeljkofilipin closed this task as Resolved.May 3 2021, 4:12 PM
zeljkofilipin claimed this task.

@Jdlrobson 683678 should fix the problem that job did not have data about failed tests. Please reopen the task if I misunderstood the problem.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits