Page MenuHomePhabricator
Create Task
Maniphest T280331

Switch to spec reporter
Open, Needs TriagePublic
Actions

Description

Status

Description

We have switched from spec to dot reporter in 2019 (528270). dot reporter provides no information on which tests suites and tests are executed. spec reporter is more verbose but it also provides much more information that is useful in general, but especially when debugging. For example, which tests are executed and which tests are skipped.

Output of dot reporter. Not super helpful. Doesn't say much. Actually, it's lying. It says five tests suites ran and all of them passed. That's not true. Read on.

P22975 @wdio/dot-reporter
1~/Documents/gerrit/mediawiki/core$ npm run selenium-test
2
3> selenium-test
4> wdio ./tests/selenium/wdio.conf.js
5
6Execution of 5 workers started at 2022-03-22T16:37:02.776Z
7
8[0-0] RUNNING in chrome - /tests/selenium/specs/page.js
9[0-0] PASSED in chrome - /tests/selenium/specs/page.js
10[0-1] RUNNING in chrome - /tests/selenium/specs/recentchanges.js
11[0-1] PASSED in chrome - /tests/selenium/specs/recentchanges.js
12[0-2] RUNNING in chrome - /tests/selenium/specs/user.js
13[0-2] PASSED in chrome - /tests/selenium/specs/user.js
14[0-3] RUNNING in chrome - /tests/selenium/specs/watchlist.js
15[0-3] PASSED in chrome - /tests/selenium/specs/watchlist.js
16[0-4] RUNNING in chrome - /tests/selenium/wdio-mediawiki/specs/BlankPage.js
17[0-4] PASSED in chrome - /tests/selenium/wdio-mediawiki/specs/BlankPage.js
18
19 "dot" Reporter:
20..............
21
22Spec Files: 5 passed, 5 total (100% completed) in 00:00:54

Output of spec reporter. Much more verbose but much more useful. It says that out of five test suites, only two are running, and three are skipped!

P22976 @wdio/spec-reporter
1~/Documents/gerrit/mediawiki/core$ npm run selenium-test
2
3> selenium-test
4> wdio ./tests/selenium/wdio.conf.js
5
6Execution of 5 workers started at 2022-03-22T16:45:16.385Z
7
8[0-0] RUNNING in chrome - /tests/selenium/specs/page.js
9[0-0] PASSED in chrome - /tests/selenium/specs/page.js
10[0-1] RUNNING in chrome - /tests/selenium/specs/recentchanges.js
11[0-1] PASSED in chrome - /tests/selenium/specs/recentchanges.js
12[0-2] RUNNING in chrome - /tests/selenium/specs/user.js
13[0-2] PASSED in chrome - /tests/selenium/specs/user.js
14[0-3] RUNNING in chrome - /tests/selenium/specs/watchlist.js
15[0-3] PASSED in chrome - /tests/selenium/specs/watchlist.js
16[0-4] RUNNING in chrome - /tests/selenium/wdio-mediawiki/specs/BlankPage.js
17[0-4] PASSED in chrome - /tests/selenium/wdio-mediawiki/specs/BlankPage.js
18
19 "spec" Reporter:
20------------------------------------------------------------------
21[Chrome 99.0.4844.83 darwin #0-0] Running: Chrome (v99.0.4844.83) on darwin
22[Chrome 99.0.4844.83 darwin #0-0] Session ID: 5b441791-dca8-4bbb-b82d-7e3713ba58c9
23[Chrome 99.0.4844.83 darwin #0-0]
24[Chrome 99.0.4844.83 darwin #0-0] » /tests/selenium/specs/page.js
25[Chrome 99.0.4844.83 darwin #0-0] Page
26[Chrome 99.0.4844.83 darwin #0-0] ? should be previewable
27[Chrome 99.0.4844.83 darwin #0-0] - should be previewable
28[Chrome 99.0.4844.83 darwin #0-0] ? should be creatable
29[Chrome 99.0.4844.83 darwin #0-0] - should be creatable
30[Chrome 99.0.4844.83 darwin #0-0] ? should be re-creatable
31[Chrome 99.0.4844.83 darwin #0-0] - should be re-creatable
32[Chrome 99.0.4844.83 darwin #0-0] ? should be editable @daily
33[Chrome 99.0.4844.83 darwin #0-0] - should be editable @daily
34[Chrome 99.0.4844.83 darwin #0-0] ? should have history @daily
35[Chrome 99.0.4844.83 darwin #0-0] - should have history @daily
36[Chrome 99.0.4844.83 darwin #0-0] ? should be deletable
37[Chrome 99.0.4844.83 darwin #0-0] - should be deletable
38[Chrome 99.0.4844.83 darwin #0-0] ? should be restorable
39[Chrome 99.0.4844.83 darwin #0-0] - should be restorable
40[Chrome 99.0.4844.83 darwin #0-0] ? should be protectable
41[Chrome 99.0.4844.83 darwin #0-0] - should be protectable
42[Chrome 99.0.4844.83 darwin #0-0] - should be undoable
43[Chrome 99.0.4844.83 darwin #0-0]
44[Chrome 99.0.4844.83 darwin #0-0] 9 skipped (1.4s)
45------------------------------------------------------------------
46[Chrome 99.0.4844.83 darwin #0-1] Running: Chrome (v99.0.4844.83) on darwin
47[Chrome 99.0.4844.83 darwin #0-1] Session ID: bf38cf84-f29f-4de6-bf6e-83736626d229
48[Chrome 99.0.4844.83 darwin #0-1]
49[Chrome 99.0.4844.83 darwin #0-1] » /tests/selenium/specs/recentchanges.js
50[Chrome 99.0.4844.83 darwin #0-1] Special:RecentChanges
51[Chrome 99.0.4844.83 darwin #0-1] ? shows page creation
52[Chrome 99.0.4844.83 darwin #0-1] - shows page creation
53[Chrome 99.0.4844.83 darwin #0-1]
54[Chrome 99.0.4844.83 darwin #0-1] 1 skipped (1.4s)
55------------------------------------------------------------------
56[Chrome 99.0.4844.83 darwin #0-2] Running: Chrome (v99.0.4844.83) on darwin
57[Chrome 99.0.4844.83 darwin #0-2] Session ID: a692643b-0b55-4d84-91db-849380031f32
58[Chrome 99.0.4844.83 darwin #0-2]
59[Chrome 99.0.4844.83 darwin #0-2] » /tests/selenium/specs/user.js
60[Chrome 99.0.4844.83 darwin #0-2] User
61[Chrome 99.0.4844.83 darwin #0-2] ✓ should be able to create account
62[Chrome 99.0.4844.83 darwin #0-2] ✓ should be able to log in @daily
63[Chrome 99.0.4844.83 darwin #0-2]
64[Chrome 99.0.4844.83 darwin #0-2] 2 passing (11.3s)
65------------------------------------------------------------------
66[Chrome 99.0.4844.83 darwin #0-3] Running: Chrome (v99.0.4844.83) on darwin
67[Chrome 99.0.4844.83 darwin #0-3] Session ID: fd021e49-412b-43d9-96b5-b53d6a9822b8
68[Chrome 99.0.4844.83 darwin #0-3]
69[Chrome 99.0.4844.83 darwin #0-3] » /tests/selenium/specs/watchlist.js
70[Chrome 99.0.4844.83 darwin #0-3] Special:Watchlist
71[Chrome 99.0.4844.83 darwin #0-3] ? should show page with new edit
72[Chrome 99.0.4844.83 darwin #0-3] - should show page with new edit
73[Chrome 99.0.4844.83 darwin #0-3]
74[Chrome 99.0.4844.83 darwin #0-3] 1 skipped (2.9s)
75------------------------------------------------------------------
76[Chrome 99.0.4844.83 darwin #0-4] Running: Chrome (v99.0.4844.83) on darwin
77[Chrome 99.0.4844.83 darwin #0-4] Session ID: b36c8b98-a4bf-4fda-a205-6fe4d1f68432
78[Chrome 99.0.4844.83 darwin #0-4]
79[Chrome 99.0.4844.83 darwin #0-4] » /tests/selenium/wdio-mediawiki/specs/BlankPage.js
80[Chrome 99.0.4844.83 darwin #0-4] BlankPage
81[Chrome 99.0.4844.83 darwin #0-4] ✓ should have its title @daily
82[Chrome 99.0.4844.83 darwin #0-4]
83[Chrome 99.0.4844.83 darwin #0-4] 1 passing (2.2s)
84
85Spec Files: 5 passed, 5 total (100% completed) in 00:00:32

TODO

  • mediawiki/core 773549 793446 2.1.0
  • mediawiki/extensions/AbuseFilter
  • mediawiki/extensions/AdvancedSearch
  • mediawiki/extensions/CentralNotice 793784
  • mediawiki/extensions/CirrusSearch
  • mediawiki/extensions/Cite
  • mediawiki/extensions/CodeMirror
  • mediawiki/extensions/Echo 797186
  • mediawiki/extensions/ElectronPdfService 798624
  • mediawiki/extensions/EntitySchema
  • mediawiki/extensions/FileImporter
  • mediawiki/extensions/GlobalWatchlist
  • mediawiki/extensions/GrowthExperiments 798629
  • mediawiki/extensions/Math 798630
  • mediawiki/extensions/MobileFrontend
  • mediawiki/extensions/Newsletter 798634
  • mediawiki/extensions/Popups
  • mediawiki/extensions/ProofreadPage
  • mediawiki/extensions/RelatedArticles 798646
  • mediawiki/extensions/RevisionSlider
  • mediawiki/extensions/TemplateWizard
  • mediawiki/extensions/TwoColConflict
  • mediawiki/extensions/VisualEditor 793463
  • mediawiki/extensions/Wikibase
  • mediawiki/extensions/WikibaseLexeme
  • mediawiki/extensions/WikiLambda 798636
  • mediawiki/skins/MinervaNeue
  • wikibase/termbox
  • wikidata/query/gui
  • wmde/wdio-wikibase

Notes

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
zeljkofilipin reopened this task as Open.Mar 23 2022, 1:01 PM
zeljkofilipin removed a project: Quality-and-Test-Engineering-Team.
zeljkofilipin moved this task from Waiting 🏹 to Deep work 🌊 on the User-zeljkofilipin board.
zeljkofilipin claimed this task.Mar 23 2022, 3:11 PM
zeljkofilipin raised the priority of this task from Medium to High.
zeljkofilipin updated the task description. (Show Details)Mar 24 2022, 1:33 PM
zeljkofilipin updated the task description. (Show Details)Mar 24 2022, 1:38 PM
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)Mar 24 2022, 1:40 PM
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)Mar 24 2022, 1:42 PM
zeljkofilipin updated the task description. (Show Details)Mar 24 2022, 2:06 PM
zeljkofilipin updated the task description. (Show Details)
gerritbot added a comment.Mar 24 2022, 2:58 PM

Change 773549 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/core@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/773549

gerritbot added a project: Patch-For-Review.Mar 24 2022, 2:58 PM
Jdforrester-WMF subscribed.Mar 24 2022, 3:31 PM

Eh. IME no-one ever looked at the endless output of selenium and it hid other issues. However, now we've split the selenium tests out into their own job, this is much less of an issue, so sure.

gerritbot added a comment.Mar 24 2022, 4:19 PM

Change 773549 merged by jenkins-bot:

[mediawiki/core@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/773549

ReleaseTaggerBot added a project: MW-1.39-notes (1.39.0-wmf.5; 2022-03-28).Mar 24 2022, 5:00 PM
Maintenance_bot removed a project: Patch-For-Review.Mar 24 2022, 5:11 PM
zeljkofilipin mentioned this in T210726: Selenium framework cleanup.Mar 25 2022, 10:29 AM
zeljkofilipin updated the task description. (Show Details)Mar 25 2022, 4:47 PM
zeljkofilipin removed zeljkofilipin as the assignee of this task.Apr 11 2022, 4:53 PM
zeljkofilipin moved this task from Deep work 🌊 to Q4 👔 on the User-zeljkofilipin board.
zeljkofilipin moved this task from Q4 👔 to Project ♟ on the User-zeljkofilipin board.Apr 29 2022, 12:25 PM
zeljkofilipin claimed this task.May 18 2022, 2:24 PM
zeljkofilipin moved this task from Project ♟ to Deep work 🌊 on the User-zeljkofilipin board.
gerritbot added a comment.May 19 2022, 11:45 AM

Change 793446 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/core@master] wdio-mediawiki: Release 2.1.0

https://gerrit.wikimedia.org/r/793446

gerritbot added a project: Patch-For-Review.May 19 2022, 11:45 AM
gerritbot added a comment.May 19 2022, 12:07 PM

Change 793446 merged by jenkins-bot:

[mediawiki/core@master] wdio-mediawiki: Release 2.1.0

https://gerrit.wikimedia.org/r/793446

Maintenance_bot removed a project: Patch-For-Review.May 19 2022, 12:30 PM
zeljkofilipin updated the task description. (Show Details)May 19 2022, 12:39 PM
zeljkofilipin updated the task description. (Show Details)May 19 2022, 12:42 PM
Restricted Application added a subscriber: Masumrezarock100. · View Herald TranscriptMay 19 2022, 12:42 PM
zeljkofilipin updated the task description. (Show Details)May 19 2022, 12:44 PM
ReleaseTaggerBot edited projects, added MW-1.39-notes (1.39.0-wmf.13; 2022-05-23); removed MW-1.39-notes (1.39.0-wmf.5; 2022-03-28).May 19 2022, 1:00 PM
zeljkofilipin updated the task description. (Show Details)May 19 2022, 1:19 PM
gerritbot added a comment.May 19 2022, 1:29 PM

Change 793463 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/VisualEditor@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/793463

gerritbot added a project: Patch-For-Review.May 19 2022, 1:29 PM
zeljkofilipin updated the task description. (Show Details)May 19 2022, 1:34 PM
zeljkofilipin removed zeljkofilipin as the assignee of this task.May 19 2022, 1:43 PM
zeljkofilipin moved this task from Deep work 🌊 to Project ♟ on the User-zeljkofilipin board.
gerritbot added a comment.May 19 2022, 2:05 PM

Change 793463 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/793463

Maintenance_bot removed a project: Patch-For-Review.May 19 2022, 2:30 PM
zeljkofilipin updated the task description. (Show Details)May 20 2022, 11:11 AM
zeljkofilipin claimed this task.May 20 2022, 11:59 AM
zeljkofilipin moved this task from Project ♟ to Deep work 🌊 on the User-zeljkofilipin board.
zeljkofilipin updated the task description. (Show Details)May 20 2022, 1:49 PM
gerritbot added a comment.May 20 2022, 1:55 PM

Change 793784 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/CentralNotice@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/793784

gerritbot added a project: Patch-For-Review.May 20 2022, 1:55 PM
zeljkofilipin updated the task description. (Show Details)May 20 2022, 1:57 PM
zeljkofilipin lowered the priority of this task from High to Medium.May 23 2022, 10:27 AM
gerritbot added a comment.May 23 2022, 11:17 AM

Change 797186 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/Echo@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/797186

zeljkofilipin updated the task description. (Show Details)May 23 2022, 11:19 AM
gerritbot added a comment.May 23 2022, 11:49 AM

Change 797186 merged by jenkins-bot:

[mediawiki/extensions/Echo@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/797186

zeljkofilipin updated the task description. (Show Details)May 23 2022, 1:51 PM
gerritbot added a comment.May 24 2022, 11:46 AM

Change 798624 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/ElectronPdfService@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/798624

zeljkofilipin updated the task description. (Show Details)May 24 2022, 11:48 AM
gerritbot added a comment.May 24 2022, 11:54 AM

Change 798629 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/GrowthExperiments@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/798629

zeljkofilipin updated the task description. (Show Details)May 24 2022, 11:55 AM
gerritbot added a comment.May 24 2022, 12:00 PM

Change 798630 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/Math@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/798630

zeljkofilipin updated the task description. (Show Details)May 24 2022, 12:01 PM
gerritbot added a comment.May 24 2022, 12:05 PM

Change 798634 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/Newsletter@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/798634

zeljkofilipin updated the task description. (Show Details)May 24 2022, 12:06 PM
gerritbot added a comment.May 24 2022, 12:12 PM

Change 798636 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/WikiLambda@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/798636

zeljkofilipin updated the task description. (Show Details)May 24 2022, 12:13 PM
gerritbot added a comment.May 24 2022, 12:36 PM

Change 798646 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/RelatedArticles@master] WIP selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/798646

zeljkofilipin updated the task description. (Show Details)May 24 2022, 12:37 PM
gerritbot added a comment.May 24 2022, 1:29 PM

Change 798630 merged by jenkins-bot:

[mediawiki/extensions/Math@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/798630

ReleaseTaggerBot edited projects, added MW-1.39-notes (1.39.0-wmf.14; 2022-05-30); removed MW-1.39-notes (1.39.0-wmf.13; 2022-05-23).May 24 2022, 2:00 PM
gerritbot added a comment.May 24 2022, 2:08 PM

Change 798629 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/798629

gerritbot added a comment.May 24 2022, 2:20 PM

Change 798636 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/798636

zeljkofilipin updated the task description. (Show Details)May 24 2022, 2:58 PM
zeljkofilipin updated the task description. (Show Details)May 24 2022, 3:01 PM
zeljkofilipin updated the task description. (Show Details)
gerritbot added a comment.May 24 2022, 4:44 PM

Change 798646 merged by jenkins-bot:

[mediawiki/extensions/RelatedArticles@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/798646

zeljkofilipin updated the task description. (Show Details)May 25 2022, 8:19 AM
gerritbot added a comment.May 25 2022, 12:36 PM

Change 798634 merged by jenkins-bot:

[mediawiki/extensions/Newsletter@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/798634

gerritbot added a comment.May 25 2022, 12:36 PM

Change 798624 merged by jenkins-bot:

[mediawiki/extensions/ElectronPdfService@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/798624

zeljkofilipin mentioned this in rEEPS316b078d8b3a: selenium: Use @wdio/spec-reporter.May 25 2022, 12:38 PM
gerritbot added a comment.May 25 2022, 12:55 PM

Change 793784 merged by jenkins-bot:

[mediawiki/extensions/CentralNotice@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/793784

zeljkofilipin updated the task description. (Show Details)May 26 2022, 3:27 PM
zeljkofilipin updated the task description. (Show Details)May 27 2022, 3:50 PM
zeljkofilipin changed the task status from Open to Stalled.May 27 2022, 3:53 PM
zeljkofilipin removed zeljkofilipin as the assignee of this task.

Blocked on T256626: Refactor WebdriverIO tests from sync to async mode. Only repositories already updated to wdio-mediawiki v2.0.0 (required by T256626) can be updated to v2.1.0.

zeljkofilipin raised the priority of this task from Medium to Needs Triage.May 27 2022, 3:53 PM
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin moved this task from Deep work 🌊 to Waiting 🏹 on the User-zeljkofilipin board.Jun 8 2022, 12:16 PM
zeljkofilipin mentioned this in T324685: Upgrade WebdriverIO to v8 in all repositories.Dec 15 2022, 11:17 AM
zeljkofilipin updated the task description. (Show Details)Mar 17 2023, 2:36 PM
Pppery removed a project: Patch-For-Review.Apr 2 2023, 1:22 AM
zeljkofilipin changed the task status from Stalled to Open.Aug 24 2023, 1:17 PM
zeljkofilipin moved this task from Waiting 🏹 to Q4 👔 on the User-zeljkofilipin board.
zeljkofilipin edited parent tasks, added: T324685: Upgrade WebdriverIO to v8 in all repositories; removed: T210726: Selenium framework cleanup.Jan 19 2024, 1:06 PM
zeljkofilipin moved this task from Q4 👔 to Backlog 🪒 on the User-zeljkofilipin board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL