Page MenuHomePhabricator

Switch to spec reporter
Open, Stalled, Needs TriagePublic

Description

Status

Description

We have switched from spec to dot reporter in 2019 (528270). dot reporter provides no information on which tests suites and tests are executed. spec reporter is more verbose but it also provides much more information that is useful in general, but especially when debugging. For example, which tests are executed and which tests are skipped.

Output of dot reporter. Not super helpful. Doesn't say much. Actually, it's lying. It says five tests suites ran and all of them passed. That's not true. Read on.

1~/Documents/gerrit/mediawiki/core$ npm run selenium-test
2
3> selenium-test
4> wdio ./tests/selenium/wdio.conf.js
5
6Execution of 5 workers started at 2022-03-22T16:37:02.776Z
7
8[0-0] RUNNING in chrome - /tests/selenium/specs/page.js
9[0-0] PASSED in chrome - /tests/selenium/specs/page.js
10[0-1] RUNNING in chrome - /tests/selenium/specs/recentchanges.js
11[0-1] PASSED in chrome - /tests/selenium/specs/recentchanges.js
12[0-2] RUNNING in chrome - /tests/selenium/specs/user.js
13[0-2] PASSED in chrome - /tests/selenium/specs/user.js
14[0-3] RUNNING in chrome - /tests/selenium/specs/watchlist.js
15[0-3] PASSED in chrome - /tests/selenium/specs/watchlist.js
16[0-4] RUNNING in chrome - /tests/selenium/wdio-mediawiki/specs/BlankPage.js
17[0-4] PASSED in chrome - /tests/selenium/wdio-mediawiki/specs/BlankPage.js
18
19 "dot" Reporter:
20..............
21
22Spec Files: 5 passed, 5 total (100% completed) in 00:00:54

Output of spec reporter. Much more verbose but much more useful. It says that out of five test suites, only two are running, and three are skipped!

1~/Documents/gerrit/mediawiki/core$ npm run selenium-test
2
3> selenium-test
4> wdio ./tests/selenium/wdio.conf.js
5
6Execution of 5 workers started at 2022-03-22T16:45:16.385Z
7
8[0-0] RUNNING in chrome - /tests/selenium/specs/page.js
9[0-0] PASSED in chrome - /tests/selenium/specs/page.js
10[0-1] RUNNING in chrome - /tests/selenium/specs/recentchanges.js
11[0-1] PASSED in chrome - /tests/selenium/specs/recentchanges.js
12[0-2] RUNNING in chrome - /tests/selenium/specs/user.js
13[0-2] PASSED in chrome - /tests/selenium/specs/user.js
14[0-3] RUNNING in chrome - /tests/selenium/specs/watchlist.js
15[0-3] PASSED in chrome - /tests/selenium/specs/watchlist.js
16[0-4] RUNNING in chrome - /tests/selenium/wdio-mediawiki/specs/BlankPage.js
17[0-4] PASSED in chrome - /tests/selenium/wdio-mediawiki/specs/BlankPage.js
18
19 "spec" Reporter:
20------------------------------------------------------------------
21[Chrome 99.0.4844.83 darwin #0-0] Running: Chrome (v99.0.4844.83) on darwin
22[Chrome 99.0.4844.83 darwin #0-0] Session ID: 5b441791-dca8-4bbb-b82d-7e3713ba58c9
23[Chrome 99.0.4844.83 darwin #0-0]
24[Chrome 99.0.4844.83 darwin #0-0] ยป /tests/selenium/specs/page.js
25[Chrome 99.0.4844.83 darwin #0-0] Page
26[Chrome 99.0.4844.83 darwin #0-0] ? should be previewable
27[Chrome 99.0.4844.83 darwin #0-0] - should be previewable
28[Chrome 99.0.4844.83 darwin #0-0] ? should be creatable
29[Chrome 99.0.4844.83 darwin #0-0] - should be creatable
30[Chrome 99.0.4844.83 darwin #0-0] ? should be re-creatable
31[Chrome 99.0.4844.83 darwin #0-0] - should be re-creatable
32[Chrome 99.0.4844.83 darwin #0-0] ? should be editable @daily
33[Chrome 99.0.4844.83 darwin #0-0] - should be editable @daily
34[Chrome 99.0.4844.83 darwin #0-0] ? should have history @daily
35[Chrome 99.0.4844.83 darwin #0-0] - should have history @daily
36[Chrome 99.0.4844.83 darwin #0-0] ? should be deletable
37[Chrome 99.0.4844.83 darwin #0-0] - should be deletable
38[Chrome 99.0.4844.83 darwin #0-0] ? should be restorable
39[Chrome 99.0.4844.83 darwin #0-0] - should be restorable
40[Chrome 99.0.4844.83 darwin #0-0] ? should be protectable
41[Chrome 99.0.4844.83 darwin #0-0] - should be protectable
42[Chrome 99.0.4844.83 darwin #0-0] - should be undoable
43[Chrome 99.0.4844.83 darwin #0-0]
44[Chrome 99.0.4844.83 darwin #0-0] 9 skipped (1.4s)
45------------------------------------------------------------------
46[Chrome 99.0.4844.83 darwin #0-1] Running: Chrome (v99.0.4844.83) on darwin
47[Chrome 99.0.4844.83 darwin #0-1] Session ID: bf38cf84-f29f-4de6-bf6e-83736626d229
48[Chrome 99.0.4844.83 darwin #0-1]
49[Chrome 99.0.4844.83 darwin #0-1] ยป /tests/selenium/specs/recentchanges.js
50[Chrome 99.0.4844.83 darwin #0-1] Special:RecentChanges
51[Chrome 99.0.4844.83 darwin #0-1] ? shows page creation
52[Chrome 99.0.4844.83 darwin #0-1] - shows page creation
53[Chrome 99.0.4844.83 darwin #0-1]
54[Chrome 99.0.4844.83 darwin #0-1] 1 skipped (1.4s)
55------------------------------------------------------------------
56[Chrome 99.0.4844.83 darwin #0-2] Running: Chrome (v99.0.4844.83) on darwin
57[Chrome 99.0.4844.83 darwin #0-2] Session ID: a692643b-0b55-4d84-91db-849380031f32
58[Chrome 99.0.4844.83 darwin #0-2]
59[Chrome 99.0.4844.83 darwin #0-2] ยป /tests/selenium/specs/user.js
60[Chrome 99.0.4844.83 darwin #0-2] User
61[Chrome 99.0.4844.83 darwin #0-2] โœ“ should be able to create account
62[Chrome 99.0.4844.83 darwin #0-2] โœ“ should be able to log in @daily
63[Chrome 99.0.4844.83 darwin #0-2]
64[Chrome 99.0.4844.83 darwin #0-2] 2 passing (11.3s)
65------------------------------------------------------------------
66[Chrome 99.0.4844.83 darwin #0-3] Running: Chrome (v99.0.4844.83) on darwin
67[Chrome 99.0.4844.83 darwin #0-3] Session ID: fd021e49-412b-43d9-96b5-b53d6a9822b8
68[Chrome 99.0.4844.83 darwin #0-3]
69[Chrome 99.0.4844.83 darwin #0-3] ยป /tests/selenium/specs/watchlist.js
70[Chrome 99.0.4844.83 darwin #0-3] Special:Watchlist
71[Chrome 99.0.4844.83 darwin #0-3] ? should show page with new edit
72[Chrome 99.0.4844.83 darwin #0-3] - should show page with new edit
73[Chrome 99.0.4844.83 darwin #0-3]
74[Chrome 99.0.4844.83 darwin #0-3] 1 skipped (2.9s)
75------------------------------------------------------------------
76[Chrome 99.0.4844.83 darwin #0-4] Running: Chrome (v99.0.4844.83) on darwin
77[Chrome 99.0.4844.83 darwin #0-4] Session ID: b36c8b98-a4bf-4fda-a205-6fe4d1f68432
78[Chrome 99.0.4844.83 darwin #0-4]
79[Chrome 99.0.4844.83 darwin #0-4] ยป /tests/selenium/wdio-mediawiki/specs/BlankPage.js
80[Chrome 99.0.4844.83 darwin #0-4] BlankPage
81[Chrome 99.0.4844.83 darwin #0-4] โœ“ should have its title @daily
82[Chrome 99.0.4844.83 darwin #0-4]
83[Chrome 99.0.4844.83 darwin #0-4] 1 passing (2.2s)
84
85Spec Files: 5 passed, 5 total (100% completed) in 00:00:32

TODO

  • mediawiki/core 773549 793446 2.1.0
  • mediawiki/extensions/AbuseFilter
  • mediawiki/extensions/AdvancedSearch
  • mediawiki/extensions/CentralNotice 793784
  • mediawiki/extensions/CirrusSearch
  • mediawiki/extensions/Cite
  • mediawiki/extensions/CodeMirror
  • mediawiki/extensions/Echo 797186
  • mediawiki/extensions/ElectronPdfService 798624
  • mediawiki/extensions/EntitySchema
  • mediawiki/extensions/FileImporter
  • mediawiki/extensions/GlobalWatchlist
  • mediawiki/extensions/GrowthExperiments 798629
  • mediawiki/extensions/Math 798630
  • mediawiki/extensions/MobileFrontend
  • mediawiki/extensions/Newsletter 798634
  • mediawiki/extensions/Popups
  • mediawiki/extensions/ProofreadPage
  • mediawiki/extensions/RelatedArticles 798646
  • mediawiki/extensions/RevisionSlider
  • mediawiki/extensions/TemplateWizard
  • mediawiki/extensions/TwoColConflict
  • mediawiki/extensions/VisualEditor 793463
  • mediawiki/extensions/Wikibase
  • mediawiki/extensions/WikibaseLexeme
  • mediawiki/extensions/WikiLambda 798636
  • mediawiki/skins/MinervaNeue
  • wikibase/termbox
  • wikidata/query/gui
  • wikimedia/portals/deploy
  • wmde/wdio-wikibase

Notes

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 680377 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/FileImporter@master] selenium: package.json cleanup

https://gerrit.wikimedia.org/r/680377

Change 680345 abandoned by Zfilipin:

[mediawiki/core@master] selenium: use spec reporter

Reason:

https://gerrit.wikimedia.org/r/680345

Change 680377 merged by jenkins-bot:

[mediawiki/extensions/FileImporter@master] selenium: package.json cleanup

https://gerrit.wikimedia.org/r/680377

zeljkofilipin raised the priority of this task from Medium to High.
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)

Change 773549 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/core@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/773549

Eh. IME no-one ever looked at the endless output of selenium and it hid other issues. However, now we've split the selenium tests out into their own job, this is much less of an issue, so sure.

Change 773549 merged by jenkins-bot:

[mediawiki/core@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/773549

Change 793446 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/core@master] wdio-mediawiki: Release 2.1.0

https://gerrit.wikimedia.org/r/793446

Change 793446 merged by jenkins-bot:

[mediawiki/core@master] wdio-mediawiki: Release 2.1.0

https://gerrit.wikimedia.org/r/793446

Change 793463 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/VisualEditor@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/793463

Change 793463 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/793463

Change 793784 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/CentralNotice@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/793784

zeljkofilipin lowered the priority of this task from High to Medium.May 23 2022, 10:27 AM

Change 797186 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/Echo@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/797186

Change 797186 merged by jenkins-bot:

[mediawiki/extensions/Echo@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/797186

Change 798624 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/ElectronPdfService@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/798624

Change 798629 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/GrowthExperiments@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/798629

Change 798630 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/Math@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/798630

Change 798634 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/Newsletter@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/798634

Change 798636 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/WikiLambda@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/798636

Change 798646 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/RelatedArticles@master] WIP selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/798646

Change 798630 merged by jenkins-bot:

[mediawiki/extensions/Math@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/798630

Change 798629 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/798629

Change 798636 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/798636

Change 798646 merged by jenkins-bot:

[mediawiki/extensions/RelatedArticles@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/798646

Change 798634 merged by jenkins-bot:

[mediawiki/extensions/Newsletter@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/798634

Change 798624 merged by jenkins-bot:

[mediawiki/extensions/ElectronPdfService@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/798624

Change 793784 merged by jenkins-bot:

[mediawiki/extensions/CentralNotice@master] selenium: Use @wdio/spec-reporter

https://gerrit.wikimedia.org/r/793784

zeljkofilipin changed the task status from Open to Stalled.May 27 2022, 3:53 PM
zeljkofilipin removed zeljkofilipin as the assignee of this task.

Blocked on T256626: Refactor WebdriverIO tests from sync to async mode. Only repositories already updated to wdio-mediawiki v2.0.0 (required by T256626) can be updated to v2.1.0.

zeljkofilipin raised the priority of this task from Medium to Needs Triage.May 27 2022, 3:53 PM
zeljkofilipin updated the task description. (Show Details)