Page MenuHomePhabricator
Create Task
Maniphest T303737

Selenium tests are being skipped
Closed, ResolvedPublic
Actions

Description

Code in page.js is skipping more tests than it should? The code was introduced in 713682. It was needed for WikiLambda 713548.

		// Don't try to run wikitext-specific tests if the test namespace isn't wikitext by default.
		if ( await Util.isTargetNotWikitext( name ) ) {
			this.skip();
		}

Master branch: 11 skipped, 3 passing. Full output: P40808.

/tests/selenium/specs/page.js
9 skipped

/tests/selenium/specs/recentchanges.js
1 skipped

/tests/selenium/specs/user.js
2 passing

/tests/selenium/specs/watchlist.js
1 skipped

/tests/selenium/wdio-mediawiki/specs/BlankPage.js
1 passing

860029: 0 skipped, 14 passing. Full output: P40811.

/tests/selenium/specs/page.js
9 passing

/tests/selenium/specs/recentchanges.js
1 passing

/tests/selenium/specs/user.js
2 passing

/tests/selenium/specs/watchlist.js
1 passing

/tests/selenium/wdio-mediawiki/specs/BlankPage.js
1 passing

Details

SubjectRepoBranchLines +/-
Fix async/await missmatch for isTargetNotWikitext.mediawiki/coremaster+2 -2
Revert "Selenium tests: Skip wikitext-specific tests if NS isn't wikitext"mediawiki/coremaster+0 -68
wdio-mediawiki: Extend isTargetNotWikitext() to cope with queries for existing pages toomediawiki/coremaster+11 -5
wdio-mediawiki: Release 2.3.0mediawiki/coremaster+10 -1
selenium: Fix failing testsmediawiki/coremaster+32 -32
selenium: Delete automationProtocol settingmediawiki/coremaster+0 -7
selenium: Delete PreferencesPagemediawiki/coremaster+0 -20
WIP selenium: Debugging setupmediawiki/coremaster+112 -31
Customize query in gerrit

Related Objects

Event Timeline

dom_walden created this task.Mar 14 2022, 3:22 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 14 2022, 3:22 PM
zeljkofilipin triaged this task as Medium priority.Mar 15 2022, 9:35 AM
zeljkofilipin added a project: User-zeljkofilipin.
zeljkofilipin moved this task from Backlog 🪒 to Deep work 🌊 on the User-zeljkofilipin board.
zeljkofilipin updated the task description. (Show Details)Mar 17 2022, 3:10 PM
zeljkofilipin updated the task description. (Show Details)Mar 18 2022, 4:35 PM
zeljkofilipin updated the task description. (Show Details)Mar 18 2022, 4:38 PM
zeljkofilipin added a project: MediaWiki-Core-Tests.Mar 22 2022, 4:19 PM
zeljkofilipin moved this task from Inbox to Selenium on the MediaWiki-Core-Tests board.
gerritbot added a comment.Mar 22 2022, 4:43 PM

Change 772879 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/core@master] WIP selenium: Debugging setup

https://gerrit.wikimedia.org/r/772879

gerritbot added a project: Patch-For-Review.Mar 22 2022, 4:43 PM
zeljkofilipin updated the task description. (Show Details)EditedMar 22 2022, 4:47 PM

Output with @wdio/dot-reporter. Not super helpful. Doesn't say much.

P22975 @wdio/dot-reporter
1~/Documents/gerrit/mediawiki/core$ npm run selenium-test
2
3> selenium-test
4> wdio ./tests/selenium/wdio.conf.js
5
6Execution of 5 workers started at 2022-03-22T16:37:02.776Z
7
8[0-0] RUNNING in chrome - /tests/selenium/specs/page.js
9[0-0] PASSED in chrome - /tests/selenium/specs/page.js
10[0-1] RUNNING in chrome - /tests/selenium/specs/recentchanges.js
11[0-1] PASSED in chrome - /tests/selenium/specs/recentchanges.js
12[0-2] RUNNING in chrome - /tests/selenium/specs/user.js
13[0-2] PASSED in chrome - /tests/selenium/specs/user.js
14[0-3] RUNNING in chrome - /tests/selenium/specs/watchlist.js
15[0-3] PASSED in chrome - /tests/selenium/specs/watchlist.js
16[0-4] RUNNING in chrome - /tests/selenium/wdio-mediawiki/specs/BlankPage.js
17[0-4] PASSED in chrome - /tests/selenium/wdio-mediawiki/specs/BlankPage.js
18
19 "dot" Reporter:
20..............
21
22Spec Files: 5 passed, 5 total (100% completed) in 00:00:54

zeljkofilipin added a comment.Mar 22 2022, 4:49 PM

Output with @wdio/spec-reporter. Much more verbose but much more useful.

P22976 @wdio/spec-reporter
1~/Documents/gerrit/mediawiki/core$ npm run selenium-test
2
3> selenium-test
4> wdio ./tests/selenium/wdio.conf.js
5
6Execution of 5 workers started at 2022-03-22T16:45:16.385Z
7
8[0-0] RUNNING in chrome - /tests/selenium/specs/page.js
9[0-0] PASSED in chrome - /tests/selenium/specs/page.js
10[0-1] RUNNING in chrome - /tests/selenium/specs/recentchanges.js
11[0-1] PASSED in chrome - /tests/selenium/specs/recentchanges.js
12[0-2] RUNNING in chrome - /tests/selenium/specs/user.js
13[0-2] PASSED in chrome - /tests/selenium/specs/user.js
14[0-3] RUNNING in chrome - /tests/selenium/specs/watchlist.js
15[0-3] PASSED in chrome - /tests/selenium/specs/watchlist.js
16[0-4] RUNNING in chrome - /tests/selenium/wdio-mediawiki/specs/BlankPage.js
17[0-4] PASSED in chrome - /tests/selenium/wdio-mediawiki/specs/BlankPage.js
18
19 "spec" Reporter:
20------------------------------------------------------------------
21[Chrome 99.0.4844.83 darwin #0-0] Running: Chrome (v99.0.4844.83) on darwin
22[Chrome 99.0.4844.83 darwin #0-0] Session ID: 5b441791-dca8-4bbb-b82d-7e3713ba58c9
23[Chrome 99.0.4844.83 darwin #0-0]
24[Chrome 99.0.4844.83 darwin #0-0] » /tests/selenium/specs/page.js
25[Chrome 99.0.4844.83 darwin #0-0] Page
26[Chrome 99.0.4844.83 darwin #0-0] ? should be previewable
27[Chrome 99.0.4844.83 darwin #0-0] - should be previewable
28[Chrome 99.0.4844.83 darwin #0-0] ? should be creatable
29[Chrome 99.0.4844.83 darwin #0-0] - should be creatable
30[Chrome 99.0.4844.83 darwin #0-0] ? should be re-creatable
31[Chrome 99.0.4844.83 darwin #0-0] - should be re-creatable
32[Chrome 99.0.4844.83 darwin #0-0] ? should be editable @daily
33[Chrome 99.0.4844.83 darwin #0-0] - should be editable @daily
34[Chrome 99.0.4844.83 darwin #0-0] ? should have history @daily
35[Chrome 99.0.4844.83 darwin #0-0] - should have history @daily
36[Chrome 99.0.4844.83 darwin #0-0] ? should be deletable
37[Chrome 99.0.4844.83 darwin #0-0] - should be deletable
38[Chrome 99.0.4844.83 darwin #0-0] ? should be restorable
39[Chrome 99.0.4844.83 darwin #0-0] - should be restorable
40[Chrome 99.0.4844.83 darwin #0-0] ? should be protectable
41[Chrome 99.0.4844.83 darwin #0-0] - should be protectable
42[Chrome 99.0.4844.83 darwin #0-0] - should be undoable
43[Chrome 99.0.4844.83 darwin #0-0]
44[Chrome 99.0.4844.83 darwin #0-0] 9 skipped (1.4s)
45------------------------------------------------------------------
46[Chrome 99.0.4844.83 darwin #0-1] Running: Chrome (v99.0.4844.83) on darwin
47[Chrome 99.0.4844.83 darwin #0-1] Session ID: bf38cf84-f29f-4de6-bf6e-83736626d229
48[Chrome 99.0.4844.83 darwin #0-1]
49[Chrome 99.0.4844.83 darwin #0-1] » /tests/selenium/specs/recentchanges.js
50[Chrome 99.0.4844.83 darwin #0-1] Special:RecentChanges
51[Chrome 99.0.4844.83 darwin #0-1] ? shows page creation
52[Chrome 99.0.4844.83 darwin #0-1] - shows page creation
53[Chrome 99.0.4844.83 darwin #0-1]
54[Chrome 99.0.4844.83 darwin #0-1] 1 skipped (1.4s)
55------------------------------------------------------------------
56[Chrome 99.0.4844.83 darwin #0-2] Running: Chrome (v99.0.4844.83) on darwin
57[Chrome 99.0.4844.83 darwin #0-2] Session ID: a692643b-0b55-4d84-91db-849380031f32
58[Chrome 99.0.4844.83 darwin #0-2]
59[Chrome 99.0.4844.83 darwin #0-2] » /tests/selenium/specs/user.js
60[Chrome 99.0.4844.83 darwin #0-2] User
61[Chrome 99.0.4844.83 darwin #0-2] ✓ should be able to create account
62[Chrome 99.0.4844.83 darwin #0-2] ✓ should be able to log in @daily
63[Chrome 99.0.4844.83 darwin #0-2]
64[Chrome 99.0.4844.83 darwin #0-2] 2 passing (11.3s)
65------------------------------------------------------------------
66[Chrome 99.0.4844.83 darwin #0-3] Running: Chrome (v99.0.4844.83) on darwin
67[Chrome 99.0.4844.83 darwin #0-3] Session ID: fd021e49-412b-43d9-96b5-b53d6a9822b8
68[Chrome 99.0.4844.83 darwin #0-3]
69[Chrome 99.0.4844.83 darwin #0-3] » /tests/selenium/specs/watchlist.js
70[Chrome 99.0.4844.83 darwin #0-3] Special:Watchlist
71[Chrome 99.0.4844.83 darwin #0-3] ? should show page with new edit
72[Chrome 99.0.4844.83 darwin #0-3] - should show page with new edit
73[Chrome 99.0.4844.83 darwin #0-3]
74[Chrome 99.0.4844.83 darwin #0-3] 1 skipped (2.9s)
75------------------------------------------------------------------
76[Chrome 99.0.4844.83 darwin #0-4] Running: Chrome (v99.0.4844.83) on darwin
77[Chrome 99.0.4844.83 darwin #0-4] Session ID: b36c8b98-a4bf-4fda-a205-6fe4d1f68432
78[Chrome 99.0.4844.83 darwin #0-4]
79[Chrome 99.0.4844.83 darwin #0-4] » /tests/selenium/wdio-mediawiki/specs/BlankPage.js
80[Chrome 99.0.4844.83 darwin #0-4] BlankPage
81[Chrome 99.0.4844.83 darwin #0-4] ✓ should have its title @daily
82[Chrome 99.0.4844.83 darwin #0-4]
83[Chrome 99.0.4844.83 darwin #0-4] 1 passing (2.2s)
84
85Spec Files: 5 passed, 5 total (100% completed) in 00:00:32

zeljkofilipin added a comment.Mar 22 2022, 5:02 PM

Uh oh. Looks like a lot of tests are skipped in CI and nobody noticed. See 142198. :/

P22977 npm run selenium-test
1> selenium-test
2> wdio ./tests/selenium/wdio.conf.js
3
4
5Execution of 5 workers started at 2022-03-22T16:46:12.785Z
6
7[0-0] RUNNING in chrome - /tests/selenium/specs/page.js
8[0-0] PASSED in chrome - /tests/selenium/specs/page.js
9[0-1] RUNNING in chrome - /tests/selenium/specs/recentchanges.js
10[0-1] PASSED in chrome - /tests/selenium/specs/recentchanges.js
11[0-2] RUNNING in chrome - /tests/selenium/specs/user.js
12[0-2] PASSED in chrome - /tests/selenium/specs/user.js
13[0-3] RUNNING in chrome - /tests/selenium/specs/watchlist.js
14[0-3] PASSED in chrome - /tests/selenium/specs/watchlist.js
15[0-4] RUNNING in chrome - /tests/selenium/wdio-mediawiki/specs/BlankPage.js
16[0-4] PASSED in chrome - /tests/selenium/wdio-mediawiki/specs/BlankPage.js
17
18 "spec" Reporter:
19------------------------------------------------------------------
20[Chrome 90.0.4430.212 linux #0-0] Running: Chrome (v90.0.4430.212) on linux
21[Chrome 90.0.4430.212 linux #0-0] Session ID: ff2bfd1e-850d-4c66-8826-b2f18c282744
22[Chrome 90.0.4430.212 linux #0-0]
23[Chrome 90.0.4430.212 linux #0-0] » /tests/selenium/specs/page.js
24[Chrome 90.0.4430.212 linux #0-0] Page
25[Chrome 90.0.4430.212 linux #0-0] ? should be previewable
26[Chrome 90.0.4430.212 linux #0-0] - should be previewable
27[Chrome 90.0.4430.212 linux #0-0] ? should be creatable
28[Chrome 90.0.4430.212 linux #0-0] - should be creatable
29[Chrome 90.0.4430.212 linux #0-0] ? should be re-creatable
30[Chrome 90.0.4430.212 linux #0-0] - should be re-creatable
31[Chrome 90.0.4430.212 linux #0-0] ? should be editable @daily
32[Chrome 90.0.4430.212 linux #0-0] - should be editable @daily
33[Chrome 90.0.4430.212 linux #0-0] ? should have history @daily
34[Chrome 90.0.4430.212 linux #0-0] - should have history @daily
35[Chrome 90.0.4430.212 linux #0-0] ? should be deletable
36[Chrome 90.0.4430.212 linux #0-0] - should be deletable
37[Chrome 90.0.4430.212 linux #0-0] ? should be restorable
38[Chrome 90.0.4430.212 linux #0-0] - should be restorable
39[Chrome 90.0.4430.212 linux #0-0] ? should be protectable
40[Chrome 90.0.4430.212 linux #0-0] - should be protectable
41[Chrome 90.0.4430.212 linux #0-0] - should be undoable
42[Chrome 90.0.4430.212 linux #0-0]
43[Chrome 90.0.4430.212 linux #0-0] 9 skipped (5.5s)
44------------------------------------------------------------------
45[Chrome 90.0.4430.212 linux #0-1] Running: Chrome (v90.0.4430.212) on linux
46[Chrome 90.0.4430.212 linux #0-1] Session ID: 1091d40a-8146-43e3-ad31-6945d7ccae00
47[Chrome 90.0.4430.212 linux #0-1]
48[Chrome 90.0.4430.212 linux #0-1] » /tests/selenium/specs/recentchanges.js
49[Chrome 90.0.4430.212 linux #0-1] Special:RecentChanges
50[Chrome 90.0.4430.212 linux #0-1] ? shows page creation
51[Chrome 90.0.4430.212 linux #0-1] - shows page creation
52[Chrome 90.0.4430.212 linux #0-1]
53[Chrome 90.0.4430.212 linux #0-1] 1 skipped (654ms)
54------------------------------------------------------------------
55[Chrome 90.0.4430.212 linux #0-2] Running: Chrome (v90.0.4430.212) on linux
56[Chrome 90.0.4430.212 linux #0-2] Session ID: 83a117c4-5981-4140-96f7-3e2a3cac303d
57[Chrome 90.0.4430.212 linux #0-2]
58[Chrome 90.0.4430.212 linux #0-2] » /tests/selenium/specs/user.js
59[Chrome 90.0.4430.212 linux #0-2] User
60[Chrome 90.0.4430.212 linux #0-2] ✓ should be able to create account
61[Chrome 90.0.4430.212 linux #0-2] ✓ should be able to log in @daily
62[Chrome 90.0.4430.212 linux #0-2]
63[Chrome 90.0.4430.212 linux #0-2] 2 passing (7.9s)
64------------------------------------------------------------------
65[Chrome 90.0.4430.212 linux #0-3] Running: Chrome (v90.0.4430.212) on linux
66[Chrome 90.0.4430.212 linux #0-3] Session ID: eb4f1154-109b-448b-b35e-4a1da5b2a64e
67[Chrome 90.0.4430.212 linux #0-3]
68[Chrome 90.0.4430.212 linux #0-3] » /tests/selenium/specs/watchlist.js
69[Chrome 90.0.4430.212 linux #0-3] Special:Watchlist
70[Chrome 90.0.4430.212 linux #0-3] ? should show page with new edit
71[Chrome 90.0.4430.212 linux #0-3] - should show page with new edit
72[Chrome 90.0.4430.212 linux #0-3]
73[Chrome 90.0.4430.212 linux #0-3] 1 skipped (1.3s)
74------------------------------------------------------------------
75[Chrome 90.0.4430.212 linux #0-4] Running: Chrome (v90.0.4430.212) on linux
76[Chrome 90.0.4430.212 linux #0-4] Session ID: c152bd0b-4ab0-443d-96b0-0f098f4892cd
77[Chrome 90.0.4430.212 linux #0-4]
78[Chrome 90.0.4430.212 linux #0-4] » /tests/selenium/wdio-mediawiki/specs/BlankPage.js
79[Chrome 90.0.4430.212 linux #0-4] BlankPage
80[Chrome 90.0.4430.212 linux #0-4] ✓ should have its title @daily
81[Chrome 90.0.4430.212 linux #0-4]
82[Chrome 90.0.4430.212 linux #0-4] 1 passing (773ms)
83
84
85Spec Files: 5 passed, 5 total (100% completed) in 00:00:24

zeljkofilipin raised the priority of this task from Medium to High.Mar 23 2022, 12:44 PM
zeljkofilipin updated the task description. (Show Details)Mar 23 2022, 1:30 PM
Peter subscribed.EditedMar 24 2022, 8:15 AM

One thing that I think can be fixed is https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/refs/heads/master/tests/selenium/wdio-mediawiki/Util.js#51 - that runs a async method so it should await for the return. Make that function (isTargetNotWikitext) async and await this.getMWApiResponse(....)

gerritbot added a comment.Mar 24 2022, 10:34 AM

Change 773471 had a related patch set uploaded (by Phedenskog; author: Phedenskog):

[mediawiki/core@master] Fix async/await missmatch for isTargetNotWikitext.

https://gerrit.wikimedia.org/r/773471

gerritbot added a comment.Mar 25 2022, 10:22 AM

Change 773589 had a related patch set uploaded (by Zfilipin; author: Jforrester):

[mediawiki/core@master] wdio-mediawiki: Extend isTargetNotWikitext() to cope with queries for existing pages too

https://gerrit.wikimedia.org/r/773589

zeljkofilipin removed zeljkofilipin as the assignee of this task.Apr 11 2022, 4:52 PM
zeljkofilipin moved this task from Deep work 🌊 to Q4 👔 on the User-zeljkofilipin board.
zeljkofilipin subscribed.
zeljkofilipin claimed this task.Nov 8 2022, 12:20 PM
zeljkofilipin moved this task from Q4 👔 to Deep work 🌊 on the User-zeljkofilipin board.
gerritbot added a comment.Nov 8 2022, 12:20 PM

Change 772879 abandoned by Zfilipin:

[mediawiki/core@master] WIP selenium: Debugging setup

Reason:

https://gerrit.wikimedia.org/r/772879

zeljkofilipin removed zeljkofilipin as the assignee of this task.Nov 11 2022, 11:30 AM
zeljkofilipin moved this task from Deep work 🌊 to Q4 👔 on the User-zeljkofilipin board.
zeljkofilipin changed the task status from Open to In Progress.Nov 23 2022, 5:29 PM
zeljkofilipin claimed this task.
zeljkofilipin moved this task from Q4 👔 to Deep work 🌊 on the User-zeljkofilipin board.
zeljkofilipin updated the task description. (Show Details)Nov 23 2022, 5:41 PM
zeljkofilipin updated the task description. (Show Details)Nov 23 2022, 5:48 PM
zeljkofilipin updated the task description. (Show Details)
gerritbot added a comment.Nov 23 2022, 5:50 PM

Change 860029 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/core@master] WIP Revert "Selenium tests: Skip wikitext-specific tests if NS isn't wikitext"

https://gerrit.wikimedia.org/r/860029

zeljkofilipin updated the task description. (Show Details)Nov 23 2022, 6:04 PM
zeljkofilipin updated the task description. (Show Details)Nov 23 2022, 6:10 PM
gerritbot added a comment.Nov 24 2022, 1:58 PM

Change 860565 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/core@master] selenium: Delete PreferencesPage

https://gerrit.wikimedia.org/r/860565

gerritbot added a comment.Nov 24 2022, 3:26 PM

Change 860606 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/core@master] selenium: Delete automationProtocol setting

https://gerrit.wikimedia.org/r/860606

zeljkofilipin updated the task description. (Show Details)Nov 24 2022, 3:35 PM
zeljkofilipin updated the task description. (Show Details)Nov 24 2022, 3:38 PM
zeljkofilipin updated the task description. (Show Details)
gerritbot added a comment.Nov 25 2022, 12:39 PM

Change 860869 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/core@master] selenium: Fix failing tests

https://gerrit.wikimedia.org/r/860869

zeljkofilipin renamed this task from Investigate why Selenium tests are being skipped and get delete page tests running on Beta to Selenium tests are being skipped.Nov 25 2022, 1:12 PM
gerritbot added a comment.Nov 26 2022, 9:56 AM

Change 860565 merged by jenkins-bot:

[mediawiki/core@master] selenium: Delete PreferencesPage

https://gerrit.wikimedia.org/r/860565

ReleaseTaggerBot added a project: MW-1.40-notes (1.40.0-wmf.12; 2022-11-28).Nov 26 2022, 10:01 AM
gerritbot added a comment.Nov 28 2022, 5:51 PM

Change 860869 merged by jenkins-bot:

[mediawiki/core@master] selenium: Fix failing tests

https://gerrit.wikimedia.org/r/860869

gerritbot added a comment.Nov 30 2022, 1:24 PM

Change 860029 merged by jenkins-bot:

[mediawiki/core@master] Revert "Selenium tests: Skip wikitext-specific tests if NS isn't wikitext"

https://gerrit.wikimedia.org/r/860029

ReleaseTaggerBot edited projects, added MW-1.40-notes (1.40.0-wmf.13; 2022-12-05); removed MW-1.40-notes (1.40.0-wmf.12; 2022-11-28).Nov 30 2022, 2:01 PM
gerritbot added a comment.Dec 6 2022, 12:05 PM

Change 860606 merged by jenkins-bot:

[mediawiki/core@master] selenium: Delete automationProtocol setting

https://gerrit.wikimedia.org/r/860606

zeljkofilipin closed this task as Resolved.Dec 6 2022, 12:58 PM
ReleaseTaggerBot edited projects, added MW-1.40-notes (1.40.0-wmf.14; 2022-12-12); removed MW-1.40-notes (1.40.0-wmf.13; 2022-12-05).Dec 6 2022, 1:00 PM
gerritbot added a comment.Sep 21 2023, 5:09 PM

Change 959812 had a related patch set uploaded (by WMDE-Fisch; author: WMDE-Fisch):

[mediawiki/core@master] wdio-mediawiki: Release 2.3.0

https://gerrit.wikimedia.org/r/959812

gerritbot added a comment.Sep 22 2023, 12:11 PM

Change 959812 merged by jenkins-bot:

[mediawiki/core@master] wdio-mediawiki: Release 2.3.0

https://gerrit.wikimedia.org/r/959812

gerritbot added a comment.Feb 14 2024, 8:06 AM

Change 773471 abandoned by Phedenskog:

[mediawiki/core@master] Fix async/await missmatch for isTargetNotWikitext.

Reason:

When we updated to v8 lets go through all missing awaits

https://gerrit.wikimedia.org/r/773471

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL