Experiment with browser testing in other software languages
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	dduvall
	Aug 12 2015, 9:21 PM

Description

It was brought up the Testing: Where does it hurt? discussion that we should try out end-to-end testing frameworks in other languages such as JS other languages that MW devs might be more familiar with.

Language	Ruby	JavaScript
Implementation	CRuby	Node.js
Package manager	RubyGems, Bundler	npm
Build automation	Rake	Grunt
Linter	RuboCop	JSHint, JSCS
Testing framework	Cucumber, RSpec	Mocha
Browser driver	Selenium	Selenium
Selenium bindings	Ruby bindings (github.com, code.google.com, API)	WebDriverJS (github.com, code.google.com, API)
Browser driver package	selenium-webdriver (rubygems.org)	selenium-webdriver (npmjs.com)
Nicer API on top of Selenium	watir-webdriver	none
Page object	page-object	in Malu
Shared Selenium code	mediawiki_selenium	Malu
Mediawiki API	mediawiki_api	nodemw

Details

Project	Branch	Lines +/-	Subject
mediawiki/core	master	+111 -2	WIP End-to-end tests with Malu
mediawiki/core	master	+64 -1	WIP Nightwatch.js
mediawiki/core	master	+55 -2	WIP WebDriverJs + QUnit
mediawiki/core	master	+64 -2	WIP WebDriverJs + Jasmine

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		zeljkofilipin	T108874 Experiment with browser testing in other software languages
		Declined		zeljkofilipin	T126435 Create a few tests using Nightwatch.js

Event Timeline

dduvall created this task.Aug 12 2015, 9:21 PM

dduvall raised the priority of this task from to Needs Triage.

dduvall updated the task description. (Show Details)

dduvall added a project: Testing-Initiative-2015.

dduvall added subscribers: dduvall, • ggellerman, Jdlrobson.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 12 2015, 9:21 PM

dduvall added a project: Browser-Tests-Infrastructure.Aug 12 2015, 9:21 PM

dduvall set Security to None.

Tweaked the title as we already do awesome testing in other human languages.

zeljkofilipin triaged this task as Low priority.Aug 25 2015, 3:52 PM

zeljkofilipin moved this task from Inbox to Ruby on the Browser-Tests-Infrastructure board.

zeljkofilipin added a subscriber: zeljkofilipin.

This also came up in the RelEng <-> VE SLU discussion:

zeljkofilipin claimed this task.Nov 25 2015, 5:03 PM

zeljkofilipin raised the priority of this task from Low to Medium.

zeljkofilipin moved this task from Backlog to Doing on the Testing-Initiative-2015 board.

zeljkofilipin moved this task from Ruby to In Progress on the Browser-Tests-Infrastructure board.

zeljkofilipin added a comment.Nov 25 2015, 5:34 PM

This comment was removed by zeljkofilipin.

zeljkofilipin added a subscriber: Vikassy.Nov 25 2015, 5:40 PM

Change 255419 had a related patch set uploaded (by Dduvall):
[WIP] Experimenting with JS end-to-end tests

https://gerrit.wikimedia.org/r/255419

gerritbot added a project: Patch-For-Review.Nov 25 2015, 6:29 PM

zeljkofilipin added a comment.Nov 27 2015, 9:29 AM

This comment was removed by zeljkofilipin.

zeljkofilipin added a comment.Nov 27 2015, 4:09 PM

This comment was removed by zeljkofilipin.

zeljkofilipin added a comment.Nov 30 2015, 10:25 AM

This comment was removed by zeljkofilipin.

zeljkofilipin added a comment.Nov 30 2015, 10:26 AM

This comment was removed by zeljkofilipin.

Change 255984 had a related patch set uploaded (by Zfilipin):
WIP WebDriverJs

https://gerrit.wikimedia.org/r/255984

zeljkofilipin updated the task description. (Show Details)Dec 2 2015, 11:17 AM

zeljkofilipin updated the task description. (Show Details)

zeljkofilipin updated the task description. (Show Details)Dec 2 2015, 11:20 AM

zeljkofilipin updated the task description. (Show Details)Dec 2 2015, 11:23 AM

zeljkofilipin updated the task description. (Show Details)Dec 2 2015, 11:25 AM

zeljkofilipin updated the task description. (Show Details)Dec 2 2015, 11:29 AM

zeljkofilipin updated the task description. (Show Details)Dec 2 2015, 11:33 AM

zeljkofilipin updated the task description. (Show Details)Dec 2 2015, 11:44 AM

zeljkofilipin updated the task description. (Show Details)Dec 2 2015, 11:47 AM

Change 256404 had a related patch set uploaded (by Zfilipin):
WIP WIP WebDriverJs Mocha

https://gerrit.wikimedia.org/r/256404

https://github.com/webdriverio/webdriverio/issues/138#issuecomment-32051980 does a loose comparaisons of selenium-webdriverjs vs WD.js vs webdriver.io. Seems the later two comes on top of selenium-webdriverjs and thus WD.js/webdriver.io would be similar to Watir in the ruby stack.

greg awarded a token.Dec 3 2015, 7:01 PM

zeljkofilipin added a project: JavaScript.Dec 9 2015, 9:58 AM

zeljkofilipin mentioned this in T120941: JavaScript + Selenium = <3.Dec 9 2015, 2:34 PM

zeljkofilipin updated the task description. (Show Details)Dec 9 2015, 2:43 PM

zeljkofilipin updated the task description. (Show Details)Dec 17 2015, 11:27 AM

zeljkofilipin added a comment.Dec 22 2015, 4:01 PM

This comment was removed by zeljkofilipin.

Ryasmeen added a subscriber: Ryasmeen.Jan 6 2016, 7:30 PM

zeljkofilipin added a subscriber: Etonkovidova.Jan 11 2016, 9:40 PM

zeljkofilipin updated the task description. (Show Details)Jan 28 2016, 3:01 PM

Change 255419 abandoned by Zfilipin:
WIP WebDriverJs Jasmine

Reason:
Abandoning in favor of https://gerrit.wikimedia.org/r/#/c/256404/

https://gerrit.wikimedia.org/r/255419

Change 255984 abandoned by Zfilipin:
WIP WebDriverJs QUnit

Reason:
Abandoning in favor of https://gerrit.wikimedia.org/r/#/c/256404/

https://gerrit.wikimedia.org/r/255984

Change 268412 had a related patch set uploaded (by Zfilipin):
WIP Nightwatch.js

https://gerrit.wikimedia.org/r/268412

zeljkofilipin closed subtask T126435: Create a few tests using Nightwatch.js as Declined.Feb 10 2016, 1:00 PM

Change 268412 abandoned by Zfilipin:
WIP Nightwatch.js

https://gerrit.wikimedia.org/r/268412

zeljkofilipin removed zeljkofilipin as the assignee of this task.Feb 16 2016, 10:01 AM

zeljkofilipin moved this task from In Progress to Next on the Browser-Tests-Infrastructure board.

zeljkofilipin moved this task from Doing to Radar on the Testing-Initiative-2015 board.

zeljkofilipin claimed this task.Feb 23 2016, 3:40 PM

zeljkofilipin moved this task from Next to In Progress on the Browser-Tests-Infrastructure board.

zeljkofilipin moved this task from Radar to Doing on the Testing-Initiative-2015 board.Mar 2 2016, 1:08 PM

Is this done? Did we decide on mocha (that's what's in malu, at least)?

greg added a project: releng-201516-q4.Mar 28 2016, 11:58 PM

@zeljkofilipin, should we abandon the WIP core patch for now and close this out? We can still reference it for implementation details but i think the actual end-to-end tests that exercise the new framework's basic features should live in the Malu project.

dduvall mentioned this in T131205: Establish first end-to-end tests for Malu testing framework.Mar 29 2016, 10:21 PM

@greg: I think this can be resolved.
@dduvall: I will do the needful.

@dduvall: I am not sure I have understood you: are you suggesting 256404 should be abandoned?

In T108874#2183697, @zeljkofilipin wrote:

@dduvall: I am not sure I have understood you: are you suggesting 256404 should be abandoned?

Yep! The way I see it, this experiment was successful but merging that patch into core at this point would be premature. I think we should abandon the patch, finish the Malu prototype and prerelease (w/ CI support), then see about fully porting over the end-to-end core tests from MW-Selenium—reviving the patch if that makes sense, or simply starting fresh and using it for reference.

Just in general, I'd like to see us get better about scoping experimental work as such, and being ok with abandoning it as a resolution. That doesn't mean the experiment wasn't fruitful or even informative, just that the code is likely half-baked and merging it would incur more debt than it would pay back.

Well reasoned/said, Dan.