Details
- Reference
- bz52424
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Declined | None | T54424 Run subset of browser tests on isolated CI instances per commit submitted in mediawiki/core | |||
| Declined | None | T52576 Select a set of stable tests for smoke test suite | |||
| Declined | greg | T53492 Create hermetic test environment runnable on labs infra | |||
| Resolved | • dduvall | T103039 Define JJB builder for running a subset of integration MW-Selenium tests | |||
| Resolved | • dduvall | T103135 Automatic account creation for MW-Selenium test user in integration environment |
Event Timeline
According to Wikipedia[1]: "In computer programming and software testing, smoke testing is preliminary testing to reveal simple failures severe enough to reject a prospective software release."
Hermetic instance in this context would probably be an instance created using a tool like Vagrant.
1: http://en.wikipedia.org/wiki/Smoke_testing#Software_development
(In reply to comment #2)
According to Wikipedia[1]: "In computer programming and software testing,
smoke
testing is preliminary testing to reveal simple failures severe enough to
reject a prospective software release."
Do you mean the subset of phpunit tests defined by bug 51491? If so, maybe you should say that.
Hermetic instance in this context would probably be an instance created
using a tool like Vagrant.1: http://en.wikipedia.org/wiki/Smoke_testing#Software_development
You mean like bug 45499? What is the advantage of spinning up virtual instances for this? Bug 48407 only suggested using virtual instances when the test needs root access for some reason.
(In reply to comment #3)
(In reply to comment #2)
According to Wikipedia[1]: "In computer programming and software testing,
smoke
testing is preliminary testing to reveal simple failures severe enough to
reject a prospective software release."Do you mean the subset of phpunit tests defined by bug 51491? If so, maybe
you
should say that.
Not just phpunit tests, but also browser tests. But yes, those defined by 50576.
Hermetic instance in this context would probably be an instance created
using a tool like Vagrant.1: http://en.wikipedia.org/wiki/Smoke_testing#Software_development
You mean like bug 45499? What is the advantage of spinning up virtual
instances
for this?
Scaling so we can run more eg browser tests. For instance, the set of known good/smoke tests, which include browser tests, per patch set submitted. Right now we're pretty bottle necked on hardware.
The hardware bottleneck is also currently a human bottle neck as Chris/Zeljko review the browser tests fails each day (they're run twice daily). They don't really have bandwidth to do more.
But, if we tie browser test passing to patch submission, then the weight is transfered from QA to dev, where it should be when developing (to also catch the issues more quickly, getting us closer to being able to increase deploy cycle).
Bug 48407 only suggested using virtual instances when the test
needs
root access for some reason.
Which is also good (though I don't know the priority of it, I haven't heard people clamoring for it), but the scaling issue is the primary win for us.
(In reply to comment #5)
Antoine, is this implemented?
Nop, still blocked by bug 51492 which is not prioritized. I am working on browser tests and Parsoid/VE integration tests right now.