If I submit a change, jenkins runs the tests; if I +2, it runs them again. there is no reason for it to do that unless something was merged in the interim.
Description
Description
Event Timeline
Comment Actions
20:00 < ori> related: if i submit a patch, jenkins starts running the tests. if i +2 it, *don't wait for the commit tests to finish, abort them*
20:00 < ori> rather than wait for the test suite to finish just so you can run it again
Comment Actions
I remember some similar conversation a few days ago which I think i replied to but it is good thing to have it there.
The reason the tests are rerun is because either:
a) something is about to be changed
b) another repository the job depends upon might have changed
Lets describe the two cases:
a) something is about to be changed
When changes enter Zuul gate via Code-Review +2 (pipeline gate-and-submit), they are enqueued speculating the patches ahead in the queue are going to pass the tests. Ensuring the branch will still be passing.
As an example, imagine two changes based on the same parent commit (tip of master branch):
- Patch A change some internal API in a non compatible way and update all calls. Tests pass
- Patch B rely on the same internal API as it was in master branch.
If one +2 B and then +2 A Zuul tests in parallel:
Patch B -> pass tests, get merged
Patch A with Patch B applied -> probably fails tests since Patch A did not update the code introduced by B.
If one +2 A and then +2 B Zuul tests in parallel:
Patch A -> pass tests, get merged
Patch B with Patch A applied -> probably fails tests because of A changing the API.
b) another repository the job depends upon might have changed
The job running unit tests for extensions depends at least on three repositories: mediawiki/core , mediawiki/vendor and the extension itself.
When a patch has been proposed for an extension 'master' branch, it has been tested with the tip of the master branches of core and vendor in whatever state they were when the patch was proposed.
The extension change can be +2ed a few day later and core/vendor master branches most probably have evolved even if the extension change parent commit is still the tip of the extension master branch. Thus we rerun the tests simply because the other repositories branches might have changed.
Combined with the first use case, you can ensure that your extension is being tested with code in other repositories that is about to land in. Ie:
Patch A propose a change to mediawiki/core and is +2ed
Patch B propose a change to an extension and is +2ed
Zuul will enter in the queue:
Patch A of mediawiki/core
Patch B of extension tested with mediawiki/core with Patch A applied
conclusion
I don't think we can easily skip running tests without a massive effort to build the dependency tree of all repositories involved in tests. It seems much easier to brute retest everything.
I can understand developers being sometime angry because it takes a few minutes to have a change merged. But in most case I don't think we need to have merge to happen in a few seconds. In case of emergency, one can still cherry pick the patch and deploy it, thus bypassing all tests.
We will probably end up testing all extensions together at some point, which will make the tests to take 15 - 20 minutes. But then Zuul tests them in parallel so that speed things up even if they are enqueued sequentially. If tests are deemed too slow, that would need developers efforts to make them faster, and I believe there is a lot of low hanging fruits that would let us easily cute the running time.
The Zuul behavior is explained in upstream documentation at http://ci.openstack.org/zuul/gating.html . I wrote the second part "Cross projects dependencies". I am more than willing to help enhance that documentation if it is unclear.