Page MenuHomePhabricator

Find and categorize different kinds of CI flips
Closed, ResolvedPublicSpike

Description

We have been discussing how we can make the CI more reliable. One thing we Test Platform has been talking about is flipping jobs. By "flipping" we mean jobs where the tests show green, but then suddenly something happens and the job becomes red (and stop the merge).

I've been digging into the last seven days of logs on Jenkins (we save logs for seven days) and searching for jobs where the test passes (are green) but the full job becomes a fail (red). I've found two different cases that almost is 100% of those cases. And then there's one other case where test pass but a npm install fails and then make the job fail.

For all failures for quibble runs around 18,5% was flipping (where you could see tests passing but then something made the job fail). There's two cases:

  • The whole build gets canceled while it is waiting (a new patchset arrives, or the gate resets and starts over). We can see that in the logs with a java.lang.InterruptedException. The problem here is that these cancelations are correct but Jenkins categorise them as failures, it should be cancel/aborted since this is normal behavior. When I measured these are 12% of all failures.
  • The other case is when the castor-save job never ran.Here the build passes every test, then blocks waiting for the castor-save-workspace-cache job to save the cache. That job gets canceled while it is still sitting in the queue, so it never starts and never returns a result. The console log shows it plainly: no InterruptedException, no timeout, it goes straight from Waiting for the completion of castor-save-workspace-cache to changed build result to FAILURE with nothingin between, and there is no completed. Result was ... line because the job never completed. We confirmed this directly: for 100% of these the castor-save build was never even created. On the live controller this is about 6,5% of the quibble failures (72 builds). Here's another Jenkins plugin categorize the cancel as a failure. I haven't been able to find out exactly what causes the castor save job to be canceled.

The third case when test runs green but then npm fails in a later case because of cache corruption (T295351) . A fix for that is merged but not released. I went through the last 7 days of Jenkins XML files and it looks like for quibble jobs 5,2% of failures comes from cache corruption (62 out of 1191) . I've prepared a couple of fixes in Quibble (T432934 T432943) so we more easily will see these failures in the future.

This is a start, lets check if we can find other examples of flips.

Event Timeline

• Peter renamed this task from Find and categorize different CI flips to Find and categorize different kinds of CI flips.Jul 24 2026, 1:04 PM
Restricted Application changed the subtype of this task from "Task" to "Spike". · View Herald TranscriptJul 24 2026, 2:26 PM
• Peter moved this task from Finishing Touches to Done on the Test Platform (Arusha 30) board.

Those. where the flips I found.