Page MenuHomePhabricator

investigate aborted labs-tools-heritage-tox-dockerjob in jenkins
Closed, ResolvedPublic

Description

There has been a drastic increase in the labs-tools-heritage-tox-docker jenkins job being aborted recently. This leads to patches not getting the verified flag and thus not merging.

Looking at the logs this is due to Build timed out (after 3 minutes). The fix job seems to take about that time meaning out sometimes finishes sometimes not.

Event Timeline

Lokal_Profil triaged this task as High priority.EditedOct 3 2018, 5:40 PM

I've taken a look at the logs and there is no single test slowing this down but it rather seems to be due to the fact that we run many short unit tests. While we could definitely optimise some of these tests (see e.g. T206148) the current situation requires we choose between adding unit tests and being able to merge patches, which isn't reasonable.

We can increase the timeout of the tests pretty easily if you want.

We can increase the timeout of the tests pretty easily if you want.

Thanks. That would probably be best. I'll look into trimming the tests a bit but since I didn't spot any one test which took longer I don't expect trimming to have a significant impact.

@Legoktm Do we need to do anything on our side? This is essentially rendering patches unmergable as illustrated by https://gerrit.wikimedia.org/r/465007

JeanFred claimed this task.

T206913 was an appropriate fix :)