Page MenuHomePhabricator

Travis and Appveyor: Major improvement to test time
Open, MediumPublic

Description

Travis and Appveyor builds take cca 4 hours. Let's find out, how we could make them faster!

What Travis suggests:

Related Objects

StatusSubtypeAssignedTask
ResolvedDalba
ResolvedDalba
OpenNone
OpenNone
Declinedjayvdb
ResolvedXqt
DeclinedXqt
ResolvedXqt
DeclinedNone
ResolvedXqt
DeclinedNone
ResolvedXqt
ResolvedXqt
ResolvedGoalXqt
ResolvedDvorapa
ResolvedLegoktm
OpenBUG REPORTNone
ResolvedBUG REPORTDvorapa
OpenNone
ResolvedBUG REPORTDvorapa
OpenNone
ResolvedXqt
ResolvedBUG REPORTDalba
ResolvedBUG REPORTDalba
ResolvedBUG REPORTXqt
ResolvedBUG REPORTXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedBUG REPORTDalba
ResolvedXqt
ResolvedXqt
ResolvedBUG REPORTDalba
ResolvedDalba
OpenBUG REPORTNone
ResolvedBUG REPORTXqt
ResolvedBUG REPORTNone
ResolvedBUG REPORTDzahn
ResolvedBUG REPORTDalba
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
OpenNone
ResolvedXqt
OpenNone
DeclinedNone
InvalidNone
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedBUG REPORTXqt
ResolvedDvorapa
ResolvedDvorapa
OpenNone
OpenNone
ResolvedUrbanecm
OpenNone
ResolvedDvorapa
ResolvedBUG REPORTDvorapa
ResolvedDvorapa
OpenNone
OpenNone
ResolvedXqt
ResolvedXqt
OpenNone
ResolvedDvorapa
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
DeclinedNone
InvalidNone
ResolvedUrbanecm
ResolvedXqt
ResolvedXqt
DeclinedNone
Resolvedjayvdb
DeclinedNone
DuplicateNone
ResolvedDvorapa
ResolvedXqt
DeclinedNone
DeclinedNone
Resolvedhashar
ResolvedMpaa
ResolvedAndrew
DeclinedNone
ResolvedXqt
ResolvedDvorapa
ResolvedXqt
ResolvedDvorapa
DeclinedNone
ResolvedXqt
DeclinedNone
ResolvedDvorapa
ResolvedXqt
ResolvedXqt
DeclinedNone
OpenNone

Event Timeline

Xqt triaged this task as Low priority.Apr 5 2018, 4:17 PM

Here is an idea (although I don't know how much it can help):
When a commit only affects a script, there is no need to run all tests, just the related test module for that script. The other parts have already been tested in previous commits. This should be safe because the pywikibot directory does not depend on scripts, and scripts are (should be) independent from each-other.

It seems ok to me, per T60942 no library file calls a script, but scripts call others sometimes.

Good idea but I've no glue how to set up tests to only run a subset of it with scripts changes

Xqt raised the priority of this task from Low to Medium.Sep 26 2018, 12:35 PM

I changed the Travis settings a bit to enable more concurrent jobs and cancel jobs when a new patch was deployed; this will shorten tests time a lot. I also cancelled some jobs but they can be restarted if necessary. Unfortunately I cannot do such administrative things on apveyor.

As result of the settings changes one test in Travis needs one hour now vs. two hours before.

Excellent!

PS: @Ladsgroup has got an access to the current Appveyor account, @jayvdb has got an access to the Wikimedia Appveyor account. To solve the issue, there are two options:

  • contact one of the guys successfully
  • create new Appveyor account (like Dvorapa/Pywikibot)
Xqt renamed this task from Travis: Major improvement to test time to Travis and Appveyor: Major improvement to test time.Oct 1 2018, 11:48 AM
Xqt updated the task description. (Show Details)

Change 463735 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [tests] Set fast_finish: true

https://gerrit.wikimedia.org/r/463735

Change 463735 merged by jenkins-bot:
[pywikibot/core@master] [tests] Set fast_finish: true

https://gerrit.wikimedia.org/r/463735

We need a lot of time for testing Python 2.7 which is 14 minutes at Travis and 52 minutes at Appveyor in total

@Ladsgroup: could you please give access to Appveyer project to @Dvorapa or me to allow settings to be changed.

Change 567337 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [tests] decrease testing time

https://gerrit.wikimedia.org/r/567337

@Xqt How about https://ci.appveyor.com/project/pywikibot/pywikibot ? (you should be able to access it using your GitHub account)

(also sorry for the noise, I changed e-mail address for Pywikibot-oauth from jayvdb@gmail.com to tools.pywikibot@tools.wmflabs.org)

@Xqt How about https://ci.appveyor.com/project/pywikibot/pywikibot ? (you should be able to access it using your GitHub account)

Sure. That was my idea after support request wasn’t complied after several attempts.

@Xqt How about https://ci.appveyor.com/project/pywikibot/pywikibot ? (you should be able to access it using your GitHub account)

Sure. That was my idea after support request wasn’t complied after several attempts.

Sorry for that, I forgot to communicate that I tried it several times before but couldn't find the settings, it's really hard to find. Apparently (and after googling a couple of times) I need to do some black magic to get @Dvorapa there. @Dvorapa Can you give me your email address so I add you?

dvorapa~seznam~cz

I sent an invitation for you, let me if it works.

Change 567337 abandoned by Xqt:
[tests] decrease testing time

Reason:
After rolling builds was set

https://gerrit.wikimedia.org/r/567337

Xqt claimed this task.

@Xqt really? Last build AppVeyor 6 hours, Travis 7 hours

@Xqt really? Last build AppVeyor 6 hours, Travis 7 hours

Travis needed two hours in total because 5 tests are running simultaneously (see my comment dated Sep 27 2018, 01:48). Auto cancel is activated.
Appveyor Needs 3,5 - 7 hours per test but test builds does not block new test anymore because rolling builds is activated.

After dropping Python 2, Travis will be 20 minutes faster and Appveyer will be 1 hour faster.

This is ok for me currently. Do you have other ideas to improve tests time?

Above we discussed caching, tests performance issues, currently also some maxlag and login issues making tests much more longer or even timeouting after 50min

Xqt removed Xqt as the assignee of this task.Feb 1 2020, 7:04 PM

I am also fine with keeping this open. Caching was removed several months ago because there where more problems with it than it saved test time. Other issues like timeout and login problems have their own task. Probably we can make them subtasks.

Probably we can make them subtasks.

Yes, that would be helpful. To address the major issues and list them as subtasks.

Change 569314 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [cleanup] Ignore L10N tests for pywikibot-core repository

https://gerrit.wikimedia.org/r/569314

Change 569314 merged by jenkins-bot:
[pywikibot/core@master] [cleanup] Ignore L10N tests for pywikibot-core repository

https://gerrit.wikimedia.org/r/569314