Page MenuHomePhabricator

Travis and Appveyor: Major improvement to test time
Open, MediumPublic

Description

Travis and Appveyor builds take cca 4 hours. Let's find out, how we could make them faster!

What Travis suggests:

Details

Related Gerrit Patches:
pywikibot/core : master[tests] decrease testing time
pywikibot/core : master[tests] Set fast_finish: true

Related Objects

StatusSubtypeAssignedTask
ResolvedDalba
ResolvedDalba
OpenNone
OpenNone
Declinedjayvdb
ResolvedXqt
OpenXqt
OpenNone
DeclinedNone
ResolvedXqt
DeclinedNone
ResolvedXqt
ResolvedXqt
OpenGoalNone
OpenNone
ResolvedLegoktm
OpenNone
ResolvedXqt
ResolvedBUG REPORTDalba
ResolvedBUG REPORTDalba
ResolvedBUG REPORTXqt
ResolvedBUG REPORTXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedBUG REPORTDalba
ResolvedXqt
ResolvedXqt
ResolvedBUG REPORTDalba
ResolvedDalba
OpenBUG REPORTNone
ResolvedBUG REPORTXqt
ResolvedBUG REPORTNone
ResolvedBUG REPORTDzahn
OpenDalba
ResolvedXqt
ResolvedXqt
ResolvedXqt
ResolvedXqt
OpenNone
OpenXqt
ResolvedXqt
OpenNone
ResolvedXqt
OpenNone
DeclinedNone
OpenNone
ResolvedXqt
OpenNone
OpenXqt
ResolvedXqt
ResolvedXqt
OpenNone
OpenNone
OpenNone
OpenNone
ResolvedDvorapa
OpenNone
ResolvedXqt
OpenNone
OpenNone
ResolvedXqt
ResolvedXqt
ResolvedXqt
DeclinedNone
InvalidNone
ResolvedUrbanecm
OpenNone
ResolvedXqt
OpenNone
Resolvedjayvdb
DeclinedNone
DuplicateNone
ResolvedDvorapa
ResolvedXqt
OpenNone
OpenNone
OpenNone
ResolvedMpaa
ResolvedAndrew
OpenNone
OpenNone
OpenNone
OpenNone
ResolvedDvorapa
OpenNone

Event Timeline

Dvorapa created this task.Feb 2 2018, 2:01 PM
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptFeb 2 2018, 2:01 PM
Dalba moved this task from Backlog to Framework on the Pywikibot-tests board.Apr 5 2018, 1:40 PM
Xqt triaged this task as Low priority.Apr 5 2018, 4:17 PM
Dalba added a comment.EditedSep 25 2018, 8:50 AM

Here is an idea (although I don't know how much it can help):
When a commit only affects a script, there is no need to run all tests, just the related test module for that script. The other parts have already been tested in previous commits. This should be safe because the pywikibot directory does not depend on scripts, and scripts are (should be) independent from each-other.

It seems ok to me, per T60942 no library file calls a script, but scripts call others sometimes.

Xqt added a comment.Sep 26 2018, 6:14 AM

Good idea but I've no glue how to set up tests to only run a subset of it with scripts changes

Xqt raised the priority of this task from Low to Medium.Sep 26 2018, 12:35 PM
Xqt added a comment.Sep 26 2018, 11:48 PM

I changed the Travis settings a bit to enable more concurrent jobs and cancel jobs when a new patch was deployed; this will shorten tests time a lot. I also cancelled some jobs but they can be restarted if necessary. Unfortunately I cannot do such administrative things on apveyor.

Xqt added a comment.Sep 28 2018, 5:04 AM

As result of the settings changes one test in Travis needs one hour now vs. two hours before.

Excellent!

PS: @Ladsgroup has got an access to the current Appveyor account, @jayvdb has got an access to the Wikimedia Appveyor account. To solve the issue, there are two options:

  • contact one of the guys successfully
  • create new Appveyor account (like Dvorapa/Pywikibot)
Xqt renamed this task from Travis: Major improvement to test time to Travis and Appveyor: Major improvement to test time.Oct 1 2018, 11:48 AM
Xqt updated the task description. (Show Details)

Change 463735 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [tests] Set fast_finish: true

https://gerrit.wikimedia.org/r/463735

Change 463735 merged by jenkins-bot:
[pywikibot/core@master] [tests] Set fast_finish: true

https://gerrit.wikimedia.org/r/463735

D3r1ck01 moved this task from Backlog to Needs Review on the Pywikibot board.Dec 22 2018, 6:18 PM
Xqt moved this task from Needs Review to Backlog on the Pywikibot board.
Xqt added a comment.EditedFeb 18 2019, 4:24 PM

We need a lot of time for testing Python 2.7 which is 14 minutes at Travis and 52 minutes at Appveyor in total

Xqt added a comment.Sun, Jan 26, 12:58 PM

@Ladsgroup: could you please give access to Appveyer project to @Dvorapa or me to allow settings to be changed.

Change 567337 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [tests] decrease testing time

https://gerrit.wikimedia.org/r/567337

Dvorapa added a comment.EditedTue, Jan 28, 12:50 PM

@Xqt How about https://ci.appveyor.com/project/pywikibot/pywikibot ? (you should be able to access it using your GitHub account)

(also sorry for the noise, I changed e-mail address for Pywikibot-oauth from jayvdb@gmail.com to tools.pywikibot@tools.wmflabs.org)

Xqt added a comment.Tue, Jan 28, 7:19 PM

@Xqt How about https://ci.appveyor.com/project/pywikibot/pywikibot ? (you should be able to access it using your GitHub account)

Sure. That was my idea after support request wasn’t complied after several attempts.

@Xqt How about https://ci.appveyor.com/project/pywikibot/pywikibot ? (you should be able to access it using your GitHub account)

Sure. That was my idea after support request wasn’t complied after several attempts.

Sorry for that, I forgot to communicate that I tried it several times before but couldn't find the settings, it's really hard to find. Apparently (and after googling a couple of times) I need to do some black magic to get @Dvorapa there. @Dvorapa Can you give me your email address so I add you?

dvorapa~seznam~cz

dvorapa~seznam~cz

I sent an invitation for you, let me if it works.

Works, thank you

Change 567337 abandoned by Xqt:
[tests] decrease testing time

Reason:
After rolling builds was set

https://gerrit.wikimedia.org/r/567337

Xqt closed this task as Resolved.Thu, Jan 30, 6:14 PM
Xqt claimed this task.
Maintenance_bot moved this task from Incoming to Done on the User-Ladsgroup board.Thu, Jan 30, 6:15 PM

@Xqt really? Last build AppVeyor 6 hours, Travis 7 hours

Dvorapa reopened this task as Open.Sat, Feb 1, 4:26 PM
Xqt added a comment.Sat, Feb 1, 5:04 PM

@Xqt really? Last build AppVeyor 6 hours, Travis 7 hours

Travis needed two hours in total because 5 tests are running simultaneously (see my comment dated Sep 27 2018, 01:48). Auto cancel is activated.
Appveyor Needs 3,5 - 7 hours per test but test builds does not block new test anymore because rolling builds is activated.

After dropping Python 2, Travis will be 20 minutes faster and Appveyer will be 1 hour faster.

This is ok for me currently. Do you have other ideas to improve tests time?

Dvorapa added a comment.EditedSat, Feb 1, 6:01 PM

Above we discussed caching, tests performance issues, currently also some maxlag and login issues making tests much more longer or even timeouting after 50min

Xqt removed Xqt as the assignee of this task.Sat, Feb 1, 7:04 PM

I am also fine with keeping this open. Caching was removed several months ago because there where more problems with it than it saved test time. Other issues like timeout and login problems have their own task. Probably we can make them subtasks.

Probably we can make them subtasks.

Yes, that would be helpful. To address the major issues and list them as subtasks.