Travis and Appveyor: Major improvement to test time
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Dvorapa
	Feb 2 2018, 2:01 PM

Description

Travis and Appveyor builds take cca 4 hours. Let's find out, how we could make them faster!

What Travis suggests:

Details

Project	Branch	Lines +/-	Subject
pywikibot/core	master	+25 -27	[tests] always use TestTimerMixin with TestCaseBase
pywikibot/core	master	+4 -20	[tests] Improvement to test time at appveyor
pywikibot/core	master	+2 -1	[cleanup] Ignore L10N tests for pywikibot-core repository
pywikibot/core	master	+15 -15	[tests] decrease testing time
pywikibot/core	master	+1 -0	[tests] Set fast_finish: true

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T152961 Improve tests suite
Resolved	Xqt	T186323 Travis and Appveyor: Major improvement to test time
Declined	jayvdb	T106070 Travis python package caching
Resolved	Xqt	T191188 error: Could not find suitable distribution for Requirement.parse
Declined	Xqt	T191632 l10n_tests should run only once for every committed patch
Open	None	T244082 tests.collector does ignore disabled_test_modules in some cases
Resolved	Xqt	T242081 Pywikibot fails to access Wikidata due to high maxlag lately

Event Timeline

Dvorapa created this task.Feb 2 2018, 2:01 PM

Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptFeb 2 2018, 2:01 PM

Dvorapa added a subtask: T106070: Travis python package caching.Feb 2 2018, 2:04 PM

Dvorapa added a parent task: T152961: Improve tests suite.

Dalba moved this task from Backlog to Framework on the Pywikibot-tests board.Apr 5 2018, 1:40 PM

Xqt triaged this task as Low priority.Apr 5 2018, 4:17 PM

Xqt added a subtask: T191632: l10n_tests should run only once for every committed patch.Apr 6 2018, 1:40 PM

Xqt closed subtask T106070: Travis python package caching as Declined.Apr 14 2018, 7:07 PM

Here is an idea (although I don't know how much it can help):
When a commit only affects a script, there is no need to run all tests, just the related test module for that script. The other parts have already been tested in previous commits. This should be safe because the pywikibot directory does not depend on scripts, and scripts are (should be) independent from each-other.

It seems ok to me, per T60942 no library file calls a script, but scripts call others sometimes.

Good idea but I've no glue how to set up tests to only run a subset of it with scripts changes

Xqt raised the priority of this task from Low to Medium.Sep 26 2018, 12:35 PM

I changed the Travis settings a bit to enable more concurrent jobs and cancel jobs when a new patch was deployed; this will shorten tests time a lot. I also cancelled some jobs but they can be restarted if necessary. Unfortunately I cannot do such administrative things on apveyor.

As result of the settings changes one test in Travis needs one hour now vs. two hours before.

Excellent!

PS: @Ladsgroup has got an access to the current Appveyor account, @jayvdb has got an access to the Wikimedia Appveyor account. To solve the issue, there are two options:

contact one of the guys successfully
create new Appveyor account (like Dvorapa/Pywikibot)

Xqt renamed this task from Travis: Major improvement to test time to Travis and Appveyor: Major improvement to test time.Oct 1 2018, 11:48 AM

Xqt updated the task description. (Show Details)

Change 463735 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [tests] Set fast_finish: true

https://gerrit.wikimedia.org/r/463735

gerritbot added a project: Patch-For-Review.Oct 1 2018, 11:48 AM

Change 463735 merged by jenkins-bot:
[pywikibot/core@master] [tests] Set fast_finish: true

https://gerrit.wikimedia.org/r/463735

Framawiki closed subtask T191632: l10n_tests should run only once for every committed patch as Resolved.Nov 17 2018, 10:30 AM

Xqt reopened subtask T191632: l10n_tests should run only once for every committed patch as Open.Nov 18 2018, 5:12 PM

xSavitar moved this task from Backlog to Needs Review on the Pywikibot board.Dec 22 2018, 6:18 PM

Xqt removed a project: Patch-For-Review.Feb 3 2019, 3:33 PM

Xqt moved this task from Needs Review to Backlog on the Pywikibot board.

Xqt added a subtask: T213287: Drop support of Python 2.7.Feb 18 2019, 4:16 PM

We need a lot of time for testing Python 2.7 which is 14 minutes at Travis and 52 minutes at Appveyor in total

@Ladsgroup: could you please give access to Appveyer project to @Dvorapa or me to allow settings to be changed.

Change 567337 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [tests] decrease testing time

https://gerrit.wikimedia.org/r/567337

gerritbot added a project: Patch-For-Review.Jan 26 2020, 1:09 PM

Xqt added a project: User-Ladsgroup.Jan 26 2020, 1:22 PM

@Xqt How about https://ci.appveyor.com/project/pywikibot/pywikibot ? (you should be able to access it using your GitHub account)

(also sorry for the noise, I changed e-mail address for Pywikibot-oauth from jayvdb@gmail.com to tools.pywikibot@tools.wmflabs.org)

In T186323#5836198, @Dvorapa wrote:

@Xqt How about https://ci.appveyor.com/project/pywikibot/pywikibot ? (you should be able to access it using your GitHub account)

Sure. That was my idea after support request wasn’t complied after several attempts.

In T186323#5836998, @Xqt wrote:

In T186323#5836198, @Dvorapa wrote:

@Xqt How about https://ci.appveyor.com/project/pywikibot/pywikibot ? (you should be able to access it using your GitHub account)

Sure. That was my idea after support request wasn’t complied after several attempts.

Sorry for that, I forgot to communicate that I tried it several times before but couldn't find the settings, it's really hard to find. Apparently (and after googling a couple of times) I need to do some black magic to get @Dvorapa there. @Dvorapa Can you give me your email address so I add you?

dvorapa~seznam~cz

In T186323#5838050, @Dvorapa wrote:

dvorapa~seznam~cz

I sent an invitation for you, let me if it works.

Works, thank you

Change 567337 abandoned by Xqt:
[tests] decrease testing time

Reason:
After rolling builds was set

https://gerrit.wikimedia.org/r/567337

Maintenance_bot removed a project: Patch-For-Review.Jan 30 2020, 6:10 PM

Xqt closed this task as Resolved.Jan 30 2020, 6:14 PM

Xqt claimed this task.

Maintenance_bot moved this task from Incoming to Done on the User-Ladsgroup board.Jan 30 2020, 6:15 PM

@Xqt really? Last build AppVeyor 6 hours, Travis 7 hours

Dvorapa reopened this task as Open.Feb 1 2020, 4:26 PM

In T186323#5840773, @Dvorapa wrote:

@Xqt really? Last build AppVeyor 6 hours, Travis 7 hours

Travis needed two hours in total because 5 tests are running simultaneously (see my comment dated Sep 27 2018, 01:48). Auto cancel is activated.
Appveyor Needs 3,5 - 7 hours per test but test builds does not block new test anymore because rolling builds is activated.

After dropping Python 2, Travis will be 20 minutes faster and Appveyer will be 1 hour faster.

This is ok for me currently. Do you have other ideas to improve tests time?

Above we discussed caching, tests performance issues, currently also some maxlag and login issues making tests much more longer or even timeouting after 50min

I am also fine with keeping this open. Caching was removed several months ago because there where more problems with it than it saved test time. Other issues like timeout and login problems have their own task. Probably we can make them subtasks.

In T186323#5843113, @Xqt wrote:

Probably we can make them subtasks.

Yes, that would be helpful. To address the major issues and list them as subtasks.

Xqt added a subtask: T244082: tests.collector does ignore disabled_test_modules in some cases.Feb 2 2020, 10:37 AM

Xqt added a subtask: T242081: Pywikibot fails to access Wikidata due to high maxlag lately.Feb 2 2020, 10:43 AM

Xqt added a subtask: T246538: http.fake_user_agent() sometimes gives the same result.Feb 29 2020, 4:51 PM

Change 569314 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [cleanup] Ignore L10N tests for pywikibot-core repository

https://gerrit.wikimedia.org/r/569314

gerritbot added a project: Patch-For-Review.Mar 7 2020, 9:13 AM

Change 569314 merged by jenkins-bot:
[pywikibot/core@master] [cleanup] Ignore L10N tests for pywikibot-core repository

https://gerrit.wikimedia.org/r/569314

Maintenance_bot removed a project: Patch-For-Review.Mar 7 2020, 10:10 AM

Xqt mentioned this in rPWBCf1e4b60b1656: [cleanup] Ignore L10N tests for pywikibot-core repository.Mar 7 2020, 10:58 AM

Dvorapa removed a subtask: T246538: http.fake_user_agent() sometimes gives the same result.Mar 20 2020, 5:01 PM

Xqt closed subtask T213287: Drop support of Python 2.7 as Resolved.Aug 4 2020, 11:29 AM

Ladsgroup closed subtask T242081: Pywikibot fails to access Wikidata due to high maxlag lately as Resolved.Oct 21 2020, 5:17 PM

Ladsgroup moved this task from Done to Monitor on the User-Ladsgroup board.May 20 2021, 10:54 AM

Xqt closed subtask T191632: l10n_tests should run only once for every committed patch as Declined.Jul 12 2021, 10:53 AM

Change 735657 had a related patch set uploaded (by Xqt; author: Xqt):