Page MenuHomePhabricator
Create Task
Maniphest T186323

Travis and Appveyor: Major improvement to test time
Closed, ResolvedPublic
Actions

Description

Travis and Appveyor builds take cca 4 hours. Let's find out, how we could make them faster!

What Travis suggests:

Details

SubjectRepoBranchLines +/-
[tests] always use TestTimerMixin with TestCaseBasepywikibot/coremaster+25 -27
[tests] Improvement to test time at appveyorpywikibot/coremaster+4 -20
[cleanup] Ignore L10N tests for pywikibot-core repositorypywikibot/coremaster+2 -1
[tests] decrease testing timepywikibot/coremaster+15 -15
[tests] Set fast_finish: truepywikibot/coremaster+1 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Dvorapa created this task.Feb 2 2018, 2:01 PM
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptFeb 2 2018, 2:01 PM
Dvorapa added a subtask: T106070: Travis python package caching.Feb 2 2018, 2:04 PM
Dvorapa added a parent task: T152961: Improve tests suite.
Dalba moved this task from Backlog to Framework on the Pywikibot-tests board.Apr 5 2018, 1:40 PM
Xqt triaged this task as Low priority.Apr 5 2018, 4:17 PM
Xqt added a subtask: T191632: l10n_tests should run only once for every committed patch.Apr 6 2018, 1:40 PM
Xqt closed subtask T106070: Travis python package caching as Declined.Apr 14 2018, 7:07 PM
Dalba added a comment.EditedSep 25 2018, 8:50 AM

Here is an idea (although I don't know how much it can help):
When a commit only affects a script, there is no need to run all tests, just the related test module for that script. The other parts have already been tested in previous commits. This should be safe because the pywikibot directory does not depend on scripts, and scripts are (should be) independent from each-other.

Dvorapa added a comment.Sep 25 2018, 9:40 AM

It seems ok to me, per T60942 no library file calls a script, but scripts call others sometimes.

Xqt added a comment.Sep 26 2018, 6:14 AM

Good idea but I've no glue how to set up tests to only run a subset of it with scripts changes

Xqt raised the priority of this task from Low to Medium.Sep 26 2018, 12:35 PM
Xqt added a comment.Sep 26 2018, 11:48 PM

I changed the Travis settings a bit to enable more concurrent jobs and cancel jobs when a new patch was deployed; this will shorten tests time a lot. I also cancelled some jobs but they can be restarted if necessary. Unfortunately I cannot do such administrative things on apveyor.

Xqt added a comment.Sep 28 2018, 5:04 AM

As result of the settings changes one test in Travis needs one hour now vs. two hours before.

Dvorapa added subscribers: jayvdb, Ladsgroup.Sep 28 2018, 5:47 AM

Excellent!

PS: @Ladsgroup has got an access to the current Appveyor account, @jayvdb has got an access to the Wikimedia Appveyor account. To solve the issue, there are two options:

  • contact one of the guys successfully
  • create new Appveyor account (like Dvorapa/Pywikibot)
Xqt renamed this task from Travis: Major improvement to test time to Travis and Appveyor: Major improvement to test time.Oct 1 2018, 11:48 AM
Xqt updated the task description. (Show Details)
gerritbot subscribed.Oct 1 2018, 11:48 AM

Change 463735 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [tests] Set fast_finish: true

https://gerrit.wikimedia.org/r/463735

gerritbot added a project: Patch-For-Review.Oct 1 2018, 11:48 AM
gerritbot added a comment.Oct 3 2018, 8:34 AM

Change 463735 merged by jenkins-bot:
[pywikibot/core@master] [tests] Set fast_finish: true

https://gerrit.wikimedia.org/r/463735

Framawiki closed subtask T191632: l10n_tests should run only once for every committed patch as Resolved.Nov 17 2018, 10:30 AM
Xqt reopened subtask T191632: l10n_tests should run only once for every committed patch as Open.Nov 18 2018, 5:12 PM
xSavitar moved this task from Backlog to Needs Review on the Pywikibot board.Dec 22 2018, 6:18 PM
Xqt removed a project: Patch-For-Review.Feb 3 2019, 3:33 PM
Xqt moved this task from Needs Review to Backlog on the Pywikibot board.
Xqt added a subtask: T213287: Drop support of Python 2.7.Feb 18 2019, 4:16 PM
Xqt added a comment.EditedFeb 18 2019, 4:24 PM

We need a lot of time for testing Python 2.7 which is 14 minutes at Travis and 52 minutes at Appveyor in total

Xqt added a comment.Jan 26 2020, 12:58 PM

@Ladsgroup: could you please give access to Appveyer project to @Dvorapa or me to allow settings to be changed.

gerritbot added a comment.Jan 26 2020, 1:09 PM

Change 567337 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [tests] decrease testing time

https://gerrit.wikimedia.org/r/567337

gerritbot added a project: Patch-For-Review.Jan 26 2020, 1:09 PM
Xqt added a project: User-Ladsgroup.Jan 26 2020, 1:22 PM
Dvorapa added a comment.EditedJan 28 2020, 12:50 PM

@Xqt How about https://ci.appveyor.com/project/pywikibot/pywikibot ? (you should be able to access it using your GitHub account)

(also sorry for the noise, I changed e-mail address for Pywikibot-oauth from jayvdb@gmail.com to tools.pywikibot@tools.wmflabs.org)

Xqt added a comment.Jan 28 2020, 7:19 PM
In T186323#5836198, @Dvorapa wrote:

@Xqt How about https://ci.appveyor.com/project/pywikibot/pywikibot ? (you should be able to access it using your GitHub account)

Sure. That was my idea after support request wasn’t complied after several attempts.

Ladsgroup added a comment.Jan 29 2020, 12:38 AM
In T186323#5836998, @Xqt wrote:
In T186323#5836198, @Dvorapa wrote:

@Xqt How about https://ci.appveyor.com/project/pywikibot/pywikibot ? (you should be able to access it using your GitHub account)

Sure. That was my idea after support request wasn’t complied after several attempts.

Sorry for that, I forgot to communicate that I tried it several times before but couldn't find the settings, it's really hard to find. Apparently (and after googling a couple of times) I need to do some black magic to get @Dvorapa there. @Dvorapa Can you give me your email address so I add you?

Dvorapa added a comment.Jan 29 2020, 12:00 PM

dvorapa~seznam~cz

Ladsgroup added a comment.Jan 29 2020, 12:08 PM
In T186323#5838050, @Dvorapa wrote:

dvorapa~seznam~cz

I sent an invitation for you, let me if it works.

Dvorapa added a comment.Jan 29 2020, 12:22 PM

Works, thank you

gerritbot added a comment.Jan 30 2020, 6:05 PM

Change 567337 abandoned by Xqt:
[tests] decrease testing time

Reason:
After rolling builds was set

https://gerrit.wikimedia.org/r/567337

Maintenance_bot removed a project: Patch-For-Review.Jan 30 2020, 6:10 PM
Xqt closed this task as Resolved.Jan 30 2020, 6:14 PM
Xqt claimed this task.
Maintenance_bot moved this task from Incoming to Done on the User-Ladsgroup board.Jan 30 2020, 6:15 PM
Dvorapa added a comment.Jan 30 2020, 9:17 PM

@Xqt really? Last build AppVeyor 6 hours, Travis 7 hours

Dvorapa reopened this task as Open.Feb 1 2020, 4:26 PM
Xqt added a comment.Feb 1 2020, 5:04 PM
In T186323#5840773, @Dvorapa wrote:

@Xqt really? Last build AppVeyor 6 hours, Travis 7 hours

Travis needed two hours in total because 5 tests are running simultaneously (see my comment dated Sep 27 2018, 01:48). Auto cancel is activated.
Appveyor Needs 3,5 - 7 hours per test but test builds does not block new test anymore because rolling builds is activated.

After dropping Python 2, Travis will be 20 minutes faster and Appveyer will be 1 hour faster.

This is ok for me currently. Do you have other ideas to improve tests time?

Dvorapa added a comment.EditedFeb 1 2020, 6:01 PM

Above we discussed caching, tests performance issues, currently also some maxlag and login issues making tests much more longer or even timeouting after 50min

Xqt removed Xqt as the assignee of this task.Feb 1 2020, 7:04 PM

I am also fine with keeping this open. Caching was removed several months ago because there where more problems with it than it saved test time. Other issues like timeout and login problems have their own task. Probably we can make them subtasks.

Dvorapa added a comment.Feb 1 2020, 8:46 PM
In T186323#5843113, @Xqt wrote:

Probably we can make them subtasks.

Yes, that would be helpful. To address the major issues and list them as subtasks.

Xqt added a subtask: T244082: tests.collector does ignore disabled_test_modules in some cases.Feb 2 2020, 10:37 AM
Xqt added a subtask: T242081: Pywikibot fails to access Wikidata due to high maxlag lately.Feb 2 2020, 10:43 AM
Xqt added a subtask: T246538: http.fake_user_agent() sometimes gives the same result.Feb 29 2020, 4:51 PM
gerritbot added a comment.Mar 7 2020, 9:13 AM

Change 569314 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [cleanup] Ignore L10N tests for pywikibot-core repository

https://gerrit.wikimedia.org/r/569314

gerritbot added a project: Patch-For-Review.Mar 7 2020, 9:13 AM
gerritbot added a comment.Mar 7 2020, 9:22 AM

Change 569314 merged by jenkins-bot:
[pywikibot/core@master] [cleanup] Ignore L10N tests for pywikibot-core repository

https://gerrit.wikimedia.org/r/569314

Maintenance_bot removed a project: Patch-For-Review.Mar 7 2020, 10:10 AM
Xqt mentioned this in rPWBCf1e4b60b1656: [cleanup] Ignore L10N tests for pywikibot-core repository.Mar 7 2020, 10:58 AM
Dvorapa removed a subtask: T246538: http.fake_user_agent() sometimes gives the same result.Mar 20 2020, 5:01 PM
Xqt closed subtask T213287: Drop support of Python 2.7 as Resolved.Aug 4 2020, 11:29 AM
Ladsgroup closed subtask T242081: Pywikibot fails to access Wikidata due to high maxlag lately as Resolved.Oct 21 2020, 5:17 PM
Ladsgroup moved this task from Done to Monitor on the User-Ladsgroup board.May 20 2021, 10:54 AM
Xqt closed subtask T191632: l10n_tests should run only once for every committed patch as Declined.Jul 12 2021, 10:53 AM
gerritbot added a comment.Oct 29 2021, 2:36 PM

Change 735657 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] [tests] Improvement to test time at appveyor

https://gerrit.wikimedia.org/r/735657

gerritbot added a project: Patch-For-Review.Oct 29 2021, 2:36 PM
gerritbot added a comment.Oct 29 2021, 2:53 PM

Change 735657 merged by jenkins-bot:

[pywikibot/core@master] [tests] Improvement to test time at appveyor

https://gerrit.wikimedia.org/r/735657

Xqt removed a project: Patch-For-Review.Oct 29 2021, 6:14 PM
gerritbot added a comment.Dec 4 2021, 12:08 PM

Change 743567 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] [tests] always use TestTimerMixin with TestCaseBase

https://gerrit.wikimedia.org/r/743567

gerritbot added a project: Patch-For-Review.Dec 4 2021, 12:08 PM
gerritbot added a comment.Dec 4 2021, 12:19 PM

Change 743567 merged by jenkins-bot:

[pywikibot/core@master] [tests] always use TestTimerMixin with TestCaseBase

https://gerrit.wikimedia.org/r/743567

Xqt removed a project: Patch-For-Review.Dec 4 2021, 2:19 PM
Xqt closed this task as Resolved.Dec 8 2021, 9:33 AM
Xqt claimed this task.

Moving from Travis to Github actions improves test times a lot. Github needs less then one hour whereas travis needs 5 hours. Closing then.

Xqt removed a subtask: T213287: Drop support of Python 2.7.May 4 2023, 3:27 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL