Page MenuHomePhabricator

TestSiteGenerators.test_pagetemplates is sometimes failing on Travis CI
Closed, ResolvedPublic

Description

=================================== FAILURES ===================================
____________________ TestSiteGenerators.test_pagetemplates _____________________

self = <tests.site_tests.TestSiteGenerators testMethod=test_pagetemplates>

    def test_pagetemplates(self):
        """Test Site.pagetemplates."""
        pagetemplates_all = set(self.site.pagetemplates(self.mainpage))
        pagetemplates_ns_10 = set(self.site.pagetemplates(self.mainpage, namespaces=[10]))
    
        for te in pagetemplates_all:
            self.assertIsInstance(te, pywikibot.Page)
    
>       self.assertLessEqual(pagetemplates_ns_10, pagetemplates_all)
       AssertionError: set([Page(Template:Wikipedia languages/core), Page(Template:Did you know/Queue/Next), Page(Template:SelAnnivFooter), Page(Template:Main Page interwikis), Page(Template:Wikipedia's sister projects), Page(Template:TFArecentlist), Page(Template:In the news), Page(Template:Wikipedia languages), Page(Template:Remove file prefix), Page(Template:Nowrap), Page(Template:Str number/trim), Page(Template:Main page image), Page(Template:DYKbotdo), Page(Template:Other areas of Wikipedia), Page(Template:POTD/2017-07-01/5), Page(Template:POTD texttitle), Page(Template:Did you know/Queue/4), Page(Template:TFAfooter), Page(Template:TFAFULL), Page(Template:In the news/image), Page(Template:POTD protected/2017-07-04), Page(Template:POTD protected/2017-07-05), Page(Template:Str endswith), Page(Template:Main Page banner), Page(Template:Mbox), Page(Template:In the news/footer), Page(Template:Did you know), Page(Template:Start date), Page(Template:If empty), Page(Template:·)]) not less than or equal to set([Page(Template:Remove file prefix), Page(Template:Start date), Page(Module:Category handler/config), Page(Module:Namespace detect/config), Page(Template:SelAnnivFooter), Page(Module:TableTools), Page(Template:Main Page interwikis), Page(Template:Wikipedia's sister projects), Page(Wikipedia:Selected anniversaries/July 5), Page(Template:TFArecentlist), Page(Template:In the news), Page(Module:Category handler/data), Page(Template:POTD/2017-07-01/2), Page(Module:Str endswith), Page(Module:Random), Page(Template:Wikipedia languages), Page(Template:Mbox), Page(Template:Did you know), Page(Module:Category handler), Page(Wikipedia:Selected anniversaries/July 4), Page(Template:Str number/trim), Page(Template:Main page image), Page(Template:DYKbotdo), Page(Template:Other areas of Wikipedia), Page(Module:List), Page(Wikipedia:Today's featured article/July 4, 2017), Page(Template:POTD texttitle), Page(Template:Wikipedia languages/core), Page(Template:Did you know/Queue/4), Page(Template:TFAfooter), Page(Template:TFAFULL), Page(Module:No globals), Page(Module:Message box), Page(Template:POTD protected/2017-07-04), Page(Wikipedia:Main Page/Tomorrow), Page(Template:POTD protected/2017-07-05), Page(Module:Arguments), Page(Wikipedia:Today's featured article/July 5, 2017), Page(Module:Category handler/blacklist), Page(Module:If empty), Page(Module:Namespace detect/data), Page(Template:Str endswith), Page(Module:Message box/configuration), Page(Module:Yesno), Page(Template:Main Page banner), Page(Template:Did you know/Queue/Next), Page(Template:In the news/footer), Page(Module:Category handler/shared), Page(Template:Nowrap), Page(Module:String), Page(Template:In the news/image), Page(Template:If empty), Page(Template:·)])

tests/site_tests.py:560: AssertionError

Event Timeline

Xqt triaged this task as High priority.Jul 4 2017, 3:08 PM

Seems it has to do with the POTD templates on en-wiki which varies by a time stamp:
pagetemplates_ns_10 has Page(Template:POTD/2017-07-01/5)
whereas pagetemplates_all has Page(Template:POTD/2017-07-01/2).

Ah, this is random generated:
[[ https://en.wikipedia.org/w/index.php?title=Template:POTD/2017-07-01&action=edit | {{POTD/2017-07-01/{{#invoke:random|number|5}}|{{{1|{{{style|default}}}}}}}} ]].

Change 363205 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [bugfix] Don't fail TestSiteGenerators.test_pagetemplates tests

https://gerrit.wikimedia.org/r/363205

After this patch there are few remaining failures but I don't see the point e.g. of
https://travis-ci.org/magul/pywikibot-core/jobs/250057193

>       self.assertLessEqual(pt_ns_10, pt_all)

E       AssertionError: [Page('Template:•'), Page('Template:HomePages'), Page('Template:Edi'), Page('Template:Ill')] not less than or equal to [Page('Wikipedia:What we do on this wiki'), Page('Template:Ill'), Page('Template:HomePages'), Page('Wikipedia:Bureaucrats'), Page('Wikipedia:Administrators'), Page('Template:•'), Page('Wikipedia:What Test Wiki is not'), Page('Template:Edi')]

tests/site_tests.py:564: AssertionError

All items of the first set are in the second.

The code in the failed build does not look like your latest patch, so is this is OK after your latest changes?

I changed the code to find the difference of the assertlessequal test. Unfortunately no Travis ci was buildt after this patch. Anyway it is very strange.

Looks like you are comparing lists when the error occurred, so order is important as well.

The newest patch should work then.

After this patch there are few remaining failures but I don't see the point e.g. of
https://travis-ci.org/magul/pywikibot-core/jobs/250057193

>       self.assertLessEqual(pt_ns_10, pt_all)

E       AssertionError: [Page('Template:•'), Page('Template:HomePages'), Page('Template:Edi'), Page('Template:Ill')] not less than or equal to [Page('Wikipedia:What we do on this wiki'), Page('Template:Ill'), Page('Template:HomePages'), Page('Wikipedia:Bureaucrats'), Page('Wikipedia:Administrators'), Page('Template:•'), Page('Wikipedia:What Test Wiki is not'), Page('Template:Edi')]

tests/site_tests.py:564: AssertionError

All items of the first set are in the second.

I'm in favour of removing it. It's making things unnecessary complicated. The current patch could fail if the random part is part of the initial title segment. It is also vulnerable to race conditions like the one in T92856#3310672.

It might be enough to just check that pagetemplates are all from template namespace.

Change 356835 had a related patch set uploaded (by Dalba; owner: Dalba):
[pywikibot/core@master] site_tests.py: Handle random errors caused by page content change

https://gerrit.wikimedia.org/r/356835

Change 376898 had a related patch set uploaded (by Dalba; owner: Dalba):
[pywikibot/core@master] site_tests.py: Used a freezed API response to avoid race conditions

https://gerrit.wikimedia.org/r/376898

Change 377047 had a related patch set uploaded (by Dalba; owner: Dalba):
[pywikibot/core@master] site_tests.py: Avoid race conditions in test_pagetemplates and test_pagelinks

https://gerrit.wikimedia.org/r/377047

Change 376898 abandoned by Dalba:
site_tests.py: Use a freezed API response to avoid race conditions

Reason:
In favour of https://gerrit.wikimedia.org/r/#/c/377047

https://gerrit.wikimedia.org/r/376898

Change 377047 merged by jenkins-bot:
[pywikibot/core@master] site_tests.py: Avoid race conditions in test_pagetemplates and test_pagelinks

https://gerrit.wikimedia.org/r/377047

Change 356835 abandoned by Dalba:
site_tests.py: Handle random errors caused by page content change

Reason:
Change 377047 was merged as an alternative.

https://gerrit.wikimedia.org/r/356835

Change 363205 abandoned by Dalba:
[bugfix] Don't fail TestSiteGenerators.test_pagetemplates tests

Reason:
Change 377047 was merged as an alternative.

https://gerrit.wikimedia.org/r/363205