The https://gerrit.wikimedia.org/r/197089 patch could try to shed some light into the reason why it's happening.

Interestingly I've only found failures on French builds but not always. I also can't say when it first occurred.

Very odd. It is also only wiktionary , so maybe there is some title first-letter case problems?

Ever since I was providing a patch to debug that problem I didn't encounter it and now the official builds on Travis are fine to. Maybe it's at a certain time when this happens.

Okay the first failure with the debug output and it's lists a not existing page. It starts with a lowercase letter which could be also the cause. A second failure could help narrow it down. In both requests to the API the page appears so it doesn't seem to be because of MediaWiki.

Regarding case sensitivity: The siteinfo reports that the site is case sensitive as well as the namespace. And I don't know how it can “forget” the case sensitivity. In Link.parse() it is checked at all times and if the namespace is not valid it'll crash there.

https://gerrit.wikimedia.org/r/198172 provides a second debug patch which shows the XOR difference (may only of all pages which are in ns 0 and 1). That way we could verify whether the name is capitalized in the other set.

Okay another failure and again a lowercase page which does not exist. I've run https://travis-ci.org/xZise/pywikibot-core/builds/55116261 multiple times which has https://gerrit.wikimedia.org/r/198172 merged in but it hasn't failed yet.

Change 202618 had a related patch set uploaded (by John Vandenberg):
Change build sites to fr.wikipedia and ar.wiktionary

https://gerrit.wikimedia.org/r/202618

Change 202618 merged by jenkins-bot:
Change build sites to fr.wikipedia and ar.wiktionary

https://gerrit.wikimedia.org/r/202618

Okay with Wikivoyage it seems it's not related to Wiktionary, the case sensitivity setting and the language French. The result itself is interesting because the symmetric difference contains two pages in namespace 0 while the check afterwards only mentions one page and in theory it should mention all pages of the symmetric difference which were in the namespaces 0 and 1. There seems to be no link to a talk page which has “שיחה” as a prefix (or the other way around).

It seems to only occur when the site tests are run under SITE_ONLY. fr.wikt has not re-appeared since it was moved to be a full test build.

Working on: https://travis-ci.org/wikimedia/pywikibot-core/jobs/177817110#L6394-L6467

Symmetric diff:

Template:@POTD/2016-11-19",
:@1986",
:@Star Trek: The Original Series",
:@Rosetta (spacecraft)",
:@Jean-Patrick Lescarboura",
:@Gettysburg Address",
Template talk:@Did you know",
:@Professional wrestling",
:@Paramount Pictures",
:@Streitraum",
Template:@POTD protected/2016-11-22",
:@1386",
:@La Máscara (wrestler)",
:@List of Star Trek Starfleet starships ordered by class",
:@New York",
:@University of Oxford",
:@Colin Mawby",
:@Carolin Emcke",
:@List of Six Nations Championship hat-tricks",
:@George Naea",
:@First Sino-Japanese War",
:@Star Trek: The Next Generation",
:@Reform Act 1832",
:@Scotland national rugby union team",
:@Congress of Manastir",
Project:@News",
Portal:@Society",
:@Resist",
:@Union Army",
:@Abraham Lincoln",
:@William S. Hart",
:@1996",
:@1928",
:@Humberto Vidal explosion",
:@Prime Minister of the United Kingdom",
:@Gilberto Gil",
:@Lüshunkou District",
:@Saturn Awards",
:@1830",
:@Ronald D. Moore",
Portal:@Technology",
:@Eugen Eckert",
:@Marc Bassingthwaighte",
:@Proceratium petrosum",
:@European Space Agency",
:@Lebanon",
:@Star Trek: First Contact",
:@George Campbell Lindsay",
:@Resist (semiconductor fabrication)",
:@Svetlana Gerasimenko",
:@John Hay",
:@Abraham Lincoln's second inaugural address",
:@Olearia ramulosa",
:@November 23",
Project:@Today's featured article/November 2016",
Portal:@Biography",
:@Powderfinger discography",
:@Industrial Light & Magic",
:@Robert W. Jones",
:@Ireland national rugby union team",
:@Palais Garnier",
:@Spanish language",
Project:@Today's featured list/November 2016",
Project:@Introduction",
:@Borg (Star Trek)",
:@Battle of Lushunkou",
:@Timur",
Project:@Featured articles",
:@Cyborg",
:@Bagrat V of Georgia",
:@Jin Chae-seon",
:@Kennedy St-Pierre",
:@Queen Emma of Hawaii",
:@Olearia viscidula",
:@Taiwanese aborigines",
:@Namibian National Road Race Championships",
:@1908",
Portal:@Mathematics",
:@Dutch Formosa",
:@Pansori",
:@The Vine (bus rapid transit)",
Portal:@Arts",
:@Jdeideh",
:@Vaile Mansion",
:@Confederate States of America",
:@American Civil War",
:@Academy Award for Best Makeup",
:@Wales national rugby union team",
Project:@Community portal",
Project:@Reference desk",
:@Cybergeddon (film)",
:@Star Trek Generations",
:@Maurice Ravel",
:@Star Trek (film series)",
:@Try",
Portal:@Current events",
:@World Hello Day",
:@Dutch pacification campaign on Formosa",
:@San Juan, Puerto Rico",
:@Brazilian battleship São Paulo",
:@Olearia tomentosa",
:@John Diefenbaker",
:@Science fiction film",
:@Ángel de Oro",
:@African Games",
:@Brazilian cruiser Bahia",
:@Valemount, British Columbia",
:@The class the stars fell on",
:@Mike Tyson",
Template:@POTD/2016-11-20",
:@Professional wrestling weight classes",
:@Port Arthur massacre (China)",
Project:@Your first article",
:@White House",
Portal:@Science",
Project:@Featured lists",
:@Computer-generated imagery",
:@Revolt of the Lash",
:@67P/Churyumov–Gerasimenko",
:@Wild Bill Hickok (film)",
:@England national rugby union team",
Project:@Help desk",
:@Wyatt Earp",
:@Río Piedras, Puerto Rico",
:@Jean-Luc Picard",
:@Neues Geistliches Lied",
:@South American dreadnought race",
:@Professional wrestling championship",
:@Massachusetts",
:@List of heavyweight boxing champions",
:@Orbital period",
:@Klim Churyumov",
Project:@Recent additions",
:@Hat-trick",
:@President of the United States",
Project:@Village pump",
Portal:@History",
:@Ahab (Moby-Dick)",
:@Mauritius",
:@Portland metropolitan area",
:@Drop goal",
:@Black American Sign Language",
:@1894",
Project:@Featured pictures",
:@Brazilian Navy",
:@Herman Zimmerman",
:@Pierre Albaladejo",
:@Grammy Award",
:@Consejo Mundial de Lucha Libre",
Project:@Today's featured list/November 21, 2016",
:@Resist dyeing",
:@Quanta Live",
:@Rachel Freier",
:@Boston",
:@Professional wrestling promotion",
:@Neil Jenkins",
:@Pierre Amine Gemayel",
:@Dr. J. Butz",
:@Comet",
:@Olearia",
:@Rugby union",
Project:@Picture of the day/November 2016",
:@Jupiter-family comet",
Portal:@Geography",
:@Turco-Mongol",
:@Commerzbank-Arena",
:@France national rugby union team",
:@Tbilisi",
:@1910",
:@Naming of comets",
:@Diego Domínguez (rugby union)",
:@Independence, Missouri",
:@2006",
Project:@Selected anniversaries/November",
:@Canoe River train crash",
Template:@POTD/2016-11-18",
:@Timur's invasions of Georgia",
:@Trevor Berbick",
:@Kuiper belt",
:@Herman Melville",
:File:@LaMascaraLuchador.jpg",
:@1635",
:@Olearia decurrens",
:@1950",
:@Ian Smith (Scottish rugby player born 1903)",
:@Boléro",
:@Earl Grey tea",
:@1943",
Template:@POTD protected/2016-11-21",
:@Brannon Braga",
:@Charles Grey, 2nd Earl Grey",
:@Six Nations Championship",
Portal:@Contents/Portals",
:@Christopher Tambling",
:@Jonathan Frakes",
Project:@Local Embassy",
:@List of lakes in Minneapolis",
:@World Boxing Council",
:@Web series",
:@Brazilian battleship Minas Geraes",
:@Bus rapid transit",
:@November 20",
:@Prime Minister of Canada",
:@Photographic plate",
:@Hasidic Judaism",
:@John Eaves",
:@Leprosy",
:@Bixby letter",
:@Charles Wade",
:@List of national independence days"

Working still on https://travis-ci.org/wikimedia/pywikibot-core/jobs/177817110#L6394-L6467

It occurs during this particular job there were some edits in templates included in Main Page:

• https://en.wikipedia.org/w/index.php?title=Template%3ADid_you_know%2FQueue%2FNext&type=revision&diff=750874261&oldid=750637244
• https://en.wikipedia.org/w/index.php?title=Template%3ADid_you_know%2FQueue%2F2&type=revision&diff=750878092&oldid=749946911
• https://en.wikipedia.org/w/index.php?title=Template%3ADid_you_know&type=revision&diff=750873841&oldid=750636654

It will require further investigation but it looks like correct path to debug this test.

The reason that (I believe) cause failing tests here (and in few other tasks) is, that these tests aren't unittests. The work on live Mediawiki instances and they assume that there will be no change during test.

As You can see in my previous comment it sometimes happend that change is made and it is failing.

I was trying to build here bullet proof solution. The solution mock BasePage.init method and decorate it logic with 2 lines that were responsible for getting last rev id for every page that was created during test and if it failed it would check if any page that was created during test changed. The problem here is that time required to run one such test was increased by at least one order of magnitude. Previously test take 3-4 seconds to pass and after implementing bullet-proof solution it was going for few minutes and haven't end.

So the question is what level of fragility is enough for us? I'm tending right now into:

• • choping long tests that cuse problems to smaller ones
• on failure reruning test (that it will take 10 failures before test fails completely).

What do You think about it?

@Dalba @Xqt @jayvdb @valhallasw @XZise @XXN

Change 327580 had a related patch set uploaded (by Magul):
Rerun site links methods tests on failure

https://gerrit.wikimedia.org/r/327580

@Magul One minor comment on the patch. Otherwise it looks good to go.

This happened again in the most recent build:

https://travis-ci.org/wikimedia/pywikibot-core/jobs/237516255

        self.assertCountEqual(

>           set(mysite.pagelinks(mainpage, namespaces=[0, 1])) - links, [])

E       AssertionError: Element counts were not equal:

E       First has 1, Second has 0:  Page('Manuel Noriega')

Apparently, the link has been added while the test has been running.

I have another solution that is simple to implement and may even be considered more trustworthy and efficient than just retrying the test suite:

We can re-fetch the set of linked pages and compare it with the original set, if the set has changed we may conclude that the page content has changed and the AssertionError is safe to be ignored.

I'll send a patch for review soon ...

Change 356835 had a related patch set uploaded (by Dalba; owner: Dalba):
[pywikibot/core@master] site_tests.testLinkMethods: Ignore errors caused by page content change

https://gerrit.wikimedia.org/r/356835

Change 357195 had a related patch set uploaded (by Merlijn van Deen; owner: Merlijn van Deen):
[pywikibot/core@master] Disable request caching on Travis

https://gerrit.wikimedia.org/r/357195

Change 376898 had a related patch set uploaded (by Dalba; owner: Dalba):
[pywikibot/core@master] site_tests.py: Used a freezed API response to avoid race conditions

https://gerrit.wikimedia.org/r/376898

Change 377047 had a related patch set uploaded (by Dalba; owner: Dalba):
[pywikibot/core@master] site_tests.py: Avoid race conditions in test_pagetemplates and test_pagelinks

https://gerrit.wikimedia.org/r/377047

Change 376898 abandoned by Dalba:
site_tests.py: Use a freezed API response to avoid race conditions

Reason:
In favour of https://gerrit.wikimedia.org/r/#/c/377047

https://gerrit.wikimedia.org/r/376898

Change 377047 merged by jenkins-bot:
[pywikibot/core@master] site_tests.py: Avoid race conditions in test_pagetemplates and test_pagelinks

https://gerrit.wikimedia.org/r/377047

Change 356835 abandoned by Dalba:
site_tests.py: Handle random errors caused by page content change

Reason:
Change 377047 was merged as an alternative.

https://gerrit.wikimedia.org/r/356835

Change 357195 abandoned by Dalba:
Disable request caching on Travis

Reason:
I guess this change can be abandoned now that 377047 has been merged.

https://gerrit.wikimedia.org/r/357195

Change 327580 abandoned by Dalba:
Rerun site links methods tests on failure

Reason:
I guess this change can be abandoned now that 377047 has been merged.

https://gerrit.wikimedia.org/r/327580