Page MenuHomePhabricator

test_allpages_langlinks_enabled is failing on Travis
Closed, ResolvedPublic

Description

______________ TestSiteGenerators.test_allpages_langlinks_enabled ______________
self = <tests.site_tests.TestSiteGenerators testMethod=test_allpages_langlinks_enabled>
    def test_allpages_langlinks_enabled(self):
        """Test allpages with langlinks enabled."""
        mysite = self.get_site()
        for page in mysite.allpages(filterlanglinks=True, total=5):
            self.assertIsInstance(page, pywikibot.Page)
            self.assertTrue(mysite.page_exists(page))
            self.assertEqual(page.namespace(), 0)
>           self.assertNotEqual(page.langlinks(), [])
tests/site_tests.py:598: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pywikibot/page.py:1440: in langlinks
    self._langlinks = list(self.iterlanglinks(include_obsolete=True))
pywikibot/site.py:3967: in pagelanglinks
    source=self)
pywikibot/page.py:5380: in langlinkUnsafe
    link._site = pywikibot.Site(lang, source.family.name)
pywikibot/__init__.py:850: in Site
    _sites[key] = interface(code=code, fam=fam, user=user, sysop=sysop)
pywikibot/site.py:1855: in __init__
    BaseSite.__init__(self, code, fam, user, sysop)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = APISite("it", "wpbeta"), code = 'it', fam = Family("wpbeta"), user = None
sysop = None
    def __init__(self, code, fam=None, user=None, sysop=None):
        """
            Constructor.
    
            @param code: the site's language code
            @type code: str
            @param fam: wiki family name (optional)
            @type fam: str or Family
            @param user: bot user name (optional)
            @type user: str
            @param sysop: sysop account user name (optional)
            @type sysop: str
            """
        if code.lower() != code:
            # Note the Site function in __init__ also emits a UserWarning
            # for this condition, showing the callers file and line no.
            pywikibot.log(u'BaseSite: code "%s" converted to lowercase' % code)
            code = code.lower()
        if not all(x in pywikibot.family.CODE_CHARACTERS for x in str(code)):
            pywikibot.log(u'BaseSite: code "%s" contains invalid characters'
                          % code)
        self.__code = code
        if isinstance(fam, basestring) or fam is None:
            self.__family = pywikibot.family.Family.load(fam)
        else:
            self.__family = fam
    
        self.obsolete = False
        # if we got an outdated language code, use the new one instead.
        if self.__code in self.__family.obsolete:
            if self.__family.obsolete[self.__code] is not None:
                self.__code = self.__family.obsolete[self.__code]
                # Note the Site function in __init__ emits a UserWarning
                # for this condition, showing the callers file and line no.
                pywikibot.log(u'Site %s instantiated using code %s'
                              % (self, code))
            else:
                # no such language anymore
                self.obsolete = True
                pywikibot.log(u'Site %s instantiated and marked "obsolete" '
                              u'to prevent access' % self)
        elif self.__code not in self.languages():
            if self.__family.name in list(self.__family.langs.keys()) and \
               len(self.__family.langs) == 1:
                self.__code = self.__family.name
                if self.__family == pywikibot.config.family \
                        and code == pywikibot.config.mylang:
                    pywikibot.config.mylang = self.__code
                    warn(u'Global configuration variable "mylang" changed to '
                         u'"%s" while instantiating site %s'
                         % (self.__code, self), UserWarning)
            else:
                raise UnknownSite(u"Language '%s' does not exist in family %s"
>                                 % (self.__code, self.__family.name))
E               UnknownSite: Language 'it' does not exist in family wpbeta
pywikibot/site.py:773: UnknownSite

See: https://travis-ci.org/wikimedia/pywikibot-core/jobs/177955830#L1613-L1697

Event Timeline

Change 323098 had a related patch set uploaded (by Magul):
Fix for generating WMF Labs families.

https://gerrit.wikimedia.org/r/323098

@jayvdb are You able what's wrong with Site.from_url (as You mentioned here).

@Xqt maybe You know, what @jayvdb had on mind? What should we do here?

I can't reproduce the issue.

Is this still failing on any of the latest travis builds? (Why is it marked as passed here?)

@Dalba it is still failing. It occurs only on sites, that doesn't have interwiki links configured in family file. To reproduce it You need to generate family file (without interwiki links - the way, that our master is doint it right now on Travis, see: .travis.yml).

So problem occurs still, particularly in job #13 on every build. See lat 5 builds: #3753, #3752, #3751, #3750 and #3749.

Actually I sent yesterday all our gerrit patches to Travis, so there is waiting pull request with that change and You could check if it changes something on Travis build. It will take few days/weeks for Travis to swallow whole queue, that I generated yestarday, but eventually we should get job #13 passing on that test.

Change 323098 merged by jenkins-bot:
Fix generating WMF Labs families

https://gerrit.wikimedia.org/r/323098

A related question (which might already be in a separate task, if so appologies) does this now also correctly identify beta.wikidata as the wikibase repo associated with the beta Wikipedias?

@Lokal_Profil this change was limited only to resolution of failing test. It change the way we generate interwiki links for newly generated family. It looks, that new family file doesn't contain any wikidata information.

My wpbeta family looks like:


Moreover data_repository method is trying to figure out wikidata url on fly. It is trying to use here

WARNING: Site "wpbeta:en" supports wikibase at "https://wikidata.beta.wmflabs.org/w/index.php", but creation failed: Unknown URL 'https://wikidata.beta.wmflabs.org/w/index.php'..

And if You will try, You will see that https://wikidata.beta.wmflabs.org/w/index.php redirect to https://wikidata.beta.wmflabs.org/wiki/Wikidata:Main_Page (simple 301 http redirect).

I'm not a mediawiki API expert, but it looks, that there's something wrong. Nevertheless feel free to report separate ticket here.

@Lokal_Profil this change was limited only to resolution of failing test. It change the way we generate interwiki links for newly generated family. It looks, that new family file doesn't contain any wikidata information.

My wpbeta family looks like:


Moreover data_repository method is trying to figure out wikidata url on fly. It is trying to use here

WARNING: Site "wpbeta:en" supports wikibase at "https://wikidata.beta.wmflabs.org/w/index.php", but creation failed: Unknown URL 'https://wikidata.beta.wmflabs.org/w/index.php'..

That is due to pywikibot not finding any family file containing that url.
I've set created a wdbeta.py family like: P4657 which it correctly identifies. Anyhow. thanks your answer answered my question.