Page MenuHomePhabricator
Create Task
Maniphest T331223

site_detect_tests fails on proofwiki when running on github
Open, LowestPublicBUG REPORT
Actions

Description

What happens?:

======================================================================
ERROR: test_standard_version_sites (tests.site_detect_tests.MediaWikiSiteTestCase.test_standard_version_sites) (url='www.proofwiki.org')
Test detection of standard MediaWiki sites.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/pywikibot/pywikibot/tests/site_detect_tests.py", line 112, in test_standard_version_sites
    self.assertSite(url)
  File "/home/runner/work/pywikibot/pywikibot/tests/site_detect_tests.py", line 36, in assertSite
    self.assertIsInstance(MWSite(url), MWSite)
                          ^^^^^^^^^^^
  File "/home/runner/work/pywikibot/pywikibot/pywikibot/site_detect.py", line 82, in __init__
    raise RuntimeError(f'Unsupported url: {self.fromurl}')
RuntimeError: Unsupported url: https://www.proofwiki.org/wiki/

The test does not fail on appveyor of if running locally

Other information (browser name/version, screenshots, etc.):
See also T175584, T162991, T134647

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
[tests] Skip test on proofwiki on github actionpywikibot/coremaster+3 -4
Customize query in gerrit

Related Objects

Event Timeline

Xqt created this task.Mar 5 2023, 9:03 AM
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptMar 5 2023, 9:03 AM
Xqt triaged this task as High priority.Mar 5 2023, 9:04 AM
Xqt moved this task from Backlog to CI test failures on the Pywikibot-tests board.
Xqt renamed this task from .site_detect_tests fails on proofwiki when running on github to site_detect_tests fails on proofwiki when running on github.Mar 5 2023, 10:02 AM
Xqt added a comment.EditedMar 5 2023, 10:48 AM

The response from proofwiki is

<!DOCTYPE html>
<html lang="en-US">
<head>
    <title>Just a moment...</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=Edge">
    <meta name="robots" content="noindex,nofollow">
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <link href="/cdn-cgi/styles/challenges.css" rel="stylesheet">
    

</head>
<body class="no-js">
    <div class="main-wrapper" role="main">
    <div class="main-content">
        <h1 class="zone-name-title h1">
            <img class="heading-favicon" src="/favicon.ico" alt="Icon for www.proofwiki.org"
                 onerror="this.onerror=null;this.parentNode.removeChild(this)">
            www.proofwiki.org
        </h1>
        <h2 class="h2" id="challenge-running">
            Checking if the site connection is secure
        </h2>
        <noscript>
            <div id="challenge-error-title">
                <div class="h2">
                    <span class="icon-wrapper">
                        <div class="heading-icon warning-icon"></div>
                    </span>
                    <span id="challenge-error-text">
                        Enable JavaScript and cookies to continue
                    </span>
                </div>
            </div>
        </noscript>
        <div id="trk_jschal_js" style="display:none;background-image:url('/cdn-cgi/images/trace/managed/nojs/transparent.gif?ray=7a31a8999e77cf55')"></div>
        <div id="challenge-body-text" class="core-msg spacer">
            www.proofwiki.org needs to review the security of your connection before proceeding.
        </div>
        <form id="challenge-form" action="/wiki/?__cf_chl_f_tk=_lMpDYq3NmBWJUBKhol.HWxVxHOgAKiA2wuVcM52kzY-1678011964-0-gaNycGzNCOU" method="POST" enctype="application/x-www-form-urlencoded">
            <input type="hidden" name="md" value="DBFPeJf36aNdNh.o4sAyI_YO3f5BIMzIASCRD_RQQps-1678011964-0-AeWh6ynm0NFwDDQngRWm9jrPmOa0OdQmr80eB3NwOLk_POwl9bj9rcn5i9uUTv1-sUQLxh_M-OfxVZlXjY35abt8u96RnKcfote0yKZOovkVrl7M-QMurjQP9lKtpCtSTGahIGXLOCAf5XX95vhZ64D3l2hycj9n5mlQPYfk_IIzTRYbhx5ER5W0bRTP7z8_lZF4jFJ9HtJ52GFszxvsUGcdP0gn10fkeAti5hd4sDduFtkoefld4COu-LmUJCZZQQCce0ym-7vhxazK5QhzWyCZZ2-Lp2rkNvRgOuHI3wKAx6LrMY8gASeur0C2BVH3XcK_n8DwPOvmDCBM52p1inpVcgSBOOiOPKDgJo2UyH1BFPV__HfSpqYwI3KeeayAcw8JOiA-37lcAd8bk2VW6OIFzDLDuN1nqmXCJirqHp_g5ovrz214V7Iru7Elsy5epUE5y70SVU-XE2O2HvS9B6z8rFPORgnpWWPStAfJAnyeIKsO5vMgaiS4PPG2WgCh3MxfSUV0oA_hyEpNGqTzklSgKp5_Yd5cNLQFf8PNcdDJnSaCs4MeyngPgv6FsGuPfpqXScYCp-PR7z1NNEEDoUmvoJtK_9TmHDRD5dTGX9URpyabxmCvrXMf6La6vARAweQSYSEroGPIIlSqNshh2SJaFMGk-lNtu2o0kJc2SvXNBcmdaTzenb68wVV5W9E5wibgZwQbxr2CMDH2ElqaVfi21E-Tl6GA4QPx5ecEHDfwPpEHg9U9fvc7VCpUGwvP434cofTU7StDcziy53MKTxjxMbTP6Wkeax3M_yrPFngZnZRvyJ4skFuuDI_Um8loV3-40kLG5Ayd6bnvGoMv4tR1_u-4C64wuC7H2hMmnQ5jgIo0Z_HjkdvVsd7qrZocMrViN-oNnWXVfgg05WCQy6bUy9xfdVaHhEUE7-HkkAzzSwdu-f9fyBJxGA8eZ707628KiivBLQHZhuf1tcUCbYNknYyt4dauNfWOeuYL1Xx-lvrYLlzpOULpkl3rfhT6isyI4l6cr2KNkAsgblbAgQKMaN8F9TRugTJYoOmWnbU_USQRPRspqAizKJiYPfyIrz_0s12_jMwrj4K-AaHTi14KU6YZGPMYHOPkbVEauzFV8tmWz-Lhdjt2N2CFImaoJRe-tWsCiJcN5DK6-eJsJZ38dMh16bBkJRcvEZGm6JgriEuobxwKJngMVzOs83Be9DNEGUWKbrTzQvIKos_CGGQ-N4V5Wf30AWEr7aWJxk4FpCGlVN89M4sh_15R2JK1xIdBH3fOtoC0HlbcJAEcRX0mx1mPhMC8zf6QNLvlisIOISaOIeOD9ZQbq3HLMapjpwHE2aKsNsptMLWkass6KaV52Iob0QhKWCzsSJ2SotvvRYH-6g5i2l-6uM6GTi735pq2WCZpO6YfQo3s8ZJKA5NXbMatX6gCWqgx6RMREg8UaO-N6yuWPVXu2fvO1MxMWzU6-yqv1JGc2blM3ODrGjd7ph1othWppnx0tkU58cVvnfK4cOnozWcE47wiUy_YMLtpIJ96p2WocDaqIQjO7RMVxEWXJ2w-OuqT58JXhalGUCwZ8yzM6Yy1tvJFr5YHDwRlsK8qrFdctmSkJFE00qTkiE9eG8XpampUwCRmXOwoPBXSqDLRXL9tWvZqVRyX8VupQTWQlASgrdgbZdwyUyd5EMeK6x2NWsSiLZs9GvNa7HL-9swQDI9BvULJfnbpbFa9SvmV_eoTqBDJRKl5gI0jatjjyu02LBU0-Ag9e2kgtkPegIgxdY9XvlmGicRidZUZwVrcb7sxfMjolEQ9jZhWHOUR01XiK4SPkUjTUarhIpvumTFl_t28CfL1IS8WYtSg1Dxm9YSBv9FQ4IOvZQErBfzmUzs2c3SbwvqVlO7RJ30URDAL9MWprt0rGBLZvn1bkLhDGvh2qw0UsSe-q-Eq8mB-YjEZw-L9dNNEz6M0B2oGC-DtfJTYarIPULnHnM2xLhyGqqQjefA_N02HUIs--AS54IPcWA5WMJW1ZVb9bzFltYox7wSvELTCcsP9l6C6XW0oJcG9uRyjJvAx7w258eMUny9eWC2H70_KPMSylh2PqR0ry7VbJAzR_ueBDf_rkYtLXLHaKhh5RAK5ooRLot7GWDMB2XCAmKPaPrHRdPojGvmGwcDrkKRAS2MpWeTiHjFZrEFkKJuUZgKTQw19_FmSAMiSk_DYu5F3Bxv-">
        </form>
    </div>
</div>
<script>
    (function(){
        window._cf_chl_opt={
            cvId: '2',
            cZone: 'www.proofwiki.org',
            cType: 'managed',
            cNounce: '27663',
            cRay: '7a31a8999e77cf55',
            cHash: '8eb85f27d694d47',
            cUPMDTk: "\/wiki\/?__cf_chl_tk=_lMpDYq3NmBWJUBKhol.HWxVxHOgAKiA2wuVcM52kzY-1678011964-0-gaNycGzNCOU",
            cFPWv: 'g',
            cTTimeMs: '1000',
            cMTimeMs: '0',
            cTplV: 4,
            cTplB: 'cf',
            cK: "",
            cRq: {
                ru: 'aHR0cHM6Ly93d3cucHJvb2Z3aWtpLm9yZy93aWtpLw==',
                ra: 'cHl0aG9uIC1tIHVuaXR0ZXN0IFB5d2lraWJvdC84LjEuMC5kZXYwIChnMSkgcmVxdWVzdHMvMi4yOC4yIFB5dGhvbi8zLjcuMTUuZmluYWwuMA==',
                rm: 'R0VU',
                d: 'a5idf5Nf1R9C9Aklw2qegFQ3wBncsiAozLmR6scvmDfQ5uDhTqp2V9x6qn/VViwLnxhoMOX7K1vX3Sl4gaj0zm44m6Rx1yLg28WLQ65bOnwsu7TcDCAOkGWSMnF2PPgWkvvmxwyjyIfNGEKpfgldSMmnrGhltdx5DI6nKd9Za4RwHtMFXpGgGDEg9XsOhxNDzXM+Ed/UBLfz1qsFJBUf99yyrp2q+FZd7fVysDU31qrFL+wDqtg9KUlvYS4zoT0PAz0/AKQyy+h4f8U3Cmjis6UDX3iuq0EaVaMAN5ZFqxMpgzRIZuBOhGiFUf8KApAAigkI1Be4YkqavGOEOca9ZkB7RzsuWlqoeSnZ7pWUC6aKHp96HUTwrIk1Mc2p0Z1mO/CxQojiuc1w52RubDXZHB4//ouEuponqUStkRt9Y9tPCDXXVtdonOnMAp1ob1i1+idJr7L37XnezaCNSwkrX1o/YuMmvUoSFQ4dLduGhpybqCqdyTuDkrCt7s09XJ48nCa4lZZ+REC3r7t4Xj7bAFnm3GfcnFkHLVct1rgzmAfXzB9SlXIQEKuPjoQo9e6E2+KOP1kuba9S6S2iC0oRDAozcMs0lsWUui41pgIyIjBGALr/Ps6M2D4tzGHi90k1LM/t6RpPPzb/e8alzCNj7r/xsKAVofSg4UvSvknCQq7btdvM+oHXA2xxhtSFCfBL',
                t: 'MTY3ODAxMTk2NC40MTAwMDA=',
                m: 'VdUtBcTaIgfgV+VzjkDhvu/Nb8+27cuJ8KqmbK9fKLo=',
                i1: '4h31AgQ3pZdg1/PJvZp/QA==',
                i2: 'USV6gJ4fbufVfYMGNCGTBw==',
                zh: 'BYU57ilg55mVCgfsVoLwjWSJho04SOB22K+EgCX8qeI=',
                uh: 'r8Hqj6haqrHCpL2FuIcbThPQqq8D59HzNefx95RVVjc=',
                hh: 'yMK1ihPJddsxD+sAL3LwI0bT9vWy1TL1MSZQw+R8e3Y=',
            }
        };
        var trkjs = document.createElement('img');
        trkjs.setAttribute('src', '/cdn-cgi/images/trace/managed/js/transparent.gif?ray=7a31a8999e77cf55');
        trkjs.setAttribute('alt', '');
        trkjs.setAttribute('style', 'display: none');
        document.body.appendChild(trkjs);
        var cpo = document.createElement('script');
        cpo.src = '/cdn-cgi/challenge-platform/h/g/orchestrate/managed/v1?ray=7a31a8999e77cf55';
        window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;
        window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, location.href.length - window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search;
        if (window.history && window.history.replaceState) {
            var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;
            history.replaceState(null, null, "\/wiki\/?__cf_chl_rt_tk=_lMpDYq3NmBWJUBKhol.HWxVxHOgAKiA2wuVcM52kzY-1678011964-0-gaNycGzNCOU" + window._cf_chl_opt.cOgUHash);
            cpo.onload = function() {
                history.replaceState(null, null, ogU);
            };
        }
        document.getElementsByTagName('head')[0].appendChild(cpo);
    }());
</script>


    <div class="footer" role="contentinfo">
        <div class="footer-inner">
            <div class="clearfix diagnostic-wrapper">
                <div class="ray-id">Ray ID: <code>7a31a8999e77cf55</code></div>
            </div>
            <div class="text-center" id="footer-text">Performance &amp; security by <a rel="noopener noreferrer" href="https://www.cloudflare.com?utm_source=challenge&utm_campaign=m" target="_blank">Cloudflare</a></div>
        </div>
    </div>
</body>
</html>

In plain text it shows

www.proofwiki.org
Checking if the site connection is secure
www.proofwiki.org needs to review the security of your connection before proceeding.
Ray ID: 7a31a8999e77cf55
Performance & security by Cloudflare

See: https://stackoverflow.com/questions/49968948/what-is-a-ray-id-cloudflare

gerritbot added a comment.Mar 5 2023, 12:38 PM

Change 894206 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] [tests] Skip test on proofwiki on github action

https://gerrit.wikimedia.org/r/894206

gerritbot added a project: Patch-For-Review.Mar 5 2023, 12:38 PM
Xqt lowered the priority of this task from High to Lowest.Mar 5 2023, 2:14 PM

See https://proofwiki.org/wiki/Talk:Main_Page/Archive_17#Spambot_attacks
Maybe admins should be contacted to enable access again.

gerritbot added a comment.Mar 5 2023, 2:24 PM

Change 894206 merged by jenkins-bot:

[pywikibot/core@master] [tests] Skip test on proofwiki on github action

https://gerrit.wikimedia.org/r/894206

Maintenance_bot removed a project: Patch-For-Review.Mar 5 2023, 2:29 PM
RoySmith subscribed.Mar 12 2023, 5:52 PM

I don't understand why this failure is specific to github. It's failing for me as well on my MacOS laptop.

As far as I can tell, WikiHTMLPageParser.server is always None, so MWSite.server will always be None, so MWSite.api will always be None, so MWSite.init() will always raise RuntimeError. This is all independent of what the remote site actually returns. Am I missing something?

Wouldn't it make more sense to mark this test as unconditionally skipped?

RoySmith added a comment.Mar 12 2023, 8:51 PM

Oh, I see how this works. WikiHTMLPageParser overrides handle_starttag() which calls set_api_url().

But, the gist of my comment is still valid: there's nothing here that's specific to github, so the skipping of the test shouldn't be conditional on that.

Xqt added a comment.Mar 14 2023, 4:48 PM

The test works for me locally and on Appveyor CI. Your patch would lead to an "unexpected success" failure then.

https://ci.appveyor.com/project/pywikibot-core/pywikibot

RoySmith added a comment.Mar 14 2023, 6:07 PM

Interesting. It seems to be platform dependent.

When I do a GET on https://www.proofwiki.org/wiki/ in Chrome, it works. Works on Safari and Firefox too.

It fails in the test with a 403 on MacOS. It also fails (403) on MacOS with curl on the command line. Both of these are on my home WiFi over IPv6. I tried switching my laptop over to my phone's hotspot, where it gets a v4 address. Both curl and the unit test still fail.

Curl fails on my wife's laptop, running a different version of MacOS.

On a NetBSD box with curl, it works (i.e. I get redirected to https://proofwiki.org/wiki/Main_Page)

On tools-sgebastion-11, curl also works (301 to https://proofwiki.org/wiki/Main_Page)

RoySmith added a comment.Mar 14 2023, 8:08 PM

How would you feel about having site_detect.py:check_response() raise ServerError if response.status_code == 403?

From the point of view of the tests, SiteDetectionTestCase.assertSite() would cause the test to be skipped. And from the point of view of production code, it would be the same as getting any other kind of remote failure.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits