Page MenuHomePhabricator

test_url_image (tests.proofreadpage_tests.TestProofreadPageValidSite) sometimes fails
Open, Stalled, MediumPublic

Description

https://travis-ci.org/5j9/pywikibot-core/jobs/310700716#L4756

======================================================================

ERROR: test_url_image (tests.proofreadpage_tests.TestProofreadPageValidSite)

Test fetching of url image of the scan of ProofreadPage.

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/travis/build/5j9/pywikibot-core/tests/proofreadpage_tests.py", line 293, in test_url_image

    self.assertEqual(page.url_image, self.valid_redlink['url_image'])

  File "/home/travis/build/5j9/pywikibot-core/pywikibot/proofreadpage.py", line 537, in url_image

    self._url_image = self._url_image.find('img')

AttributeError: 'NoneType' object has no attribute 'find'

Event Timeline

Dalba created this task.Dec 3 2017, 3:16 AM
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptDec 3 2017, 3:16 AM
Dalba added a comment.Dec 3 2017, 3:30 AM

I can reproduce this issue.
In line 536 we have self._url_image = soup.find(class_='prp-page-image').
But the soup does not contain any element with prp-page-image class.

Here is the response.content value after fetching https://en.wikisource.org/w/index.php?title=Page%3APywikibot_test_page_3.jpg&action=edit&redlink=1:

1<!DOCTYPE html>
2<html class="client-nojs" lang="en" dir="ltr">
3<head>
4<meta charset="UTF-8"/>
5<title>Page:Pywikibot test page 3.jpg - Wikisource, the free online library</title>
6<script>document.documentElement.className = document.documentElement.className.replace( /(^|\s)client-nojs(\s|$)/, "$1client-js$2" );</script>
7<script>(window.RLQ=window.RLQ||[]).push(function(){mw.config.set({"wgCanonicalNamespace":"Page","wgCanonicalSpecialPageName":false,"wgNamespaceNumber":104,"wgPageName":"Page:Pywikibot_test_page_3.jpg","wgTitle":"Pywikibot test page 3.jpg","wgCurRevisionId":0,"wgRevisionId":0,"wgArticleId":0,"wgIsArticle":true,"wgIsRedirect":false,"wgAction":"view","wgUserName":null,"wgUserGroups":["*"],"wgCategories":[],"wgBreakFrames":false,"wgPageContentLanguage":"en","wgPageContentModel":"proofread-page","wgSeparatorTransformTable":["",""],"wgDigitTransformTable":["",""],"wgDefaultDateFormat":"dmy","wgMonthNames":["","January","February","March","April","May","June","July","August","September","October","November","December"],"wgMonthNamesShort":["","Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"],"wgRelevantPageName":"Page:Pywikibot_test_page_3.jpg","wgRelevantArticleId":0,"wgRequestId":"WiNqBwpAICwAALn0ND8AAAAP","wgIsProbablyEditable":true,"wgRelevantPageIsProbablyEditable":true,"wgRestrictionCreate":[],"wgWikiEditorEnabledModules":{"toolbar":true,"preview":false,"publish":false},"wgBetaFeaturesFeatures":[],"wgMediaViewerOnClick":true,"wgMediaViewerEnabledByDefault":true,"wgVisualEditor":{"pageLanguageCode":"en","pageLanguageDir":"ltr","pageVariantFallbacks":"en","usePageImages":false,"usePageDescriptions":true},"wgPreferredVariant":"en","wgMFExpandAllSectionsUserOption":false,"wgMFDisplayWikibaseDescriptions":{"search":true,"nearby":true,"watchlist":true,"tagline":false},"wgULSCurrentAutonym":"English","wgNoticeProject":"wikisource","wgCentralNoticeCookiesToDelete":[],"wgCentralNoticeCategoriesUsingLegacy":["Fundraising","fundraising"],"wgCategoryTreePageCategoryOptions":"{\"mode\":0,\"hideprefix\":20,\"showcount\":true,\"namespaces\":false}","wgCentralAuthMobileDomain":false,"wgCodeMirrorEnabled":false,"wgVisualEditorToolbarScrollOffset":0,"wgVisualEditorUnsupportedEditParams":["undo","undoafter","veswitched"],"wgEditSubmitButtonLabelPublish":true});mw.loader.state({"ext.gadget.Site":"ready","ext.globalCssJs.user.styles":"ready","ext.globalCssJs.site.styles":"ready","site.styles":"ready","noscript":"ready","user.styles":"ready","user":"ready","user.options":"ready","user.tokens":"loading","ext.visualEditor.desktopArticleTarget.noscript":"ready","ext.uls.interlanguage":"ready","ext.wikimediaBadges":"ready","mediawiki.legacy.shared":"ready","mediawiki.legacy.commonPrint":"ready","mediawiki.sectionAnchor":"ready","mediawiki.skinning.interface":"ready","skins.vector.styles":"ready","ext.globalCssJs.user":"ready","ext.globalCssJs.site":"ready"});mw.loader.implement("user.tokens@1dqfd7l",function ( $, jQuery, require, module ) {
8mw.user.tokens.set({"editToken":"+\\","patrolToken":"+\\","watchToken":"+\\","csrfToken":"+\\"});/*@nomin*/
9
10});mw.loader.load(["site","mediawiki.page.startup","mediawiki.user","mediawiki.hidpi","mediawiki.page.ready","mediawiki.searchSuggest","ext.proofreadpage.page.navigation","ext.gadget.charinsert","ext.gadget.Easy_LST","ext.gadget.dynamicLayoutOverrides","ext.gadget.collapsibleTables","ext.gadget.dynamicNavBars","ext.gadget.WSexport","ext.gadget.PurgeTab","ext.centralauth.centralautologin","ext.visualEditor.desktopArticleTarget.init","ext.visualEditor.targetLoader","ext.eventLogging.subscriber","ext.wikimediaEvents","ext.navigationTiming","ext.uls.eventlogger","ext.uls.init","ext.uls.interface","ext.centralNotice.geoIP","ext.centralNotice.startUp","skins.vector.js"]);});</script>
11<link rel="stylesheet" href="/w/load.php?debug=false&amp;lang=en&amp;modules=ext.uls.interlanguage%7Cext.visualEditor.desktopArticleTarget.noscript%7Cext.wikimediaBadges%7Cmediawiki.legacy.commonPrint%2Cshared%7Cmediawiki.sectionAnchor%7Cmediawiki.skinning.interface%7Cskins.vector.styles&amp;only=styles&amp;skin=vector"/>
12<script async="" src="/w/load.php?debug=false&amp;lang=en&amp;modules=startup&amp;only=scripts&amp;skin=vector"></script>
13<meta name="ResourceLoaderDynamicStyles" content=""/>
14<link rel="stylesheet" href="/w/load.php?debug=false&amp;lang=en&amp;modules=ext.gadget.Site&amp;only=styles&amp;skin=vector"/>
15<link rel="stylesheet" href="/w/load.php?debug=false&amp;lang=en&amp;modules=site.styles&amp;only=styles&amp;skin=vector"/>
16<meta name="generator" content="MediaWiki 1.31.0-wmf.10"/>
17<meta name="referrer" content="origin-when-cross-origin"/>
18<meta name="robots" content="noindex,nofollow"/>
19<link rel="alternate" type="application/x-wiki" title="Edit" href="/w/index.php?title=Page:Pywikibot_test_page_3.jpg&amp;action=edit"/>
20<link rel="edit" title="Edit" href="/w/index.php?title=Page:Pywikibot_test_page_3.jpg&amp;action=edit"/>
21<link rel="shortcut icon" href="/static/favicon/wikisource.ico"/>
22<link rel="search" type="application/opensearchdescription+xml" href="/w/opensearch_desc.php" title="Wikisource (en)"/>
23<link rel="EditURI" type="application/rsd+xml" href="//en.wikisource.org/w/api.php?action=rsd"/>
24<link rel="license" href="//creativecommons.org/licenses/by-sa/3.0/"/>
25<link rel="alternate" type="application/atom+xml" title="Wikisource Atom feed" href="/w/index.php?title=Special:RecentChanges&amp;feed=atom"/>
26<link rel="canonical" href="https://en.wikisource.org/wiki/Page:Pywikibot_test_page_3.jpg"/>
27<link rel="dns-prefetch" href="//login.wikimedia.org"/>
28<link rel="dns-prefetch" href="//meta.wikimedia.org" />
29<!--[if lt IE 9]><script src="/w/load.php?debug=false&amp;lang=en&amp;modules=html5shiv&amp;only=scripts&amp;skin=vector&amp;sync=1"></script><![endif]-->
30</head>
31<body class="mediawiki ltr sitedir-ltr mw-hide-empty-elt ns-104 ns-subject page-Page_Pywikibot_test_page_3_jpg rootpage-Page_Pywikibot_test_page_3_jpg skin-vector action-view"> <div id="mw-page-base" class="noprint"></div>
32 <div id="mw-head-base" class="noprint"></div>
33 <div id="content" class="mw-body" role="main">
34 <a id="top"></a>
35 <div id="siteNotice" class="mw-body-content"><!-- CentralNotice --></div><div class="mw-indicators mw-body-content">
36</div>
37<h1 id="firstHeading" class="firstHeading" lang="en">Page:Pywikibot test page 3.jpg</h1> <div id="bodyContent" class="mw-body-content">
38 <div id="siteSub" class="noprint">From Wikisource</div> <div id="contentSub"></div>
39 <div id="jump-to-nav" class="mw-jump">
40 Jump to: <a href="#mw-head">navigation</a>, <a href="#p-search">search</a>
41 </div>
42 <div id="mw-content-text" lang="en" dir="ltr" class="mw-content-ltr"><div class="noarticletext mw-content-ltr" dir="ltr" lang="en">
43<div class="plainlinks" style="border: 1px solid #ccc; padding: 0ex 1ex 1ex 1ex; background-color: #fff; color: #000;">
44<h2><span class="mw-headline" id="Wikisource_does_not_have_a_page_with_this_exact_name">Wikisource does not have a page with this exact name</span></h2>
45<h3><span id="You_can_contribute_this_page.21"></span><span class="mw-headline" id="You_can_contribute_this_page!">You can contribute this page!</span></h3>
46<ul><li> Please read <a href="/wiki/Help:Adding_texts" title="Help:Adding texts">Help:Adding texts</a> and our <a href="/wiki/Wikisource:WWI" class="mw-redirect" title="Wikisource:WWI">inclusion guidelines</a>, then <b><a class="external text" href="//en.wikisource.org/w/index.php?title=Page:Pywikibot_test_page_3.jpg&amp;action=edit">add <i>Page:Pywikibot test page 3.jpg</i> to Wikisource</a></b>.</li>
47<li> If you recently added this text, try <a class="external text" href="//en.wikisource.org/w/index.php?title=Page:Pywikibot_test_page_3.jpg&amp;action=purge">purging the server cache</a>.</li>
48<li> If this is a <a href="/wiki/Help:DjVu_files" title="Help:DjVu files">.djvu file</a> with an underlying text layer, the text layer will be applied when you <a class="external text" href="//en.wikisource.org/w/index.php?title=Page:Pywikibot_test_page_3.jpg&amp;action=edit">edit</a> the page.</li>
49<li> For easier proofreading, see <a href="/wiki/Help:Proofreading" class="mw-redirect" title="Help:Proofreading">Help:Proofreading</a>.</li></ul>
50<hr />
51<div class="plainlinks" style="font-size: 83%; text-align: right;">If this error message wasn't helpful for you, you can <a class="external text" href="//en.wikisource.org/w/index.php?title=MediaWiki_talk:Noarticletext&amp;action=edit&amp;section=new">suggest an improvement.</a></div>
52</div>
53</div>
54<noscript><img src="//en.wikisource.org/wiki/Special:CentralAutoLogin/start?type=1x1" alt="" title="" width="1" height="1" style="border: none; position: absolute;" /></noscript></div> <div class="printfooter">
55 Retrieved from "<a dir="ltr" href="https://en.wikisource.org/wiki/Page:Pywikibot_test_page_3.jpg">https://en.wikisource.org/wiki/Page:Pywikibot_test_page_3.jpg</a>" </div>
56 <div id="catlinks" class="catlinks catlinks-allhidden" data-mw="interface"></div> <div class="visualClear"></div>
57 </div>
58 </div>
59 <div id="mw-navigation">
60 <h2>Navigation menu</h2>
61 <div id="mw-head">
62 <div id="p-personal" role="navigation" class="" aria-labelledby="p-personal-label">
63 <h3 id="p-personal-label">Personal tools</h3>
64 <ul>
65 <li id="pt-anonuserpage">Not logged in</li><li id="pt-anontalk"><a href="/wiki/Special:MyTalk" title="Discussion about edits from this IP address [n]" accesskey="n">Talk</a></li><li id="pt-anoncontribs"><a href="/wiki/Special:MyContributions" title="A list of edits made from this IP address [y]" accesskey="y">Contributions</a></li><li id="pt-createaccount"><a href="/w/index.php?title=Special:CreateAccount&amp;returnto=Page%3APywikibot+test+page+3.jpg" title="You are encouraged to create an account and log in; however, it is not mandatory">Create account</a></li><li id="pt-login"><a href="/w/index.php?title=Special:UserLogin&amp;returnto=Page%3APywikibot+test+page+3.jpg" title="You are encouraged to log in; however, it is not mandatory [o]" accesskey="o">Log in</a></li> </ul>
66 </div>
67 <div id="left-navigation">
68 <div id="p-namespaces" role="navigation" class="vectorTabs" aria-labelledby="p-namespaces-label">
69 <h3 id="p-namespaces-label">Namespaces</h3>
70 <ul>
71 <li id="ca-nstab-page" class="selected new"><span><a href="/w/index.php?title=Page:Pywikibot_test_page_3.jpg&amp;action=edit&amp;redlink=1" title="View the book page">Page</a></span></li><li id="ca-talk" class="new"><span><a href="/w/index.php?title=Page_talk:Pywikibot_test_page_3.jpg&amp;action=edit&amp;redlink=1" rel="discussion" title="Discussion about the content page [t]" accesskey="t">Discussion</a></span></li><li id="ca-proofreadPageScanLink"><span><a href="//upload.wikimedia.org/wikisource/en/3/37/Pywikibot_test_page_3.jpg">Image</a></span></li> </ul>
72 </div>
73 <div id="p-variants" role="navigation" class="vectorMenu emptyPortlet" aria-labelledby="p-variants-label">
74 <h3 id="p-variants-label">
75 <span>Variants</span>
76 </h3>
77 <div class="menu">
78 <ul>
79 </ul>
80 </div>
81 </div>
82 </div>
83 <div id="right-navigation">
84 <div id="p-views" role="navigation" class="vectorTabs" aria-labelledby="p-views-label">
85 <h3 id="p-views-label">Views</h3>
86 <ul>
87 <li id="ca-edit" class="collapsible"><span><a href="/w/index.php?title=Page:Pywikibot_test_page_3.jpg&amp;action=edit" title="Edit this page [e]" accesskey="e">Create</a></span></li> </ul>
88 </div>
89 <div id="p-cactions" role="navigation" class="vectorMenu emptyPortlet" aria-labelledby="p-cactions-label">
90 <h3 id="p-cactions-label"><span>More</span></h3>
91 <div class="menu">
92 <ul>
93 </ul>
94 </div>
95 </div>
96 <div id="p-search" role="search">
97 <h3>
98 <label for="searchInput">Search</label>
99 </h3>
100 <form action="/w/index.php" id="searchform">
101 <div id="simpleSearch">
102 <input type="search" name="search" placeholder="Search Wikisource" title="Search Wikisource [f]" accesskey="f" id="searchInput"/><input type="hidden" value="Special:Search" name="title"/><input type="submit" name="fulltext" value="Search" title="Search the pages for this text" id="mw-searchButton" class="searchButton mw-fallbackSearchButton"/><input type="submit" name="go" value="Go" title="Go to a page with this exact name if it exists" id="searchButton" class="searchButton"/> </div>
103 </form>
104 </div>
105 </div>
106 </div>
107 <div id="mw-panel">
108 <div id="p-logo" role="banner"><a class="mw-wiki-logo" href="/wiki/Main_Page" title="Visit the main page"></a></div>
109 <div class="portal" role="navigation" id='p-navigation' aria-labelledby='p-navigation-label'>
110 <h3 id='p-navigation-label'>Navigation</h3>
111 <div class="body">
112 <ul>
113 <li id="n-mainpage"><a href="/wiki/Main_Page" title="Visit the main page [z]" accesskey="z">Main Page</a></li><li id="n-portal"><a href="/wiki/Wikisource:Community_portal" title="About the project, what you can do, where to find things">Community portal</a></li><li id="n-scriptorium"><a href="/wiki/Wikisource:Scriptorium">Central discussion</a></li><li id="n-recentchanges"><a href="/wiki/Special:RecentChanges" title="A list of recent changes in the wiki [r]" accesskey="r">Recent changes</a></li><li id="n-Subject-index"><a href="/wiki/Portal:Portals">Subject index</a></li><li id="n-Authors"><a href="/wiki/Category:Authors_by_alphabetical_order">Authors</a></li><li id="n-Random-work"><a href="/wiki/Special:RandomRootpage/Main">Random work</a></li><li id="n-Random-author"><a href="/wiki/Special:Random/Author">Random author</a></li><li id="n-Random-transcription"><a href="/wiki/Special:Random/Index">Random transcription</a></li><li id="n-help"><a href="/wiki/Help:Contents" title="The place to find out">Help</a></li><li id="n-sitesupport"><a href="//donate.wikimedia.org/wiki/Special:FundraiserRedirector?utm_source=donate&amp;utm_medium=sidebar&amp;utm_campaign=C13_en.wikisource.org&amp;uselang=en" title="Support us">Donate</a></li> </ul>
114 </div>
115 </div>
116 <div class="portal" role="navigation" id='p-do' aria-labelledby='p-do-label'>
117 <h3 id='p-do-label'></h3>
118 <div class="body">
119 <ul>
120 </ul>
121 </div>
122 </div>
123 <div class="portal" role="navigation" id='p-tb' aria-labelledby='p-tb-label'>
124 <h3 id='p-tb-label'>Tools</h3>
125 <div class="body">
126 <ul>
127 <li id="t-whatlinkshere"><a href="/wiki/Special:WhatLinksHere/Page:Pywikibot_test_page_3.jpg" title="A list of all wiki pages that link here [j]" accesskey="j">What links here</a></li><li id="t-specialpages"><a href="/wiki/Special:SpecialPages" title="A list of all special pages [q]" accesskey="q">Special pages</a></li><li id="t-info"><a href="/w/index.php?title=Page:Pywikibot_test_page_3.jpg&amp;action=info" title="More information about this page">Page information</a></li> </ul>
128 </div>
129 </div>
130 <div class="portal" role="navigation" id='p-lang' aria-labelledby='p-lang-label'>
131 <h3 id='p-lang-label'>In other languages</h3>
132 <div class="body">
133 <ul>
134 </ul>
135 </div>
136 </div>
137 </div>
138 </div>
139 <div id="footer" role="contentinfo">
140 <ul id="footer-places">
141 <li id="footer-places-privacy"><a href="https://wikimediafoundation.org/wiki/Privacy_policy" class="extiw" title="wmf:Privacy policy">Privacy policy</a></li>
142 <li id="footer-places-about"><a href="/wiki/Wikisource:About" title="Wikisource:About">About Wikisource</a></li>
143 <li id="footer-places-disclaimer"><a href="/wiki/Wikisource:General_disclaimer" title="Wikisource:General disclaimer">Disclaimers</a></li>
144 <li id="footer-places-developers"><a href="https://www.mediawiki.org/wiki/Special:MyLanguage/How_to_contribute">Developers</a></li>
145 <li id="footer-places-cookiestatement"><a href="https://wikimediafoundation.org/wiki/Cookie_statement">Cookie statement</a></li>
146 <li id="footer-places-mobileview"><a href="//en.m.wikisource.org/w/index.php?title=Page:Pywikibot_test_page_3.jpg&amp;mobileaction=toggle_view_mobile" class="noprint stopMobileRedirectToggle">Mobile view</a></li>
147 </ul>
148 <ul id="footer-icons" class="noprint">
149 <li id="footer-copyrightico">
150 <a href="https://wikimediafoundation.org/"><img src="/static/images/wikimedia-button.png" srcset="/static/images/wikimedia-button-1.5x.png 1.5x, /static/images/wikimedia-button-2x.png 2x" width="88" height="31" alt="Wikimedia Foundation"/></a> </li>
151 <li id="footer-poweredbyico">
152 <a href="//www.mediawiki.org/"><img src="/static/images/poweredby_mediawiki_88x31.png" alt="Powered by MediaWiki" srcset="/static/images/poweredby_mediawiki_132x47.png 1.5x, /static/images/poweredby_mediawiki_176x62.png 2x" width="88" height="31"/></a> </li>
153 </ul>
154 <div style="clear: both;"></div>
155 </div>
156 <script>(window.RLQ=window.RLQ||[]).push(function(){mw.config.set({"wgBackendResponseTime":66,"wgHostname":"mw1323"});});</script>
157 </body>
158</html>

There is no prp-page-image.

Dalba added a comment.Dec 3 2017, 5:36 AM

I cannot reproduce it anymore. The strange thing is that running the following script on toolforge prints 100, while on my local computer prints 0.

import requests
error_count = 0
for i in range(100):
    r = requests.get('https://en.wikisource.org/w/index.php?title=Page%3APywikibot_test_page_3.jpg&action=edit&redlink=1')
    try:
        assert 'prp-page-image' in r.text
    except AssertionError:
        error_count += 1
print(error_count)

Perhaps some caching mechanism is involved.

Change 396917 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [bugfix] Catch AttributeError and raise the ValueError

https://gerrit.wikimedia.org/r/396917

Xqt claimed this task.Dec 11 2017, 12:43 PM

Change 396917 merged by jenkins-bot:
[pywikibot/core@master] [bugfix] Catch AttributeError and raise the ValueError

https://gerrit.wikimedia.org/r/396917

Dalba closed this task as Resolved.Dec 13 2017, 2:38 AM
Dalba added a comment.Jan 21 2018, 3:08 AM

The test still fails from time to time, the new traceback is:

https://travis-ci.org/wikimedia/pywikibot/jobs/331196015#L5369

======================================================================
ERROR: test_url_image (tests.proofreadpage_tests.TestProofreadPageValidSite)
Test fetching of url image of the scan of ProofreadPage.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/wikimedia/pywikibot/tests/proofreadpage_tests.py", line 293, in test_url_image
    self.assertEqual(page.url_image, self.valid_redlink['url_image'])
  File "/home/travis/build/wikimedia/pywikibot/pywikibot/proofreadpage.py", line 539, in url_image
    raise ValueError('No prp-page-image src found for %s.' % self)
ValueError: No prp-page-image src found for [[wikisource:en:Page:Pywikibot test page 3.jpg]].
Dalba reopened this task as Open.Jan 21 2018, 3:09 AM
Dalba removed Xqt as the assignee of this task.Jul 8 2018, 3:19 AM

Change 444424 had a related patch set uploaded (by Dalba; owner: dalba):
[pywikibot/core@master] Skip TestIndexPageMappings and test_url_image

https://gerrit.wikimedia.org/r/444424

Change 444424 merged by jenkins-bot:
[pywikibot/core@master] Skip TestIndexPageMappings and test_url_image

https://gerrit.wikimedia.org/r/444424

Dalba changed the task status from Open to Stalled.EditedJul 8 2018, 5:42 AM
Dalba lowered the priority of this task from High to Medium.
Dalba removed a project: Patch-For-Review.

Now we should wait for the subtask (T114318) to be resolved, then we can revert the patch above.