Page MenuHomePhabricator

-liverecentchanges not working
Closed, ResolvedPublic

Description

>pwb.py interwiki -lang:pt -family:wiktionary -ns:14 -liverecentchanges
NOTE: Number of pages queued is 0, trying to add 50 more.
WARNING:root:stream.wikimedia.org:80/socket.io [waiting for connection] unexpected status code (404 404 Not Found)

(..... hung ......)

>pwb.py version
Pywikibot: [ssh] pywikibot-core.git (a42f9a7, g6632, 2015/10/06, 14:55:06, ok)
Release version: 2.0b3
requests version: 2.7.0
  cacerts: C:\Program Files\Python\lib\site-packages\requests\cacert.pem
    certificate test: ok
Python: 2.7.9 (default, Dec 10 2014, 12:28:03) [MSC v.1500 64 bit (AMD64)]
PYWIKIBOT2_DIR: Not set
PYWIKIBOT2_DIR_PWB: D:\Work\pywikipedia
PYWIKIBOT2_NO_USER_CONFIG: Not set
Config base dir: D:\Work\pywikipedia

Details

Related Gerrit Patches:

Event Timeline

Malafaya created this task.Oct 7 2015, 5:22 PM
Malafaya raised the priority of this task from to Medium.
Malafaya updated the task description. (Show Details)
Malafaya added a project: Pywikibot.
Malafaya added a subscriber: Malafaya.
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptOct 7 2015, 5:22 PM
jayvdb closed this task as Invalid.Oct 8 2015, 8:32 AM
jayvdb claimed this task.

It is working for me

$ python pwb.py listpages -lang:pt -family:wiktionary -ns:0 -liverecentchanges
   1 adoptarás
   2 adoptará
   3 adoptares
   4 adoptaremos
   5 adoptareis
   6 adoptaras
   7 adoptara
...

Also we have fairly good tests covering this functionality.
Note that setup.py requires socketIO-client<0.6.1

Could you try again.

If it still fails for you, please re-open and also report the version of socketIO_client you have installed, etc

$ python 
Python 2.7.5 (default, Apr 10 2015, 08:09:05) 
[GCC 4.8.3 20140911 (Red Hat 4.8.3-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import socketIO_client
>>> socketIO_client.__version__
'0.5.4'
>>> quit()

$ python3
Python 3.3.2 (default, Dec  4 2014, 12:49:00) 
[GCC 4.8.3 20140911 (Red Hat 4.8.3-7)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import socketIO_client
>>> socketIO_client.__version__
'0.5.4'
>>> quit()

Version was 0.6.3. I have 0.5.5 (but reports 0.5.4) now.

Error is now (probably some other problem):

>pwb.py interwiki -lang:pt -family:wiktionary -ns:14 -liverecentchanges
NOTE: Number of pages queued is 0, trying to add 50 more.
Dump pt (wiktionary) appended.
Traceback (most recent call last):
  File "D:\Work\pywikipedia\pwb.py", line 248, in <module>
    if not main():
  File "D:\Work\pywikipedia\pwb.py", line 242, in main
    run_python_file(filename, [filename] + args, argvu, file_package)
  File "D:\Work\pywikipedia\pwb.py", line 120, in run_python_file
    main_mod.__dict__)
  File ".\scripts\interwiki.py", line 2647, in <module>
    main()
  File ".\scripts\interwiki.py", line 2622, in main
    bot.run()
  File ".\scripts\interwiki.py", line 2366, in run
    self.queryStep()
  File ".\scripts\interwiki.py", line 2339, in queryStep
    self.oneQuery()
  File ".\scripts\interwiki.py", line 2307, in oneQuery
    site = self.selectQuerySite()
  File ".\scripts\interwiki.py", line 2277, in selectQuerySite
    self.generateMore(globalvar.maxquerysize - mycount)
  File ".\scripts\interwiki.py", line 2178, in generateMore
    page = next(self.pageGenerator)
  File "D:\Work\pywikipedia\pywikibot\pagegenerators.py", line 1260, in NamespaceFilterPageGenerator
    for page in generator:
  File "D:\Work\pywikipedia\pywikibot\pagegenerators.py", line 2138, in LiveRCPageGenerator
    for entry in site_rc_listener(site, total=total):
  File "D:\Work\pywikipedia\pywikibot\comms\rcstream.py", line 182, in rc_listener
    total=total
  File "D:\Work\pywikipedia\pywikibot\comms\rcstream.py", line 121, in __init__
    self.client.define(RCListener, rcpath)
  File "C:\Program Files\Python\lib\site-packages\socketio_client-0.5.5-py2.7.egg\socketIO_client\__init__.py", line 221
, in define
    self._transport.connect(path)
  File "C:\Program Files\Python\lib\site-packages\socketio_client-0.5.5-py2.7.egg\socketIO_client\__init__.py", line 332
, in _transport
    socketIO_session, self._transport_name)
  File "C:\Program Files\Python\lib\site-packages\socketio_client-0.5.5-py2.7.egg\socketIO_client\__init__.py", line 380
, in _get_transport
    }[transport_name](session, self.is_secure, self._base_url, **self._kw)
  File "C:\Program Files\Python\lib\site-packages\socketio_client-0.5.5-py2.7.egg\socketIO_client\transports.py", line 1
46, in __init__
    self._connection = websocket.create_connection(url, header=headers)
  File "C:\Program Files\Python\lib\site-packages\websocket\_core.py", line 104, in create_connection
    websock.connect(url, **options)
  File "C:\Program Files\Python\lib\site-packages\websocket\_core.py", line 266, in connect
    self.sock, addrs = connect(url, self.sock_opt, proxy_info(**options))
  File "C:\Program Files\Python\lib\site-packages\websocket\_http.py", line 65, in connect
    sock = _tunnel(sock, hostname, port, auth)
  File "C:\Program Files\Python\lib\site-packages\websocket\_http.py", line 186, in _tunnel
    "failed CONNECT via proxy status: %r" + status)
TypeError: cannot concatenate 'str' and 'int' objects
<type 'exceptions.TypeError'>
CRITICAL: Closing network session.
jayvdb reopened this task as Open.Oct 9 2015, 10:18 AM

Ok. What version of websocket do you have installed?

python pip.exe list

argparse (1.2.1)
autobahn (0.7.4)
flake8 (2.4.1)
git-review (1.24)
ipaddr (2.1.11)
ipaddress (1.0.14)
mccabe (0.3.1)
memento-client (0.5.1)
pep8 (1.5.7)
pip (1.5.6)
pyflakes (0.8.1)
pywikibot (2.0rc1.post2)
pywin32 (218)
requests (2.7.0)
setuptools (7.0)
six (1.10.0)
socketio-client (0.5.5)
stratum (0.2.12)
stratum-mining-proxy (1.5.2)
twisted (13.2.0)
unicodedata2 (7.0.0-2)
websocket-client (0.32.0)
zope.interface (3.8.0)

The bug in websocket first appeared in v0.27.0 and exists in all versions up to the latest release v0.32.0

https://github.com/liris/websocket-client/commit/cac1cf234e7f1560556c817f1a828e4e3b53a608

It has been fixed in their master:

https://github.com/liris/websocket-client/pull/196

But that fix hasnt been released yet.

You could try installing v0.26 (released around March 2015). That should work, as we had builds running tests on that verson, without any problems.

If v0.26 doesnt work, let us know and we'll blacklist that version. And then try installing their git master, and/or harrass the project team to release a new version ;-)

This fix was included in v0.33 . Now we need to update our minimum acceptable version of websocket-client in setup.py and requirements.txt.

jayvdb moved this task from Backlog to Patch merged upstream on the Upstream board.Jan 6 2016, 6:40 AM
Restricted Application added a subscriber: TerraCodes. · View Herald TranscriptSep 24 2016, 2:39 PM

Change 312732 had a related patch set uploaded (by Anoopchandu18):
Set minimum requirement of websocket-client

https://gerrit.wikimedia.org/r/312732

Change 312732 merged by jenkins-bot:
Set minimum requirement of websocket-client

https://gerrit.wikimedia.org/r/312732

Patch merged. Is this task fixed (=status should be set to resolved), or is there more to do?

Xqt closed this task as Resolved.Feb 27 2017, 6:47 AM
Xqt added a subscriber: Xqt.