Page MenuHomePhabricator

rcstream service - gevent dependency incompatibility
Closed, DeclinedPublic

Description

While T153772 was worked an apt-get upgrade was issued to upgrade a bunch of outdated packages, including nginx itself. Unfortunately after the upgrade the rcstream service started to throw a lot of errors of the type"

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 523, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/gevent/baseserver.py", line 153, in _close_when_done
    return handle(*args)
  File "/usr/lib/python2.7/dist-packages/socketio/server.py", line 124, in handle
    handler.handle()
  File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 410, in handle
    result = self.handle_one_request()
  File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 625, in handle_one_request
    self.handle_one_response()
  File "/usr/lib/python2.7/dist-packages/socketio/handler.py", line 122, in handle_one_response
    return self._do_handshake(handshake_tokens.groupdict())
  File "/usr/lib/python2.7/dist-packages/socketio/handler.py", line 64, in _do_handshake
    self.write_smart(data)
  File "/usr/lib/python2.7/dist-packages/socketio/handler.py", line 88, in write_smart
    self.write_plain_result(data)
  File "/usr/lib/python2.7/dist-packages/socketio/handler.py", line 78, in write_plain_result
    ("Content-Type", "text/plain"),
  File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 740, in start_response
    raise UnicodeError("The value must be a native string", header, value)
UnicodeError: ('The value must be a native string', 'Access-Control-Max-Age', 3600)
<Greenlet at 0x7f6bbd788910: _close_when_done(<socket at 0x7f6baf6d6c50 fileno=[Errno 9] Bad fil, ('127.0.0.1', 37904))> failed with UnicodeError

After some debugging between @BBlack, @ema and me we found the issue in the incompatibility of the newer python-gevent installed (1.1b6-1~trusty1) with the current python-socketio installed (0.3.6-2).

The new python-gevent added a check that all custom headers must be strings, both the key and the value, while the old python-socketio was setting one as an integer:

# File "/usr/lib/python2.7/dist-packages/socketio/handler.py", line 78, in write_plain_result
("Access-Control-Max-Age", 3600),

To unblock the situation we decided to downgrade the python-gevent pacakge to the previous version (1.0-1ubuntu1).
The python-socketio seems to have fixed this issue in their master branch, but there were no official release after that.
Hence the problem of incompatibility of those dependencies remains and should be fixed in a safer way that avoid issues of this type.

I couldn't help but notice also that we're using the gevent-socketio library that has it's last release on PyPi in 2014, instead of a more maintained one, like python-socketio for example.

Event Timeline

Volans created this task.Dec 20 2016, 6:40 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 20 2016, 6:40 PM
Krinkle renamed this task from Socket.io - gevent dependency incompatibility to rcstream service - gevent dependency incompatibility.Jan 30 2017, 10:59 PM
Krinkle moved this task from Inbox to Backlog-RCStream on the Wikimedia-Stream board.

FYI: RCStream is planned for decommission in July this year.

Ottomata triaged this task as Low priority.Mar 6 2017, 6:42 PM
Aklapper closed this task as Declined.Jul 7 2017, 3:35 PM

Closing this task as "declined" as RCStream is deprecated and scheduled to be shut down today. See T156919 for more information.