While T153772 was worked an apt-get upgrade was issued to upgrade a bunch of outdated packages, including nginx itself. Unfortunately after the upgrade the rcstream service started to throw a lot of errors of the type"
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 523, in run
result = self._run(*self.args, **self.kwargs)
File "/usr/lib/python2.7/dist-packages/gevent/baseserver.py", line 153, in _close_when_done
return handle(*args)
File "/usr/lib/python2.7/dist-packages/socketio/server.py", line 124, in handle
handler.handle()
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 410, in handle
result = self.handle_one_request()
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 625, in handle_one_request
self.handle_one_response()
File "/usr/lib/python2.7/dist-packages/socketio/handler.py", line 122, in handle_one_response
return self._do_handshake(handshake_tokens.groupdict())
File "/usr/lib/python2.7/dist-packages/socketio/handler.py", line 64, in _do_handshake
self.write_smart(data)
File "/usr/lib/python2.7/dist-packages/socketio/handler.py", line 88, in write_smart
self.write_plain_result(data)
File "/usr/lib/python2.7/dist-packages/socketio/handler.py", line 78, in write_plain_result
("Content-Type", "text/plain"),
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 740, in start_response
raise UnicodeError("The value must be a native string", header, value)
UnicodeError: ('The value must be a native string', 'Access-Control-Max-Age', 3600)
<Greenlet at 0x7f6bbd788910: _close_when_done(<socket at 0x7f6baf6d6c50 fileno=[Errno 9] Bad fil, ('127.0.0.1', 37904))> failed with UnicodeErrorAfter some debugging between @BBlack, @ema and me we found the issue in the incompatibility of the newer python-gevent installed (1.1b6-1~trusty1) with the current python-socketio installed (0.3.6-2).
The new python-gevent added a check that all custom headers must be strings, both the key and the value, while the old python-socketio was setting one as an integer:
# File "/usr/lib/python2.7/dist-packages/socketio/handler.py", line 78, in write_plain_result
("Access-Control-Max-Age", 3600),To unblock the situation we decided to downgrade the python-gevent pacakge to the previous version (1.0-1ubuntu1).
The python-socketio seems to have fixed this issue in their master branch, but there were no official release after that.
Hence the problem of incompatibility of those dependencies remains and should be fixed in a safer way that avoid issues of this type.
I couldn't help but notice also that we're using the gevent-socketio library that has it's last release on PyPi in 2014, instead of a more maintained one, like python-socketio for example.