Page MenuHomePhabricator

Fix UnicodeEncodeError on CSV downloads
Closed, ResolvedPublic

Description

This came up before, it seems like we're still not encoding CSVs properly in all cases:

Traceback:  

File "/venv/lib/python2.7/site-packages/django/core/handlers/exception.py" in inner
  41.             response = get_response(request)

File "/venv/lib/python2.7/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.             response = self._get_response(request)

File "/venv/lib/python2.7/site-packages/django/core/handlers/base.py" in _get_response
  187.                 response = self.process_exception_by_middleware(e, request)

File "/venv/lib/python2.7/site-packages/django/core/handlers/base.py" in _get_response
  185.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/venv/lib/python2.7/site-packages/django/views/generic/base.py" in view
  68.             return self.dispatch(request, *args, **kwargs)

File "/venv/lib/python2.7/site-packages/django/views/generic/base.py" in dispatch
  88.         return handler(request, *args, **kwargs)

File "/app/TWLight/graphs/views.py" in get
  175.         self._write_data(response)

File "/app/TWLight/graphs/views.py" in _write_data
  266.                          'data': _('Number of applications')})

File "/usr/lib/python2.7/csv.py" in writerow
  152.         return self.writer.writerow(self._dict_to_list(rowdict))

Exception Type: UnicodeEncodeError at /csv/app_distribution/69/
Exception Value: 'ascii' codec can't encode characters in position 0-8: ordinal not in range(128)

Event Timeline

Hm. I actually can't reproduce this - https://wikipedialibrary.wmflabs.org/csv/app_distribution/69/ sends me the CSV as expected. I'm also confused about how this error is being raised in the first place - we shouldn't have anything in this CSV that isn't encodable as ascii.

Reproduced - this error occurs when the user's language is set to something with non-ascii characters; the CSV header causes the error.

Samwalton9-WMF moved this task from Done to Open tasks on the Library-Card-Platform board.

This seems to have come back:

Traceback: 

File "/venv/lib/python2.7/site-packages/django/core/handlers/exception.py" in inner
  41.             response = get_response(request)

File "/venv/lib/python2.7/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.             response = self._get_response(request)

File "/venv/lib/python2.7/site-packages/django/core/handlers/base.py" in _get_response
  187.                 response = self.process_exception_by_middleware(e, request)

File "/venv/lib/python2.7/site-packages/django/core/handlers/base.py" in _get_response
  185.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/venv/lib/python2.7/site-packages/django/views/generic/base.py" in view
  68.             return self.dispatch(request, *args, **kwargs)

File "/venv/lib/python2.7/site-packages/django/views/generic/base.py" in dispatch
  88.         return handler(request, *args, **kwargs)

File "/app/TWLight/graphs/views.py" in get
  192.         self._write_data(response)

File "/app/TWLight/graphs/views.py" in _write_data
  281.             writer.writerow(row)

File "/usr/lib/python2.7/csv.py" in writerow
  152.         return self.writer.writerow(self._dict_to_list(rowdict))

Exception Type: UnicodeEncodeError at /csv/app_distribution/69/
Exception Value: 'ascii' codec can't encode character u'\u0432' in position 1: ordinal not in range(128)