Page MenuHomePhabricator

UnicodeDecodeError when applying for resources
Closed, ResolvedPublic

Description

We're getting a UnicodeDecodeError when some users attempt to apply to partners on the platform. In a potentially related issue, we also get an AssertionError when applying to multiple partners sometimes.

Original report
i tried to fix this issue by myself w an anchor or whatever but there like was an error so can ur coder like fix wat that error glitch might have just been kthx!! not reporting a security issue or anything instead of this one or anything but ya kthx. also it was not letting me edit 411 just now so ya

Event Timeline

Hi @ZoeSky89 thanks for reporting this, but we'd need a little more than that. Where exactly do you see this error and what were you trying to do? This guide should help you be more descriptive and precise.

Samwalton9-WMF subscribed.

Looks like this corresponded to the following traceback:

Traceback:  

File "/venv/lib/python2.7/site-packages/django/core/handlers/exception.py" in inner
  41.             response = get_response(request)

File "/venv/lib/python2.7/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.             response = self._get_response(request)

File "/venv/lib/python2.7/site-packages/django/core/handlers/base.py" in _get_response
  187.                 response = self.process_exception_by_middleware(e, request)

File "/venv/lib/python2.7/site-packages/django/core/handlers/base.py" in _get_response
  185.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/venv/lib/python2.7/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  23.                 return view_func(request, *args, **kwargs)

File "/venv/lib/python2.7/site-packages/django/views/generic/base.py" in view
  68.             return self.dispatch(request, *args, **kwargs)

File "/app/TWLight/applications/views.py" in dispatch
  358.         return super(SubmitApplicationView, self).dispatch(request, *args, **kwargs)

File "/app/TWLight/view_mixins.py" in dispatch
  208.             request, *args, **kwargs)

File "/app/TWLight/view_mixins.py" in dispatch
  281.             request, *args, **kwargs)

File "/app/TWLight/view_mixins.py" in dispatch
  312.             request, *args, **kwargs)

File "/app/TWLight/view_mixins.py" in dispatch
  333.             request, *args, **kwargs)

File "/venv/lib/python2.7/site-packages/django/views/generic/base.py" in dispatch
  88.         return handler(request, *args, **kwargs)

File "/venv/lib/python2.7/site-packages/django/views/generic/edit.py" in get
  174.         return self.render_to_response(self.get_context_data())

File "/venv/lib/python2.7/site-packages/django/views/generic/edit.py" in get_context_data
  93.             kwargs['form'] = self.get_form()

File "/app/TWLight/applications/views.py" in get_form
  224.         return form_class(**kwargs)

File "/app/TWLight/applications/forms.py" in __init__
  93.         self._add_user_data_subform(user_data)

File "/app/TWLight/applications/forms.py" in _add_user_data_subform
  202.                         partner_list= ", ".join(user_data[datum])))

File "/venv/lib/python2.7/site-packages/django/utils/translation/__init__.py" in ugettext
  89.     return _trans.ugettext(message)

File "/venv/lib/python2.7/site-packages/django/utils/translation/trans_real.py" in ugettext
  345.         return do_translate(message, 'ugettext')

File "/venv/lib/python2.7/site-packages/django/utils/translation/trans_real.py" in do_translate
  324.         result = getattr(translation_object, translation_function)(eol_message)

File "/usr/lib/python2.7/gettext.py" in ugettext
  480.             return unicode(message)

Exception Type: UnicodeDecodeError at /applications/apply/
Exception Value: 'ascii' codec can't decode byte 0xc3 in position 195: ordinal not in range(128)
Samwalton9-WMF renamed this task from Wikipedia Library Server Error to UnicodeDecodeError when applying for resources.Jun 28 2019, 9:10 AM

Reported again today by User:Zapte. It's not obvious to me where the decoding issue is.

I thought I'd managed to reproduce this by attempting to apply to multiple partners at https://wikipedialibrary.wmflabs.org/applications/request/, with one of them as Érudit. That said, that provided a ERROR: Number of partners found does not match number of IDs provided error instead. Confusing.

Traceback:

File "/app/TWLight/applications/views.py" in _get_partners
  381.             assert len(partner_ids) == partners.count()

Exception Type: AssertionError
Samwalton9-WMF updated the task description. (Show Details)

Noting here that our tests should almost certainly include partners with non-ascii names so that things like this get caught.

Ok this time when I applied for multiple partners I got the expected server error for a UnicodeDecodeError...

Confirmed this is still occurring post-T227659 fix.

Don't seem to be able to reproduce this locally with the same partner name that's causing the error on production.

jsn.sherman claimed this task.
jsn.sherman subscribed.

This was a hangover from an accidental encoding regression that occurred when we migrated to docker. It's been resolved with a combination of config, code, and ops.