Page MenuHomePhabricator

Upgrade project to Django 3.0
Closed, ResolvedPublic

Description

The Library Card platform is currently running on Django 1.11, which stopped receiving extended support in April 2020 (https://www.djangoproject.com/download/). We need to upgrade to a newer, supported, version of Django.

At the time of writing, Django 2.2 is the current long-term support version, running until April 2022, but now that we have more engineering capacity we can comfortably move to 3.0 and upgrade on an ongoing basis.

Event Timeline

Samwalton9 triaged this task as Medium priority.Nov 20 2019, 4:10 PM
Samwalton9 raised the priority of this task from Medium to High.Feb 14 2020, 1:14 PM

Hello! I'm new to the Library Card Platform project. May I take this as a first task?

Samwalton9 renamed this task from Upgrade to Django 2 to Upgrade project to Django 2.2.Jun 5 2020, 10:29 AM
Samwalton9 updated the task description. (Show Details)

Wow that's super weird timing - I was just editing this task as you commented!

This is a task we really want to make sure we get right so the Wikipedia Library team will be working on it later in the month. I might suggest one of the following tasks instead: T242196, T243137, T252877.

This process requires that we check all the backwards-incompatible changes and deprecated features for each ‘final’ release from our current Django version (1.11.28), up to and including the latest lts release (2.2.14 as of today). You can find the final release list (anchored to 2.2) here:
https://docs.djangoproject.com/en/3.0/releases/#id1

If I didn't care to estimate the effort up front, I would just start plowing through that list sequentially, reading about the necessary changes and implementing them in order. If we want to estimate the level of effort up front, it would be helpful for one of us to go through that list and copy+paste backwards-incompatible changes and deprecated features out in a nested list for each release. That way we could see see the whole list here in one place.

I think we can just go ahead, I only asked initially about time expectations in terms of weighing it up against T161175 but I think we're in agreement that's the priority anyway.

Samwalton9 renamed this task from Upgrade project to Django 2.2 to Upgrade project to Django 3.0.Jun 26 2020, 10:15 AM
Samwalton9 updated the task description. (Show Details)

We chatted about this at our engineering meeting today, and decided that the best way to tackle this would be a single PR for the full upgrade from 1.11 to 3.0. We can use the Staging tool to test the upgrade.

Just did some poking around on Staging and got a Server error: When marking an Application as 'Sent' either directly on the application evaluation page or via the Sent interface, a server error sometimes (but not always) occurs.

Traceback (most recent call last):
  File "/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/venv/lib/python3.8/site-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/app/TWLight/view_mixins.py", line 320, in dispatch
    return super(NotDeleted, self).dispatch(request, *args, **kwargs)
  File "/app/TWLight/view_mixins.py", line 175, in dispatch
    return super(PartnerCoordinatorOrSelf, self).dispatch(request, *args, **kwargs)
  File "/app/TWLight/view_mixins.py", line 245, in dispatch
    return super(ToURequired, self).dispatch(request, *args, **kwargs)
  File "/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 97, in dispatch
    return handler(request, *args, **kwargs)
  File "/app/TWLight/applications/views.py", line 1026, in post
    return super(EvaluateApplicationView, self).post(request, *args, **kwargs)
  File "/venv/lib/python3.8/site-packages/django/views/generic/edit.py", line 194, in post
    return super().post(request, *args, **kwargs)
  File "/venv/lib/python3.8/site-packages/django/views/generic/edit.py", line 142, in post
    return self.form_valid(form)
  File "/app/TWLight/applications/views.py", line 920, in form_valid
    app.save()
  File "/venv/lib/python3.8/site-packages/reversion/revisions.py", line 322, in do_revision_context
    return func(*args, **kwargs)
  File "/app/TWLight/applications/models.py", line 174, in save
    super(Application, self).save(*args, **kwargs)
  File "/venv/lib/python3.8/site-packages/django/db/models/base.py", line 745, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/venv/lib/python3.8/site-packages/django/db/models/base.py", line 793, in save_base
    post_save.send(
  File "/venv/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 173, in send
    return [
  File "/venv/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 174, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
  File "/app/TWLight/applications/signals.py", line 210, in post_revision_commit
    authorization.save()
  File "/venv/lib/python3.8/site-packages/django/db/models/base.py", line 745, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/venv/lib/python3.8/site-packages/django/db/models/base.py", line 769, in save_base
    pre_save.send(
  File "/venv/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 173, in send
    return [
  File "/venv/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 174, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
  File "/app/TWLight/users/signals.py", line 29, in validate_authorization
    instance.full_clean()
  File "/venv/lib/python3.8/site-packages/django/db/models/base.py", line 1222, in full_clean
    raise ValidationError(errors)

Exception Type: ValidationError at /applications/evaluate/7307/
Exception Value: {'__all__': ['Authorization authorizer must be a coordinator or staff.']}

The root issue seems to be this function, called when the app attempts to create the Authorization object: https://github.com/WikipediaLibrary/TWLight/blob/e63a84db6a1a32c74b211134bddc403103fb0d4e/TWLight/users/helpers/validation.py#L27. The core error here could actually be the one fixed in T257667, so a rebase might be worthwhile before investigating further.

The placeholder text on the Review page dropdowns appears to have become misaligned:

Staging

Production

I just got a server error when logging out:

Traceback (most recent call last):
  File "/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)

Exception Type: TypeError at /accounts/logout/
Exception Value: __init__() takes 1 positional argument but 2 were given

We might be missing as_view() in urls.py per https://stackoverflow.com/questions/52391139/why-am-i-getting-init-takes-1-positional-argument-but-2-were-given.

Thank you for pointing those bugs out!

I just got a server error when logging out:

Traceback (most recent call last):
  File "/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)

Exception Type: TypeError at /accounts/logout/
Exception Value: __init__() takes 1 positional argument but 2 were given

We might be missing as_view() in urls.py per https://stackoverflow.com/questions/52391139/why-am-i-getting-init-takes-1-positional-argument-but-2-were-given.

This has been fixed.

The placeholder text on the Review page dropdowns appears to have become misaligned:

Staging

Production

I think this error has been introduced when I upgraded the django-autocomplete-light library that uses Select2 to create autocomplete select tags. Unfortunately, I cannot downgrade the version of that library because it uses a library deprecated in Django 3.0. I can raise an issue in its GitHub repository and check if that's a quick fix or not.

Just did some poking around on Staging and got a Server error: When marking an Application as 'Sent' either directly on the application evaluation page or via the Sent interface, a server error sometimes (but not always) occurs.

Traceback (most recent call last):
  File "/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/venv/lib/python3.8/site-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/app/TWLight/view_mixins.py", line 320, in dispatch
    return super(NotDeleted, self).dispatch(request, *args, **kwargs)
  File "/app/TWLight/view_mixins.py", line 175, in dispatch
    return super(PartnerCoordinatorOrSelf, self).dispatch(request, *args, **kwargs)
  File "/app/TWLight/view_mixins.py", line 245, in dispatch
    return super(ToURequired, self).dispatch(request, *args, **kwargs)
  File "/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 97, in dispatch
    return handler(request, *args, **kwargs)
  File "/app/TWLight/applications/views.py", line 1026, in post
    return super(EvaluateApplicationView, self).post(request, *args, **kwargs)
  File "/venv/lib/python3.8/site-packages/django/views/generic/edit.py", line 194, in post
    return super().post(request, *args, **kwargs)
  File "/venv/lib/python3.8/site-packages/django/views/generic/edit.py", line 142, in post
    return self.form_valid(form)
  File "/app/TWLight/applications/views.py", line 920, in form_valid
    app.save()
  File "/venv/lib/python3.8/site-packages/reversion/revisions.py", line 322, in do_revision_context
    return func(*args, **kwargs)
  File "/app/TWLight/applications/models.py", line 174, in save
    super(Application, self).save(*args, **kwargs)
  File "/venv/lib/python3.8/site-packages/django/db/models/base.py", line 745, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/venv/lib/python3.8/site-packages/django/db/models/base.py", line 793, in save_base
    post_save.send(
  File "/venv/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 173, in send
    return [
  File "/venv/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 174, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
  File "/app/TWLight/applications/signals.py", line 210, in post_revision_commit
    authorization.save()
  File "/venv/lib/python3.8/site-packages/django/db/models/base.py", line 745, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/venv/lib/python3.8/site-packages/django/db/models/base.py", line 769, in save_base
    pre_save.send(
  File "/venv/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 173, in send
    return [
  File "/venv/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 174, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
  File "/app/TWLight/users/signals.py", line 29, in validate_authorization
    instance.full_clean()
  File "/venv/lib/python3.8/site-packages/django/db/models/base.py", line 1222, in full_clean
    raise ValidationError(errors)

Exception Type: ValidationError at /applications/evaluate/7307/
Exception Value: {'__all__': ['Authorization authorizer must be a coordinator or staff.']}

The root issue seems to be this function, called when the app attempts to create the Authorization object: https://github.com/WikipediaLibrary/TWLight/blob/e63a84db6a1a32c74b211134bddc403103fb0d4e/TWLight/users/helpers/validation.py#L27. The core error here could actually be the one fixed in T257667, so a rebase might be worthwhile before investigating further.

I haven't been able to replicate this in my local build. This PR is already rebased to changes in T257667, so I'm not sure how to proceed with that one.

I think this error has been introduced when I upgraded the django-autocomplete-light library that uses Select2 to create autocomplete select tags. Unfortunately, I cannot downgrade the version of that library because it uses a library deprecated in Django 3.0. I can raise an issue in its GitHub repository and check if that's a quick fix or not.

Sure! I don't think this is a blocker to merging if it's not an easy fix. The text is still readable and this view is coordinator/staff-only.

I haven't been able to replicate this in my local build. This PR is already rebased to changes in T257667, so I'm not sure how to proceed with that one.

I've figured out the issue. I was using an account (Samwalton9) which is flagged only as a Superuser. The account doesn't have the Staff flag or the Coordinator user group. It looks like I was able to see the review pages due to the Superuser flag, but when it came to checking my account's eligibility to create an authorization, I failed due to not being a Staff or Coordinator. This is a weird edge case due to, effectively, an improperly configured staff account. Not a blocker :)