Received a server error when attempting to update status of a partner to waitlisted
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Invalid | None | T170977 Post-migration issues and priorities | |||
| Resolved | jsn.sherman | T173990 Waitlisting partner with pending applications results in server error |
Event Timeline
Comment Actions
Just to clarify - does this happen for one partner or all? And was wait listing the only action you took?
Comment Actions
Waitlisting was the only action yes, but at the moment out of a few I tested the problem only occurs on Newspaperarchive (which conveniently is the only one I actually want to change right now...).
Comment Actions
It looks like the issue happens when you set a partner with pending applications to waitlisted; it appears that this is related to the code that sends waitlist notifications.
Comment Actions
I've identified that the root cause of this and several other recent showstopping bugs is inadequate testing. I realized that the existing unit tests became inadequate after I wrapped a large portion of the views with login_required; the current tests don't run any of the middleware that would give them the context required to turn up the kind of problems we're encountering. I'm moving a large portion of our tests from factory-based unit tests to client-based integration tests.
I've been mired in bug squashing for the last few weeks. In this situation, spending some time bringing the test suite back in alignment with the platform will allow me to resume my previous rate of improvement to the platform, without causing so many errors for our editors and coordinators.
Comment Actions
I believe I have this issue resolved in the codebase, but there have been a lot of changes to the puppet module to support the testing improvements, so there are some steps I need to take to make sure deploying all of the changes together won't break the production system.
Comment Actions
The updated code has been deployed, but the issue isn't resolved. Some of the automated tests that are passing without issue in dev and staging are failing out in production. This is ... interesting. I'm going to pull down a fresh dump of the production db and see if I can reproduce the issue in another environment. At least we have the tooling in place now to ferret out things kinds of issues.
Comment Actions
So, I'm unable to reproduce this issue anywhere besides production, using the same code checkout and the same data. I spent some time working on some of these individual issues, but the big question is why we're seeing them only in production. I'm going to redeploy the production server tomorrow, to see if that resolves the issue.
Creating test database for alias 'default'...
..........F...F..E..........F...F..F......EE.......................FFFFFFFFF.............................................F................................................
======================================================================
ERROR: test_editor_cannot_see_other_editor_page (TWLight.users.tests.ViewsTestCase)
Editors cannot see other editors' pages.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/www/html/TWLight/TWLight/users/tests.py", line 146, in test_editor_cannot_see_other_editor_page
_ = views.EditorDetailView.as_view()(request, pk=self.editor2.pk)
File "/home/www/TWLight/local/lib/python2.7/site-packages/django/views/generic/base.py", line 71, in view
return self.dispatch(request, *args, **kwargs)
File "/var/www/html/TWLight/TWLight/view_mixins.py", line 58, in dispatch
messages.add_message(request, messages.WARNING, 'You must be a '
File "/home/www/TWLight/local/lib/python2.7/site-packages/django/contrib/messages/api.py", line 27, in add_message
raise MessageFailure('You cannot add messages without installing '
MessageFailure: You cannot add messages without installing django.contrib.messages.middleware.MessageMiddleware
======================================================================
ERROR: test_request_application_view_context_1 (TWLight.resources.tests.WaitlistBehaviorTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/www/html/TWLight/TWLight/resources/tests.py", line 372, in test_request_application_view_context_1
self.assertEqual(response.context['any_waitlisted'], True)
File "/home/www/TWLight/local/lib/python2.7/site-packages/django/test/utils.py", line 68, in __getitem__
raise KeyError(key)
KeyError: 'any_waitlisted'
======================================================================
ERROR: test_request_application_view_context_2 (TWLight.resources.tests.WaitlistBehaviorTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/www/html/TWLight/TWLight/resources/tests.py", line 392, in test_request_application_view_context_2
self.assertEqual(response.context['any_waitlisted'], False)
File "/home/www/TWLight/local/lib/python2.7/site-packages/django/test/utils.py", line 68, in __getitem__
raise KeyError(key)
KeyError: 'any_waitlisted'
======================================================================
FAIL: test_terms_page_displays (TWLight.users.tests.TermsTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/www/html/TWLight/TWLight/users/tests.py", line 598, in test_terms_page_displays
self.assertEqual(response.status_code, 200)
AssertionError: 301 != 200
======================================================================
FAIL: test_anon_user_cannot_see_editor_details (TWLight.users.tests.ViewsTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/www/html/TWLight/TWLight/users/tests.py", line 116, in test_anon_user_cannot_see_editor_details
self.assertEqual(response.status_code, 302)
AssertionError: 301 != 302
======================================================================
FAIL: test_approved_app_page_includes_not_available (TWLight.resources.tests.PartnerModelTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/www/html/TWLight/TWLight/resources/tests.py", line 283, in test_approved_app_page_includes_not_available
self.assertContains(response, partner.company_name)
File "/home/www/TWLight/local/lib/python2.7/site-packages/django/test/testcases.py", line 358, in assertContains
msg_prefix + "Couldn't find %s in response" % text_repr)
AssertionError: Couldn't find 'Publisher McPubface' in response
======================================================================
FAIL: test_rejected_app_page_includes_not_available (TWLight.resources.tests.PartnerModelTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/www/html/TWLight/TWLight/resources/tests.py", line 270, in test_rejected_app_page_includes_not_available
self.assertContains(response, partner.company_name)
File "/home/www/TWLight/local/lib/python2.7/site-packages/django/test/testcases.py", line 358, in assertContains
msg_prefix + "Couldn't find %s in response" % text_repr)
AssertionError: Couldn't find 'Publisher McPubface' in response
======================================================================
FAIL: test_sent_app_page_includes_not_available (TWLight.resources.tests.PartnerModelTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/www/html/TWLight/TWLight/resources/tests.py", line 257, in test_sent_app_page_includes_not_available
self.assertContains(response, partner.company_name)
File "/home/www/TWLight/local/lib/python2.7/site-packages/django/test/testcases.py", line 358, in assertContains
msg_prefix + "Couldn't find %s in response" % text_repr)
AssertionError: Couldn't find 'Publisher McPubface' in response
======================================================================
FAIL: test_bogus_applications_parameter_handled (TWLight.applications.tests.BatchEditTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/www/html/TWLight/TWLight/applications/tests.py", line 2244, in test_bogus_applications_parameter_handled
self.assertEqual(response.status_code, 302)
AssertionError: 301 != 302
======================================================================
FAIL: test_missing_params_raise_http_bad_request (TWLight.applications.tests.BatchEditTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/www/html/TWLight/TWLight/applications/tests.py", line 2194, in test_missing_params_raise_http_bad_request
self.assertEqual(response.status_code, 400)
AssertionError: 200 != 400
======================================================================
FAIL: test_only_coordinators_can_batch_edit (TWLight.applications.tests.BatchEditTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/www/html/TWLight/TWLight/applications/tests.py", line 2267, in test_only_coordinators_can_batch_edit
self.assertEqual(response.status_code, 302)
AssertionError: 301 != 302
======================================================================
FAIL: test_sets_date_closed (TWLight.applications.tests.BatchEditTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/www/html/TWLight/TWLight/applications/tests.py", line 2352, in test_sets_date_closed
self.assertEqual(self.application.date_closed, date.today())
AssertionError: None != datetime.date(2017, 10, 12)
======================================================================
FAIL: test_sets_days_open (TWLight.applications.tests.BatchEditTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/www/html/TWLight/TWLight/applications/tests.py", line 2331, in test_sets_days_open
self.assertEqual(self.application.days_open, 3)
AssertionError: None != 3
======================================================================
FAIL: test_sets_status (TWLight.applications.tests.BatchEditTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/www/html/TWLight/TWLight/applications/tests.py", line 2310, in test_sets_status
self.assertEqual(self.application.status, Application.APPROVED)
AssertionError: 0L != 2
======================================================================
FAIL: test_sets_date_closed (TWLight.applications.tests.EvaluateApplicationTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/www/html/TWLight/TWLight/applications/tests.py", line 2153, in test_sets_date_closed
self.assertEqual(self.application.date_closed, date.today())
AssertionError: None != datetime.date(2017, 10, 12)
======================================================================
FAIL: test_sets_days_open (TWLight.applications.tests.EvaluateApplicationTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/www/html/TWLight/TWLight/applications/tests.py", line 2133, in test_sets_days_open
self.assertEqual(self.application.days_open, 3)
AssertionError: None != 3
======================================================================
FAIL: test_sets_status (TWLight.applications.tests.EvaluateApplicationTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/www/html/TWLight/TWLight/applications/tests.py", line 2113, in test_sets_status
self.assertEqual(self.application.status, Application.APPROVED)
AssertionError: 0L != 2
======================================================================
FAIL: test_missing_session_key (TWLight.applications.tests.SubmitApplicationTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/www/html/TWLight/TWLight/applications/tests.py", line 691, in test_missing_session_key
self.assertEqual(response.status_code, 302)
AssertionError: 301 != 302
----------------------------------------------------------------------
Ran 170 tests in 12.517s
FAILED (failures=15, errors=3)
Destroying test database for alias 'default'...Comment Actions
Freshly deployed production does the same thing. At this point there are only a very small number of differences in code execution between vagrant and production. This outcome tells me that's where I need to look, even though the two don't appear related at first glance.
Comment Actions
I finally managed to reproduce this issue in another environment and resolve it. There were several issues contributing to this problem. There was a missing email template, and separate worker and proxy timeouts due to our current email strategy. All this was made more difficult to troubleshoot due to the differences in email configuration between production and non-production environments (we only really want to send email from prod), and the configuration of the test suite in different environments.
Everything except async email T149436 is now sorted out well enough for now.