Enourmous mailman3 outgoing queue
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Legoktm
	May 31 2021, 6:58 AM

Description

Screenshot_2021-05-30 Mailman3 - Grafana.png (955×1 px, 69 KB)

We got a bunch of bounces, those users were unsubscribed (on the order of like 7k!) and then for each one, mailman3 sends an email to the list owner saying the address was unsubscribed and then to the address itself, which probably results in another bounce. There's also some issue when a list owner is bouncing.

The outgoing queue is also processing very slowly, I think there's another bad/unoptimized query here, it's sending 1 email every 5 seconds...at this rate it'll take 70 hours to empty the queue.

We should just delete all the "You have been unsubscribed" emails out of the queue. I don't know if those can just be deleted off of disk or if they need to be removed from the database too. maybe there's a REST API method for it.

Related Objects

Mentioned Here: T282348: Mailman3 bounce runner is running very slowly

Event Timeline

Legoktm triaged this task as High priority.May 31 2021, 6:58 AM

Legoktm created this task.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 31 2021, 6:58 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-31T07:23:00Z] <legoktm> deleting all outgoing list mail that has a subject that starts with "You have been unsubscribed from the" T284003

Mentioned in SAL (#wikimedia-operations) [2021-05-31T07:30:45Z] <legoktm> deleted all outoing list mail that is for a yahoo/aol address being unsubscribed T284003

Mentioned in SAL (#wikimedia-operations) [2021-05-31T07:32:04Z] <legoktm> deleted all outoing list mail that is for a gmail address being unsubscribed T284003

I deleted all messages in the outgoing queue that matched:

should_delete = subject.startswith('You have been unsubscribed from the') or \
    '@yahoo.com> unsubscribed from' in subject or \
    '@yahoo.com unsubscribed from' in subject or \
    '@gmail.com> unsubscribed from' in subject or \
    '@gmail.com unsubscribed from' in subject or \
    '@aol.com> unsubscribed from' in subject or \
    '@aol.com unsubscribed from' in subject

which brought us down from 25k to 5k messages. The bounce runner crashed again, so it won't add any more messages into the outgoing queue, just normal mail traffic will. I think it's OK to let the bounce queue to grow large until we figure out what to do with it...tomorrow.

Maintenance_bot added a project: SRE.May 31 2021, 7:45 AM

RhinosF1 added a subscriber: RhinosF1.May 31 2021, 8:25 AM

Urbanecm added a subscriber: Urbanecm.May 31 2021, 10:54 AM

It has somewhat recovered but still really high
https://grafana.wikimedia.org/d/GvuAmuuGk/mailman3?orgId=1&from=1622435134965&to=1622472934965

Nemo_bis added a subscriber: Nemo_bis.Jun 1 2021, 3:48 AM

So we were running into an exception T282348#7124014, and when the bounce runner crashed, it rolled back the transaction and un-unsubscribed all the users (but obviously couldn't rollback the email notifications). Then I'd restart the bounce runner and it would reprocess everything, send out more emails, but actually make no progress.

The crash is hotpatched now and the out queue has mostly recovered (under 1k) and should be 0 in an hour or so.

Filed upstream: https://gitlab.com/mailman/mailman/-/issues/909

	F34476032: Screenshot_2021-05-30 Mailman3 - Grafana.png
	May 31 2021, 6:58 AM

Enourmous mailman3 outgoing queueClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Enourmous mailman3 outgoing queue
Closed, ResolvedPublic
Actions