Page MenuHomePhabricator

QueueWrapper should fall back to damaged/retry message table when redis connection is dead
Closed, ResolvedPublic

Description

If the Redis connection dies when we are doing something long-running like audit parsing or billing recurring donations, we just drop a donation on the floor before crashing. It would be neat to instead write that donation to the damaged message table (with a retry date) so that the message requeue job picks it up a little later.

We could put this logic in the QueueWrapper::push function as long as it has a way to tell whether the damaged message table is available. For example, on the IPN listener and the payments server it is not accessible.

Event Timeline

Change 821320 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/SmashPig@master] Option for queue push to fall back to db

https://gerrit.wikimedia.org/r/821320

greg triaged this task as Medium priority.Aug 12 2022, 8:39 PM

Change 821320 merged by jenkins-bot:

[wikimedia/fundraising/SmashPig@master] Option for queue push to fall back to db

https://gerrit.wikimedia.org/r/821320

Change 825869 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/crm@master] Use QueueWrapper fallback to db in audit + recur charge

https://gerrit.wikimedia.org/r/825869

Change 825869 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Use QueueWrapper fallback to db in audit + recur charge

https://gerrit.wikimedia.org/r/825869

Ejegg claimed this task.