If the orphan slayer crashes mid-process, it can cause the message being processed to be lost forever. This is a new weakness I accidentally introduced with the Redis changes.
Don't pop the top message on the queue... "peek" instead. Delete it only once the output has been committed.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Restricted Task | |||||
| Resolved | awight | T92922 Use the new queue wrapper for limbo, antimessage, and orphan slaying | |||
| Resolved | awight | T99018 Stop writing to ActiveMQ once orphan slaying on Redis has been tested | |||
| Restricted Task | |||||
| Resolved | awight | T99017 Orphan slayer reads from frack Redis rather than ActiveMQ | |||
| Resolved | awight | T107852 Fix cheesiness: be more transactional when popping messages from the limbo queue |
Change 229987 had a related patch set uploaded (by Awight):
Provide a method to peek at the top element of a queue
Change 230175 had a related patch set uploaded (by Awight):
WIP Don't delete limbo messages until we've finished processing
Change 229987 merged by Ejegg:
Provide a method to peek at the top element of a queue
Change 230175 merged by jenkins-bot:
Don't delete limbo messages until we've finished processing
Change 230948 had a related patch set uploaded (by Awight):
Don't fetch data if we're just testing for existence
Sorry @awight, I didn't review this as thoroughly as I should have. We're still deleting the messages inside the loop that fetches the orphans, before doing anything to rectify them, leaving us open to data loss if the process loop fails or times out. Is there any benefit to fetching in a batch before rectifying, or can we do a peek-rectify-delete loop?
Change 231441 had a related patch set uploaded (by Awight):
Don't delete the message until we're done with it
Change 231441 merged by jenkins-bot:
Don't delete the message until we're done with it
Change 231452 had a related patch set uploaded (by Awight):
Rectify orphans inside the fetch loop