I see the cancel message come into our IPN listener at 11:06 AM that day, just a few minutes after a payment message on the same recurring. Since it was during Big English, the donation queues were pretty full and the cancel message got processed first. I can see in the db logs that the status went to cancelled, and then when the payment message came in we uncancelled it. I will update our uncancel logic to check the donation date.

Change #1235355 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/crm@master] Don't uncancel when messages processed out of order

https://gerrit.wikimedia.org/r/1235355

Thanks Elliott!

I do see 50 or so records that have similarly short times between cancel and uncancel - I'll set them back to cancelled.

Awesome, thank you for the quick fix @Ejegg! :)

OK, those records are back in cancelled state. Just waiting for someone to review the code change to prevent it from happening again in the future.

Change #1235355 merged by Eileen:

[wikimedia/fundraising/crm@master] Don't uncancel when messages processed out of order

https://gerrit.wikimedia.org/r/1235355

OK, the fix to check dates is deployed, and the previously-reactivated ones have been set back to cancelled.