Looking at the stack trace from the access_denied_donation_confirmation errors, we can see the error happening when the GetDonationUseCase returns a "not allowed" response. Looking at GetDonationUseCase shows that there can be two triggers for this:
- Access check failed
- Donation was not found or donation was deleted
Looking at existing Donations (with IDs from the error messages) shows that the donations exist and are not deleted. That means that it's probably the access check failed
Looking at the donations of all the logs where we get access_denied_donation_confirmation errors, we can see that they have one thing in common: they were created before 2023-10-26. This is the date where we deployed a changed access token storage system, see https://phabricator.wikimedia.org/T341795
The new system uses DoctrineTokenRepository to look for a token in the table legacy_auth_tokens instead of storing/retrieving them from the "data blob". When we deployed, we did not migrate the tokens from the data blob, but instead created the class FallbackTokenRepository, which wraps DoctrineTokenRepository, checking the data blob of donation/membership tables in case the tokens are not in legacy_auth_tokens. It's very likely that the source of the error is in this class.
The investigation in this ticket would be to
- Copy all or parts of the production database (a section with "old" donations) into your local dev environment. Remember to also copy the content from the corresponding payment tables.
- Debug FallbackTokenRepository (using debugger or PHP functions like error_log, var_dump and die) to check the data flow/checks to see why the check fails.
A fix would involve code and test changes for FallbackTokenRepository to make the token retrieval more robust.