Making a task since this is still happening. Some IPNS are running into timeouts but not all, donations are being completed.
Starting Friday afternoon US time (I'll hunt down the exact time) we started getting failmail saying:
2021-12-11T20:01:21+00:00 [INFO] Starting processing for request, configuration view: 'amazon', action: 'listener'
2021-12-11T20:01:21+00:00 [INFO] Starting processing of listener request from 72.21.217.44
2021-12-11T20:01:21+00:00 [INFO] No IP whitelist specified. Continuing and not validating remote IP '72.21.217.44'.
2021-12-11T20:01:21+00:00 [INFO] (RawData) Incoming message (raw)
2021-12-11T20:01:52+00:00 [ALERT] Error validating Amazon message. Firewall problem?
Looking at more of the logs the full error message is:
Dec 11 00:03:55 frpig1001 SmashPig: SPCID-0851076298 | Error validating Amazon message. Firewall problem? | |
Dec 11 00:03:55 frpig1001 SmashPig: SPCID-0851076298 | Message denied by security policy, death is me. | #0 /srv/www/org/wikimedia/listeners/SmashPig/Core/Listeners/RestListener.php(16
): SmashPig\PaymentProviders\Amazon\AmazonListener->parseEnvelope(Object(SmashPig\Core\Http\Request))
Dec 11 00:03:55 frpig1001 SmashPig: #1 /srv/www/org/wikimedia/listeners/SmashPig/Core/Http/RequestHandler.php(101): SmashPig\Core\Listeners\RestListener->execute(Object(SmashPig\Core\H
ttp\Request), Object(SmashPig\Core\Http\Response))
Dec 11 00:03:55 frpig1001 SmashPig: #2 /srv/www/org/wikimedia/listeners/SmashPig/PublicHttp/smashpig_http_handler.php(13): SmashPig\Core\Http\RequestHandler::process()
Dec 11 00:03:55 frpig1001 SmashPig: #3 {main} | SmashPig\Core\Listeners\ListenerSecurityException@/srv/www/org/wikimedia/listeners/SmashPig/PaymentProviders/Amazon/AmazonListener.php:6
4 (Unable to post request, underlying exception of Failed to connect to sns.us-east-1.amazonaws.com port 443: Connection timed out Debug logging: * Expire in 0 ms for 6 (transfer 0x557
8f82bd250)
Dec 11 00:03:55 frpig1001 SmashPig: * Added mws.amazonservices.com:443:34.233.223.241 to DNS cache
Dec 11 00:03:55 frpig1001 SmashPig: * Expire in 1 ms for 1 (transfer 0x5578f82bd250)
(the above line repeated until)
Dec 11 00:03:55 frpig1001 SmashPig: * Trying 209.54.182.11...
Dec 11 00:03:55 frpig1001 SmashPig: * TCP_NODELAY set
Dec 11 00:03:55 frpig1001 SmashPig: * Expire in 200 ms for 4 (transfer 0x5578f82bd250)
Dec 11 00:03:55 frpig1001 SmashPig: * connect to 209.54.182.11 port 443 failed: Connection timed out
Dec 11 00:03:55 frpig1001 SmashPig: * Failed to connect to sns.us-east-1.amazonaws.com port 443: Connection timed out
Dec 11 00:03:55 frpig1001 SmashPig: * Closing connection 0
Dec 11 00:03:55 frpig1001 SmashPig: )
Dec 11 00:03:55 frpig1001 SmashPig: SPCID-0851076298 | Finished processing listener request | |
The IPN messages that this error happens on are resent and are processed correctly.