Page MenuHomePhabricator

Ingenico: server timeout message for successful contribs? 7/21/2020
Closed, ResolvedPublic

Description

Kris Hancock noticed right after our small Japan email send that multiple donors were reporting errors. When we look them up at Ingenico, they don't know that they've donated successfully, so they make multiple attempts. Some donors report seeing "server/time out" messaging.

This feels cluster-y because of volume. Can we tell by looking at the logs whether this is an issue on our side, or something that Ingenico can fix? We're following up with donors individually, so some of these transactions may be refunded or canceled now, but the common thread seems to be donors not knowing their first attempts succeeded.

donortickettransactions
17680377000245572, 7000245636, 7000245709, 7000245791
27680407000245937, 7000246005, 7000246029
3768049no attempts @ console (or maybe different email address used): "After I entered the credit card information to make a donation, I sent it, but I was told that the site could not be opened."
47680357000244950, 7000245068 (after two status 100 rejections)
57680557000246684, 7000246698 "the page timed out when I was attempting to donate and I clicked “refresh”"

Event Timeline

MBeat33 created this task.Jul 21 2020, 3:20 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 21 2020, 3:20 AM
MBeat33 updated the task description. (Show Details)Jul 21 2020, 3:23 AM
Ejegg added a subscriber: Ejegg.Jul 21 2020, 4:11 AM

Dang - could this have anything to do with the TY page switchover? It loads fine for me, but I notice that the DNS resolution for donate.wikipedia.org is slightly different from that of donate.wikimedia.org and therefore might have different results in different parts of the world.

Have any reports come in with a screenshot showing which domain the timeout / load failure is on?

Ejegg added a comment.Jul 21 2020, 4:38 AM

Just looked briefly so far, and donors 1 and 2 have multiple lines in the logs showing a redirect to the new TY page, meaning everything was successful on payments-wiki. I tried some free tools to test page load in different countries and the new TY URL seems to load fine from Japanese test servers.

#768065 / CID 11090563

It's possible we have just found an example of one of these who used PayPal. I'm not 100% sure it's the same error, but it sounds similar, and this donor donated three times in error: "I received an e-mail requesting donation earlier and paid 5200 yen with PayPal, but at that time I got a server error screen display, I thought that the donation could not be completed, and after that I paid the donation about twice I tried to process it, but all of them resulted in a server error."

I tried the TY page from Japan using a VPN, and it looked fine

krobinson added a comment.EditedJul 21 2020, 12:42 PM

Not sure if this is related, but as you mentioned DNS, @Ejegg - we may have a screenshot:

#768146 / CID 140690. This one was via PayPal, NL donor.

It doesn't seem to be the same 'server / time out' error message, but I'm wondering if it's part of the same problem.

Ah, something is redirecting mobile devices to the .m.wikipedia.org domain. I can reproduce on my phone.

mepps added a subscriber: mepps.Jul 21 2020, 2:37 PM

For now, I've rolled back the thankyou page going to donate.wikipedia.org.

MBeat33 added a comment.EditedJul 21 2020, 2:38 PM

Thank you @mepps We're up to 21 cases, and these are high-touch tickets with lots of refunds and back and forth with donors.

@Pcoombe given your discovery, is it still important to ask donors which domain the timeout / load failure is on?

mepps added a comment.Jul 21 2020, 3:08 PM

Traffic has a patch up for this. We'll just need to wait for them to deploy it. https://gerrit.wikimedia.org/r/c/operations/puppet/+/615229

Thank you for working on a fix for this, fr-tech team! From my understanding, the issue came from fixing this phabtask https://phabricator.wikimedia.org/T251780#6295730 is that right?

Is there a step in the QA process that we could add in order to avoid this type of issue to come up again? Would love your thoughts @DStrine and @XenoRyet. Thank you!

Ejegg added a comment.Jul 21 2020, 7:01 PM

Is there a step in the QA process that we could add in order to avoid this type of issue to come up again? Would love your thoughts @DStrine and @XenoRyet. Thank you!

We should test things on mobile too whenever we make a big change like this.

DStrine closed this task as Resolved.Jul 21 2020, 7:59 PM

@Ejegg and @DStrine - We should test things on mobile too whenever we make a big change like this.

sounds like. a good idea to me! Has this been added to the QA process going forward? TY!

We have seen more donors reporting this again after today's email send. A handful of donors reached out to us and we refunded duplicates, and a scan of around 8k of the donations threw up 20 or so more duplicate donations. Nowhere near the volume we saw last week, but it feels like something is still amiss:

#770436 / CID 35144540 / iPhone X/iOS 13.6/Safari browser / "I have donated I believe but it took me to a Wikipedia page that then showed an error."
#770398 / CID 4341830 "I did the donation procedure earlier, but an error message was displayed and a message instructing me to process again after a while was displayed. When I went through the procedure again as instructed, the error was displayed again and the process was interrupted, but I received two emails saying that the acceptance of the donation was completed." (duplicate refunded)
#770660 / CID 11044678 / "There was an error when making a payment I thought again that it was unsettled I got an error again However, I received two receipt emails from them, and both have different receipt numbers." (duplicate refunded)
#770648 / CID 35479279 / "I am trying to donate Yen, but cannot. What should I do?" (4 successful payments, 3 refunded).

No screenshots thus far.

Ejegg added a comment.Jul 28 2020, 2:52 PM

Nothing helpful in the logs for these 4, unfortunately

CID 35144540 was monthlyConvert - we're using the same logic to get the TY page URL, so it should be the new address. However we can't really log server-side as we redirect client-side. No entries in the error logs

CID 4341830 was also on mobile, but NOT monthlyConvert, and both of their donation logs end with "Displaying thank you page https://donate.wikipedia.org/wiki/Thank_You/ja?country=JP for successful paymentResult".

CID 11044678 was on linux, NOT monthlyConvert, and both donation logs end with "Displaying thank you page https://donate.wikipedia.org/wiki/Thank_You/ja?country=JP for successful paymentResult".

CID 35479279 on mobile, NOT monthlyConvert, all donation logs end in redirect to TY page as above.

Thanks @Ejegg !

WE also proactively refunded all these ones that were duplicates. Not sure if more examples are helpful...?
Ingenico order ID:
7000336665
7000325872
7000330263
7000323237
7000331765
7000331122
7000325630
7000322668
7000330592
7000327377
7000327543
7000330305
7000328103
7000322277
7000324069
7000321311
7000321024
7000325913
7000326939
7000330541
7000331661
7000328191
7000321149
7000328778

DStrine reopened this task as Open.Jul 28 2020, 3:26 PM
DStrine moved this task from Current Sprint to Triage on the Fundraising-Backlog board.
SHust added a subscriber: SHust.Jul 28 2020, 3:28 PM
mepps added a comment.Jul 28 2020, 5:36 PM

@Ejegg if they're getting to the thank you page, isn't it likely that the issue is with donate.wikipedia.org?

DStrine closed this task as Resolved.Jul 28 2020, 7:40 PM
DStrine claimed this task.

I am splitting out the latest list of issues on this task: T259070 and closing this task again for now.

Thanks @Ejegg !

WE also proactively refunded all these ones that were duplicates. Not sure if more examples are helpful...?
Ingenico order ID:
7000336665
7000325872
7000330263
7000323237
7000331765
7000331122
7000325630
7000322668
7000330592
7000327377
7000327543
7000330305
7000328103
7000322277
7000324069
7000321311
7000321024
7000325913
7000326939
7000330541
7000331661
7000328191
7000321149
7000328778

Hi All, I had Sielo test a transaction on donate wiki in JP. He was able to get a TY page: https://donate.wikipedia.org/w/index.php?title=Thank_You&country=JP

Seilo in JP was able to get to the TY page in a test transaction or two today.