Page MenuHomePhabricator

Payments-wiki sessions timing out quickly?
Closed, ResolvedPublic

Description

see for example contribution tracking id 39012241 in payments-adyen from 10/25, all on payments1001. They get to our site at 10:04:16, open the iframe at 10:04:56 and get redirected back from the CC processor at 10:07:28. Unfortunately their session is dead - Resultswitcher: 39012241.0 SHOULD BE FORBIDDEN. Reason: No active donation in the session - and they get a new ct_id 39012552.

39048372.0 on payments1003 had a session die between requests at 20:19:42 and 20:24:09.
39037084.1 on payments1002 died between 16:54:37 and 16:56:50

Event Timeline

Ejegg created this task.Oct 25 2016, 10:08 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 25 2016, 10:08 PM

There's definitely an uptick in these over last year:
zgrep -c 'in the session' payments-globalcollect-201510*.gz
shows an average around 10 a day, with the maximum 25. for this year the average is closer to 35 and hit 74 on 10/13 and 10/15.

These donors often go on to create duplicate donations.

DStrine moved this task from Triage to FR-Ops on the Fundraising-Backlog board.Oct 26 2016, 10:42 PM
cwdent added a subscriber: cwdent.EditedOct 28 2016, 10:47 PM

Appears that most of https://phabricator.wikimedia.org/T147378#2731929 is due to this bug also

MBeat33 added a subscriber: MBeat33.Nov 1 2016, 6:10 PM
Jgreen added a comment.Nov 7 2016, 8:18 PM

I don't think we're hitting memcache LRU, judging from the stats we collect in ganglia we generally don't come close to the memory limit. However since we have plenty of RAM and we're configured with a very small memcache pool I bumped the queue size from 64*4-hosts to 256*3-hosts, so we ~really~ should never see active sessions pushed out.

Ejegg noticed a drop in memcache utilization that seemed to correlate to our last major payments mediawiki upgrade, which suggests it could be related to the software change.

Jgreen closed this task as Resolved.Nov 16 2016, 5:40 PM

Closing because AFAICT we've reigned in the problem as well as we can.