tl;dr: Being iterative is okay; We can make more graphs for more context later, but I still want this one more or less as described.

I have also granted admin to @Ejegg, and bureaucrat to @Pcoombe on donate wiki. Done.

I have removed @awight 's admin privs on donate wiki.

@dpatrick : Yes, I'm using google authenticator.

I think what happened, is that fundraising tech / fundraising ops generally use WMF-NDA, and I am definitely in that one.

When we investigate this further, please make sure we drill down into these two points:

• When users experience this problem, are they logged in, or logged out?
• Are the banners that appear post-close, part of the same campaign that was dismissed?

Approved!

@Ejegg Yes, I was envisioning Phase 1 being not SmashPig as it is now, but SmashPig as all the meaty payments integration stuff that currently lives in DI. Sounds like your Phase 1 and 2 are more specific bits of my phase 1, so I have to add 1 to the rest of my phase numbers to square it all up.

The way I figure it, we should break this work up into phases necessary to complete before we can comfortably start trying to find people who might use these libraries, without *them* going crazy / getting fed up / never speaking to us again. Off the top of my head, that looks something like this:

I suspect very strongly that the plan was not followed for the email announcement that went out today. What happened?

I get these emails, too. Are we all owners?

I'm seeing signs that this happened again last night at the same hour.
Can someone remind me why we didn't think the activity in the server admin logs was relevant?

Update: Just kidding! This spike that was landing in an unknown category, is largely "MGAnnual_email". Apparently we decided to split this medium out from the regular "email" utm_medium, which probably makes sense. Wouldn't recommend changing it.

I have done some archaeology in my own old inbox, and here are some things that I found which should help track down what we did back in 2012 to fix what sounds like an identical issue we had back then:

• There are probably actually two things going on. The first is that we are expected to retry until one works. The old listener did in fact include a pause between retries, of 250,000 microseconds (better known as 1/4 second).
• There is a second issue going on, in which instead of a VERIFIED or INVALID, you receive an entire webpage. This was much more difficult to solve.
  • The patch to fix this issue with the old listener was committed at some point between 10/4 and 10/24/2012, probably by user "pgehres". If you have access to the old repo, you should be able to find what we did right around that date.
  • Whatever we did, it had something to do with encoding the response, and according to my emails I clearly didn't think it was going to solve the problem. I was wrong.
• I still suspect that they have something fishy in their server pool. But the fact that it's been broken for exactly this way for at least four years, makes me think they have no interest or reason to fix the thing.

Check for enabled 3DS currencies in LocalSettings. I believe there's an array we can alter fairly trivially if enabling doesn't hurt conversion too much.

@Aklapper @DStrine I can verify that this is indeed Eileen.

@RobLa-WMF - Thanks for getting involved on this one. Looks like we might need some additional assistance chasing this down before November.

@Aklapper: I can confirm that @XenoRyet is indeed who he says - He has been on the fundraising engineering team for a while now. Can I help unblock this?

I approve this access request.

Approved!

FormChooser settings?

Definitely a techsupport ticket thing.

Also: We can absolutely grep for the forbidden log message in our logs. So, we totally have a way of finding out how many people are getting this.

Bread Crumb #1 - This can happen on the payments cluster if something happens to the user's session between the time they leave for GC, and the time they come back. This can also happen if the user tries to hit the resultswitcher twice for their one payment. Either way, there should be a log trail on our end.

Good news, everyone! The queries have landed. 3,512,665 remain as of Nov 12 21:52:12 UTC.

Queries Remaining still doesn't reflect the main... payload. Can I call it a payload?
Why not.
Still waiting.

Yes, but it's all phenomenally annoying.

Will a single person still be able to do things like quickly disable components on the payments cluster in case a third party goes down or starts emitting garbage? If not, I might take some convincing, even though I love the idea of having this in version control.

I approve this request! Thanks everybody.

I did, yes. But I was not able to get the patch I was working on, to a place where I could commit something mergeable before I left.
This task should probably be broken down a bit before we take another swipe at it.

Approved!

All it requires is her email address. :) Just throw that in there and she should be able to do the rest, if she can access her wmf email.

Nope: Probably just me.
However: There is a password reset link on the login page. Is that not working?

@atgo Looks like her account was never disabled. Probably just needs the cert.

Hey, look! I'm Helping: