Page MenuHomePhabricator

Rate-limit is too harsh and affects human users
Closed, ResolvedPublic

Subscribers
Tokens
"Evil Spooky Haunted Tree" token, awarded by Tbayer."Love" token, awarded by Jdlrobson."Love" token, awarded by Niedzielski."The World Burns" token, awarded by jcrespo."Orange Medal" token, awarded by Krinkle.
Assigned To
Authored By
Jc86035, Jul 6 2018

Description

Phabricator rate-limited me by blocking my IP address while I was writing my last comment. I am probably a human, so this probably shouldn't be happening. Presumably the rate limit was created in response to T198552, although this has clearly had some collateral damage (see T198915).

Error messages:

TOO MANY REQUESTS
You ("$ip") are issuing too many requests too quickly.
TOO MANY CONCURRENT CONNECTIONS
You ("$ip, 10.192.16.138, 10.192.16.138, 10.192.16.138") have too many concurrent connections.

Proposals mentioned:

  • Make rate limit more tolerant
  • Make rate limit only affect certain actions

Event Timeline

Jc86035 created this task.Jul 6 2018, 3:39 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 6 2018, 3:39 PM
Isarra added a subscriber: Isarra.Jul 7 2018, 6:01 PM

I started my browser and most of my phabricator tabs turned into complaints about too many concurrent connections. Is this the same thing?

TOO MANY CONCURRENT CONNECTIONS
You ("<ip>, 10.192.16.138, 10.192.16.138, 10.192.16.138") have too many concurrent connections.
Paladox triaged this task as High priority.Jul 7 2018, 6:13 PM
Paladox added a subscriber: Paladox.

It's happening to alot of users. I think we need to move the rate limit code further into phabricator where it can tell if your a user and in a group.

Triaging as high.

Yes, it's the same thing. Maybe it's an issue with the comment preview, which is rendered every second. I'm noticing that it also reloads my profile picture at the same rate when I'm typing. I have no idea why anyone would want to do that. I got blocked again while I was writing this comment.

TOO MANY REQUESTS
You ("$ip") are issuing too many requests too quickly.

I hit this as well, and I wasn't even writing a comment or reloading a million tabs at the time. Just researching some tasks.

I got this right now as well. I was creating a task with exception crash code from AWB. I then pressed the save/create task button, and I got this message. I then tried to go back to recover the text, but it was gone. :/ All lost

Jc86035 renamed this task from Rate-limit is too harsh to Rate-limit is too harsh and affects human users.Jul 10 2018, 8:05 AM
Jc86035 updated the task description. (Show Details)
Jc86035 updated the task description. (Show Details)Jul 10 2018, 8:09 AM

Change 444810 had a related patch set uploaded (by 20after4; owner: 20after4):
[operations/puppet@production] Phabricator: Double the rate limit and connection limit

https://gerrit.wikimedia.org/r/444810

Krinkle added a subscriber: Krinkle.

I belive I think what is the source of the issues, when you write a comment, like this, it generates a live preview. This creates dozens of requests to phabricator, and I have been banned mid-writing of one- I think the rate limiting should be just based on actual number of actual writes to tickets, and not other kind of requests. I understand this may not be possible, but maybe we could not apply such a filter to a long list of trusted contributors.

Agreed that this is truly annoying. Took me a lot of time to file a report due to "concurrent connections".

Jc86035 updated the task description. (Show Details)Jul 11 2018, 12:10 PM

I just got rate limited on my home connection. This was during relatively normal use (reviewing 4 different tickets, assigning them to myself, adding some projects, and creating a parent task).

Niedzielski added a subscriber: Niedzielski.

I have been hit too :)

I think we should disable this as the account approvals is now manual meaning less likely a spammer will get through.

I belive I think what is the source of the issues, when you write a comment, like this, it generates a live preview. This creates dozens of requests to phabricator, and I have been banned mid-writing of one- I think the rate limiting should be just based on actual number of actual writes to tickets, and not other kind of requests. I understand this may not be possible, but maybe we could not apply such a filter to a long list of trusted contributors.

Can we whitelist that endpoint so it doesn't count towards the rate limit?

I belive I think what is the source of the issues, when you write a comment, like this, it generates a live preview. This creates dozens of requests to phabricator, and I have been banned mid-writing of one- I think the rate limiting should be just based on actual number of actual writes to tickets, and not other kind of requests. I understand this may not be possible, but maybe we could not apply such a filter to a long list of trusted contributors.

Can we whitelist that endpoint so it doesn't count towards the rate limit?

I don't think you can.

I submitted a patch to raise the limits. I just need someone from sre to merge.

I submitted a patch to raise the limits. I just need someone from sre to merge.

Link?

Oh, my bad, it's above.

I think we should disable this as the account approvals is now manual meaning less likely a spammer will get through.

That comment makes no sense. You cannot suddenly find out who's good or bad just because you temporarily have to approve accounts.

Change 444810 merged by Rush:
[operations/puppet@production] Phabricator: Double the rate limit and connection limit

https://gerrit.wikimedia.org/r/444810

Tbayer added a subscriber: Tbayer.

I've disabled the rate limit because even after merging 7562c262da5699d61634ffb8e4ea3aab54a0048d we still saw regular users hitting the limit. I think the rate limiting code is buggy or somehow not working as intended.

Note that the rate limiting is disabled only temporarily. We need to remove the config from puppet to make it permanent.

Change 445328 had a related patch set uploaded (by 20after4; owner: 20after4):
[operations/puppet@production] Disable phabricator rate limits

https://gerrit.wikimedia.org/r/445328

I got this right now as well. I was creating a task with exception crash code from AWB. I then pressed the save/create task button, and I got this message. I then tried to go back to recover the text, but it was gone. :/ All lost

This happened to me two times. I found out that once you can access Phabricator again, it is possible to refresh the page for the form to be resent again.That saved me to spend the time writing the task again, at least the second time.

Nikki added a subscriber: Nikki.Jul 14 2018, 8:38 PM
Cirdan added a subscriber: Cirdan.Jul 17 2018, 5:45 AM

Change 445328 merged by Jcrespo:
[operations/puppet@production] Disable phabricator rate limits

https://gerrit.wikimedia.org/r/445328

mmodell closed this task as Resolved.Jul 18 2018, 7:35 AM
mmodell claimed this task.