Page MenuHomePhabricator

Ratelimit on Commons is way too aggressive/broken
Closed, ResolvedPublic

Description

Action throttled

As an anti-abuse measure, you are limited from performing this action too many times in a short space of time, and you have exceeded this limit. Please try again in a few minutes.

Stop that! I'm a patroller, my rate limit should be 10500/3 minutes. And all I'm trying to do is edit 1100 file pages or so using VisualFileChange.

  • I had 528 remaining edits (I had already been throttled several times at this point), VFC waits 3 minutes, resumes and with 479 I get throttled again.
  • VFC waits 3 minutes and with 418 remaining edits I get throttled again.
  • VFC waits 3 minutes and with 305 remaining edits I get throttled again.
  • VFC waits 3 minutes and with 232 remaining edits I get throttled again.
  • VFC waits 3 minutes and with 169 remaining edits I get throttled again.
  • VFC waits 3 minutes and with 32 remaining edits I get throttled again. Almost there!!

Details

Related Gerrit Patches:
operations/mediawiki-config : masterIncrease Commons linkpurge rate limit for patrollers

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 13 2020, 9:41 PM
AlexisJazz updated the task description. (Show Details)Feb 13 2020, 9:46 PM

I did almost nothing (2 edits) for 19 minutes. Started VFC again for some forgotten edits. 5 edits and BOOM, throttled. 3 more and BOOM, throttled.

Maybe VFC is performing null edits and the limit on that is lower or something? But that's rubbish, if I try to edit (not null-edit) right after VFC pauses, the action gets throttled.

Tgr added a subscriber: Tgr.Feb 13 2020, 10:19 PM

Hi @AlexisJazz, thanks for taking the time to report this. I don't know what VFC is and we don't know which specific steps you performed. Please see https://www.mediawiki.org/wiki/How_to_report_a_bug and provide clear steps to reproduce. Thanks.

Hi @AlexisJazz, thanks for taking the time to report this. I don't know what VFC is and we don't know which specific steps you performed. Please see https://www.mediawiki.org/wiki/How_to_report_a_bug and provide clear steps to reproduce. Thanks.

Sorry, VFC is so well known to power users on Commons that I forgot. It's https://commons.wikimedia.org/wiki/Help:VisualFileChange.js.

I had https://commons.wikimedia.org/wiki/Category:Media_from_Beeld_en_Geluid_Wiki_with_unchecked_source opened in VFC and replaced:

Less than 1000 files were selected and 76 edits ultimately resulted. I started the job at 22:07 and it finished at 23:39. Two edits failed because it took so damn long I got edit conflicts for https://commons.wikimedia.org/wiki/File:Slade_-_TopPop_1973_12.png and https://commons.wikimedia.org/wiki/File:StephenStills1972.jpg.

Tgr added a comment.Feb 14 2020, 12:09 AM

So the Commons edit ratelimit configuration is

"ip" => [ 120, 900, 8, 60 ],
"newbie" => [ 120, 300, 8, 60 ],
"user" => [ 900, 180, 90, 60 ],
"image-reviewer" => [ 10500, 180 ],
"patroller" => [ 10500, 180 ],
"autopatrolled" => [ 10500, 180 ],

(tries / seconds - no idea what are the extra two values in some of them).

If this is something similar to T234743: User rights validation is sometimes malfunctioning (with FlaggedRevs only?), then maybe you are getting the user limits (on some edits only) instead of the patroller limits. If you have more tasks lined up, can you hit the limit and then manually test a few times whether you are being reliably throttled?

I guess throttling is owned by Security?

Is the extra values maybe the array keys not merged properly? I think 8, 60 is default for ip.

I did almost nothing (2 edits) for 19 minutes. Started VFC again for some forgotten edits. 5 edits and BOOM, throttled. 3 more and BOOM, throttled.
Maybe VFC is performing null edits and the limit on that is lower or something? But that's rubbish, if I try to edit (not null-edit) right after VFC pauses, the action gets throttled.

This sounds like the rate limit key doesnt have an expiry. I think that happened once before in the past with abusefilter and nobody figured out why so we just manually deleted they key

So the Commons edit ratelimit configuration is

"ip" => [ 120, 900, 8, 60 ],
"newbie" => [ 120, 300, 8, 60 ],
"user" => [ 900, 180, 90, 60 ],
"image-reviewer" => [ 10500, 180 ],
"patroller" => [ 10500, 180 ],
"autopatrolled" => [ 10500, 180 ],

(tries / seconds - no idea what are the extra two values in some of them).
If this is something similar to T234743: User rights validation is sometimes malfunctioning (with FlaggedRevs only?), then maybe you are getting the user limits (on some edits only) instead of the patroller limits. If you have more tasks lined up, can you hit the limit and then manually test a few times whether you are being reliably throttled?

Okay, I selected 924 files in VFC. None of these would result in actual edits, but possibly null edits as VFC didn't color them yellow which I'm guessing is the result of my substitution. (pre-substitution wikitext differs from the source, so VFC will submit it, but post-substitution is identical so no actual edit is made)

With 889 edits remaining (924-889=35..?) VFC paused. I tried editing 6 different pages by hand (not null edits) and all 6 edits were rejected. I kept trying and when I was able submit edits again (made two edits) I looked at VFC: 120-something seconds remaining. So about a minute had passed and I was able to edit again.

Tgr added a comment.Feb 14 2020, 1:45 AM

Is the extra values maybe the array keys not merged properly? I think 8, 60 is default for ip.

Yeah, that must be it. I guess not much harm in it, User::pingLimiter() just takes the first two items.

This sounds like the rate limit key doesnt have an expiry. I think that happened once before in the past with abusefilter and nobody figured out why so we just manually deleted they key

Shouldn't the user/IP get locked out permanently in that case?

Shouldn't the user/IP get locked out permanently in that case?

Good point

AlexisJazz added a comment.EditedFeb 14 2020, 2:02 PM

I tried it again, selected 500 files (same stuff, just null edits) and with 467 files remaining I got throttled.

I'm now going to request Account Creator (noratelimit) because I simply can't work like this. (I can do more testing with a sock if needed)

Edit:
A note: "Well, this was introduced because of vandals editing with speed over 30000 edits per hour (this is too high). The reference is T192668 but the task is restricted because of security." (T194864, see also here)

This kind of vandalism has (to my knowledge) never been a thing on Commons to begin with. And it CERTAINLY has never been a thing with users who are autopatrolled and up. Those are NOT vandals.

  • Translation administrators had problems with the ratelimit: T155162
  • When the ratelimit for edits was first installed, Commons just broke and the limit had to be raised significantly.
  • Not all was solved, eventually people spoke up because moving pages was a problem. So we raised that limit too: T232657

And I am under the impression (but can't provide hard data to back this up) that ever since ratelimit was introduced, bulk editing became significantly slower. By which I mean without actually getting throttled. I remember how VFC would just swoosh through the files, it was beautiful. Never seen that again after May 2018.

Ratelimit is a solution in search of a problem for autopatrolled and up. It's done literally no good, only give people headaches. Just something for you to tinker about.

AlexisJazz added a comment.EditedFeb 14 2020, 8:35 PM

191 files selected in VFC. With 40 remaining (191-40=151) I got throttled. 120 actual edits were made and I guess 31 null edits.

Before this I made 190 or so edits (no null edits) without being throttled.

So I suspect it's really the null edits that get throttled at roughly 30/minute, and for some reason when you get throttled for null edits, you also get throttled for regular edits. Where's that image of Darth Vader with a jug in the ocean?

Tgr added a comment.Feb 14 2020, 8:48 PM

The purge rate limit is 30/60 so apparently that is being triggered for null edits.

Change 572339 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[operations/mediawiki-config@master] Increase Commons purge rate limit for patrollers

https://gerrit.wikimedia.org/r/572339

Tgr added a comment.Feb 14 2020, 8:55 PM

Patch is probably not ideal, as real (recursive) purges are expensive. Null edits probably should not count as purges (they are cheaper than real edits). But it works as a quick fix.

Thanks for figuring out what's going on @AlexisJazz.

Patch is probably not ideal, as real (recursive) purges are expensive. Null edits probably should not count as purges (they are cheaper than real edits). But it works as a quick fix.
Thanks for figuring out what's going on @AlexisJazz.

Thanks! I know these null edits waste some resources, but excluding those pages from my selection is very difficult. It would mean either categorizing files before I run VFC (that would be an additional edit, that's more expensive! I do this only when there's no other way) or spending like an hour or two figuring out a way every time I make a run. Between a human wasting 2 hours of their life or a CPU wasting 2 minutes of its life, I prefer putting the burden on the CPU. ;-)

With 1000/minute I can continue my work without fear of getting throttled. :-)

Btw, I guess this does mean that if you exceed any ratelimit, your entire account gets locked out, not just whatever action you were performing?

Tgr added a comment.Feb 14 2020, 10:06 PM

Btw, I guess this does mean that if you exceed any ratelimit, your entire account gets locked out, not just whatever action you were performing?

Oh, right, you were blocked from normal editing as well, not just null edits. I probably misidentified the problem then.
(Also, looking at the code, I see no obvious way how the purge throttle would be involved, although I might be missing some non-obvious way.)

Bawolff added a comment.EditedFeb 14 2020, 10:43 PM

At the very least, I think the purge rate limit should be higher than the edit rate limit.

I guess it would be possible to confirm this theory by next time Alexis is rate limited, have someone with shell access look at all the various memcache keys to see which one's are too high. Weirdly we don't seem to log throttle events (unless its login related).

(Also, looking at the code, I see no obvious way how the purge throttle would be involved, although I might be missing some non-obvious way.)

I think linkpurge throttle is more likely than purge ( https://github.com/wikimedia/mediawiki/blob/master/includes/EditPage.php#L2112 ) . linkpurge will be checked before any edit (but not incremented unless your edit is a null edit) so if you hit it, you get locked out of editing until it is reset. It is also currently set to 30 per minute.

The other question is why is this happening just now? linkpurge limits have been in place since at least 2013-ish

Ideally of course, VFC would just see that its edit is a null edit and not do it. Null edits are kind of expensive for the servers to do (Because they trigger refreshing all db tables), so avoiding doing them unintentionally would probably be good.

At the very least, I think the purge rate limit should be higher than the edit rate limit.
I guess it would be possible to confirm this theory by next time Alexis is rate limited, have someone with shell access look at all the various memcache keys to see which one's are too high. Weirdly we don't seem to log throttle events (unless its login related).

(Also, looking at the code, I see no obvious way how the purge throttle would be involved, although I might be missing some non-obvious way.)

I think linkpurge throttle is more likely than purge ( https://github.com/wikimedia/mediawiki/blob/master/includes/EditPage.php#L2112 ) . linkpurge will be checked before any edit (but not incremented unless your edit is a null edit) so if you hit it, you get locked out of editing until it is reset. It is also currently set to 30 per minute.
The other question is why is this happening just now? linkpurge limits have been in place since at least 2013-ish

I guess few people do what I do. (and not everyone reports issues) Though I have done this before, and probably made mass null edits before without issue. But, probably. Maybe I didn't. Can't say for sure.

Ideally of course, VFC would just see that its edit is a null edit and not do it. Null edits are kind of expensive for the servers to do (Because they trigger refreshing all db tables), so avoiding doing them unintentionally would probably be good.

It does, but I'm using a substitution to insert data I gathered locally and dumped into my userspace. VFC has no way to see whether or not that substitution will result in an edit or not.

Easy reproduction steps:

  • Load any category (or uploads from a user, or search, whatever) with VFC. Make sure enough (say, 100) files are loaded and select all.
  • Select "custom replace"
  • Replace "a" with "a{{subst:void}}" (without the quotation marks)
  • Execute

This results in nothing but null edits and quickly hitting the ratelimit.

Tgr added a comment.Feb 15 2020, 4:16 AM

Ideally of course, VFC would just see that its edit is a null edit and not do it. Null edits are kind of expensive for the servers to do (Because they trigger refreshing all db tables), so avoiding doing them unintentionally would probably be good.

Ideally, we'd have an API option to say "if this is a null edit, just discard it silently". Or maybe just never trigger the null edit purge logic on API edits. They were always a UI hack to provide editors with an intuitive way to refresh the page.

Given the state of EditPage code, probably better to wait until it gets refactored for MCR.

Easy reproduction steps:

  • Load any category (or uploads from a user, or search, whatever) with VFC. Make sure enough (say, 100) files are loaded and select all.
  • Select "custom replace"
  • Replace "a" with "a{{subst:void}}" (without the quotation marks)
  • Execute

This results in nothing but null edits and quickly hitting the ratelimit.

Thanks. I saved the profile in VFC. Will use it for testing.

Tgr added a comment.Feb 15 2020, 5:46 AM

The next deploy window is Tuesday 12:00 UTC, can one of you be around to test it?

AlexisJazz added a comment.EditedFeb 15 2020, 5:17 PM

The next deploy window is Tuesday 12:00 UTC, can one of you be around to test it?

If Masumrezarock100 isn't around for some reason and if I forget, just ping me on-wiki.

The patch only affects patrollers. I've just been granted account creator (noratelimit) and I have no alternate accounts with patroller right. (I have an alternate account with autopatroller right..)

@Masumrezarock100, can you test it after deployment?

revi added a subscriber: revi.EditedFeb 16 2020, 4:55 PM

If nobody is around for testing at that moment, maybe ping me (@ IRC!) and I might be able to grant patrollers for few hours.

revi removed a subscriber: revi.Feb 16 2020, 4:56 PM
Masumrezarock100 added a comment.EditedFeb 16 2020, 5:36 PM

The next deploy window is Tuesday 12:00 UTC, can one of you be around to test it?

If Masumrezarock100 isn't around for some reason and if I forget, just ping me on-wiki.
The patch only affects patrollers. I've just been granted account creator (noratelimit) and I have no alternate accounts with patroller right. (I have an alternate account with autopatroller right..)
@Masumrezarock100, can you test it after deployment?

Sure.

Tgr added a comment.Feb 16 2020, 11:17 PM

Sure.

Thanks! I added you for SWAT.

Change 572339 merged by jenkins-bot:
[operations/mediawiki-config@master] Increase Commons linkpurge rate limit for patrollers

https://gerrit.wikimedia.org/r/572339

Mentioned in SAL (#wikimedia-operations) [2020-02-18T12:06:20Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 4b193dd: Increase Commons linkpurge rate limit for patrollers (T245214) (duration: 01m 31s)

I've deployed the fix @Tgr uploaded. Wasn't tested yet, but the purge limit is indeed increased.

>>> $wgRateLimits['linkpurge']
=> [
     "patroller" => [
       3000,
       180,
     ],
     "ip" => [
       30,
       60,
     ],
     "user" => [
       30,
       60,
     ],
   ]
>>>
Masumrezarock100 added a comment.EditedFeb 18 2020, 12:44 PM

I have tested the patch per reproduction steps in T245214#5886683. My rate limit indeed got increased. The API response from https://commons.wikimedia.org/w/api.php?action=query&format=json&meta=userinfo&formatversion=2&uiprop=ratelimits is as follows.

{
  "batchcomplete": true,
  "query": {
    "userinfo": {
      "id": 7299736,
      "name": "Masumrezarock100",
      "ratelimits": {
        "move": {
          "user": {
            "hits": 8,
            "seconds": 60
          },
          "patroller": {
            "hits": 32,
            "seconds": 60
          }
        },
        "edit": {
          "user": {
            "hits": 900,
            "seconds": 180
          },
          "patroller": {
            "hits": 10500,
            "seconds": 180
          }
        },
        "upload": {
          "user": {
            "hits": 380,
            "seconds": 4320
          },
          "patroller": {
            "hits": 999,
            "seconds": 1
          }
        },
        "linkpurge": {
          "user": {
            "hits": 30,
            "seconds": 60
          },
          "patroller": {
            "hits": 3000,
            "seconds": 180
          }
        },
        "badcaptcha": {
          "user": {
            "hits": 30,
            "seconds": 60
          }
        },
        "emailuser": {
          "user": {
            "hits": 20,
            "seconds": 86400
          }
        },
        "changeemail": {
          "user": {
            "hits": 4,
            "seconds": 86400
          }
        },
        "rollback": {
          "user": {
            "hits": 100,
            "seconds": 60
          }
        },
        "purge": {
          "user": {
            "hits": 30,
            "seconds": 60
          }
        },
        "renderfile": {
          "user": {
            "hits": 700,
            "seconds": 30
          }
        },
        "renderfile-nonstandard": {
          "user": {
            "hits": 70,
            "seconds": 30
          }
        },
        "cxsave": {
          "user": {
            "hits": 10,
            "seconds": 30
          }
        },
        "urlshortcode": {
          "user": {
            "hits": 50,
            "seconds": 120
          }
        },
        "thanks-notification": {
          "user": {
            "hits": 10,
            "seconds": 60
          }
        },
        "badoath": {
          "user": {
            "hits": 10,
            "seconds": 60
          }
        }
      }
    }
  }
}

I opened VFC and selected 250 files (means 250 null edits) https://commons.wikimedia.org/wiki/Category:Featured_pictures_on_Wikimedia_Commons to test the limit. I didn't ran into any rate-limits during the run and I was able to complete the job without any interruption.
I tested it again, now with 1000 files (means 1000 null edits) but I didn't run into any rate limit.

Should we keep this task private for a couple of days just in case there are any unintended consequences and we have to revert?

sbassett triaged this task as Low priority.Feb 18 2020, 2:45 PM
sbassett removed a project: Patch-For-Review.
Tgr closed this task as Resolved.Feb 18 2020, 4:28 PM
Tgr claimed this task.

Seems fixed. Thanks @Urbanecm / @Masumrezarock100 for deploying / testing! @AlexisJazz please reopen if you still run into rate limits.

Should we keep this task private for a couple of days just in case there are any unintended consequences and we have to revert?

It was public from the start. Rate limiting is in the public config so changes to it are no secret anyway.