Page MenuHomePhabricator

Multiple blocks by AbuseFilter without Multiblocks enabled
Closed, ResolvedPublicBUG REPORT

Description

On itwiki a LTA is somehow managing to get blocked multiple times by the AbuseFilter. While we opted in as a pilot wiki, we don't have Multiblocks enabled yet.

At first it looked like a single edit was triggering the same filter multiple times. Upon closer inspection of the timestamps, they may be repeated, very close edit attempts. Maybe a race condition triggered by a misconfigured bot?

I don't know how to reproduce it, but here's a list of the affected accounts. The maximum number of the issued blocks appears to be 8, which is consistent with wgRateLimits for new users.

Timestamps of one of them, according to the APIs: abuselog, blocklog.

What happens?:
User is blocked multiple times

What should have happened instead?:
User is blocked once

Software version (on Special:Version page; skip for WMF-hosted wikis like Wikipedia):

Other information (browser name/version, screenshots, etc.):
Originally reported here.
As a side note, said blocks can't be changed or lifted. We verified that the blocks can instead be removed one by one by using the new Codex interface, as reported on T388743#10650990.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Yep, just to confirm @Titore, I noticed a lot of "User tripped rate limit" within a few hundredths of a second (even 10-15 times in the same second). There is also T362403 about (likely) the same issue.

I think it's worth noting that this specific vandal is using automation tools to carry out the vandalism, which explains why they are being blocked at such a high rate.

I tried to reproduce this on testwiki by sending 15 API requests at once with no delay whatsoever and its pretty easy to do (esp. with javascript):
https://test.wikipedia.org/w/index.php?title=Special:Log/block&page=User%3AXXBlackburnXx

But once i started adding a few milliseconds delay to it, it becomes way harder to pull it off, or at least not at the same rate observed in the initial example.

Adding high network latency and slight packet loss (via tc on Linux) also made this behavior easier to reproduce.

This is definitely a race condition. I had a brief look at core's BlockUser, and there doesn't seem to be any protection (mutex, lock) against it when placing a block.

In this regard, we probably need to look at the blocking management in AbuseFilter as a whole and adapt it to multiblocks. For example, when multiple filters want to block a user, deduplication is not mandatory anymore.

I tried to reproduce this on testwiki by sending 15 API requests at once with no delay whatsoever and its pretty easy to do

I tried this on my local (with no delay) and was not able to reproduce, both with and without multiblocks enabled.

This is definitely a race condition. I had a brief look at core's BlockUser, and there doesn't seem to be any protection (mutex, lock) against it when placing a block.

I think that sort of thing is done in the storage layer, i.e. DatabaseBlockStore. But either way, the fact that multiblocks was utilized when the feature flag was not enabled is indicative that there is a problem.

In this regard, we probably need to look at the blocking management in AbuseFilter as a whole and adapt it to multiblocks. For example, when multiple filters want to block a user, deduplication is not mandatory anymore.

See also T351511: How to deal with redundant block

This is definitely a race condition. I had a brief look at core's BlockUser, and there doesn't seem to be any protection (mutex, lock) against it when placing a block.

I think that sort of thing is done in the storage layer, i.e. DatabaseBlockStore. But either way, the fact that multiblocks was utilized when the feature flag was not enabled is indicative that there is a problem.

Thanks. I had another look, and there is indeed an invocation of atomic section on the path. No idea what is going on.

Thanks. I had another look, and there is indeed an invocation of atomic section on the path. No idea what is going on.

The atomic section doesn't help, there's a huge window when duplicate blocks can be inserted. See T388743/T389028.

Anything else that needs to be done here? I can no longer reproduce this now that T389028 is complete.

matej_suchanek assigned this task to tstarling.

Probably not. We still have to adjust the behavior with regards to Multiblocks, but that's in the scope of T351511.