Page MenuHomePhabricator

Ongoing spambot attack 2019-08-{10,11,.*}
Closed, ResolvedPublic

Description

We are under another mass spambot registration attack as evidenced in https://cy.wiktionary.org/wiki/Arbennig:RecentChanges?hidebots=1&hidecategorization=1&hideWikibase=1&limit=500&days=7&urlversion=2. This is not limited just to cywiktionary but to a number of several other small/medium sized wikis so if we could perhaps enable account creation throttles for the small.dblist and medium.dblist I guess it'd disrupt their activity a lot.

Event Timeline

MarcoAurelio created this object with visibility "Custom Policy".
MarcoAurelio created this object with edit policy "Custom Policy".
MarcoAurelio set Summary to Spambot registration and spamming.Aug 11 2019, 10:11 PM
MarcoAurelio set Impacted to All Wikis.
MarcoAurelio removed Summary.
MarcoAurelio removed Impacted.

The script that refreshes our CAPTCHA database is currently broken and has been for some time cfr.: T230245.

Set account creation throttle to 2 per day for small and medium wikis per @MarcoAurelio's request (529621). They apparently can bypass captchas, given there are links in the spam entries, so nothing like emergency captcha will help :(.

Thanks @Urbanecm - this will reduce, hopefully, the ammount of spambots that get registered. I still see spambots being created at a lower rate, apparently; so this is working. Unfortunatelly botnets have tons of infected devices (and thus, IP addresses) so the attack will continue until they give up.

Listing some tasks that might be of interest as well:

@MarcoAurelio, @Urbanecm:

  1. It's fine (imo) to file separate tasks like these for (potentially) new incidents. Though I'd personally prefer if we could consolidate or resolve older tasks so as not to have several open at once, as that can become a bit chaotic to manage.
  2. Hopefully T230245 gets resolved soon, though I'm honestly not sure it will make that much of a difference since our current captchas aren't very effective anyways. Some interested WMF folks (including @Dsharpe and myself) are currently working towards investigating solutions for improved CAPTCHAs.
  3. Thanks @Urbanecm for adjusting the new account creation throttles. I honestly wouldn't be opposed to making those even more restrictive or potentially even disabling new account creations for a period of time. Unfortunately, this and a handful of other measures are about all we can do from a configuration standpoint right now, outside of digging through logs and attempting to find IPs/UAs/GeoIPs we can block, which is a lot of work and difficult to guarantee there won't be a large number of false positives.
  4. @Reedy and I are still working to get StopForumSpam deployed to beta, which should help substantially, however the Security-Team is extremely short-staffed right now for a number of reasons and we don't have anything remotely like a 24/7 SOC.

I've enabled global abuse filters on warwiki to lower the impact of this. I see warwiki is in large.dblist, and thus, has global abuse filters disabled unless explicitly enabled in IS.php. Even there is over one milion of content pages on warwiki, it has just three sysops. It's more a small wiki than a large wiki, at least from the general point of view. Since I have little to none idea what does large.dblist do, I've enabled filters explicitly rather than moving the wiki from one dblist to another.

@Reedy, since you moved the wiki to large in 1fc7d2eae4, could you comment on this, please?

Set account creation throttle to 2 per day for small and medium wikis per @MarcoAurelio's request (529621).

Enabled the same for warwiki, since it's in large.

urbanecm@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Account creation throttle to 2 everywhere (T230304) (duration: 00m 47s)

Set account creation throttle to 2 everywhere, spammers seems to moved to commonswiki per @Praxidicae.

I enabled a filter to disallow their creations on 4 projects, warwiki, tumwiki, cywiktionary, huwikisource. Warwiki is still getting the brunt of it with almost 100 triggers (Aside from a test) and no false positives since I enabled it this morning. This isn't a good long term solution since it's a very strict filter but figured it might help you guys here. Worth noting that they also moved to commons and enwikisource briefly but don't appear to be getting through successfully nearly as often as they do on warwiki.

MarcoAurelio renamed this task from Ongoing spambot attack 2019-08-{10,11} to Ongoing spambot attack 2019-08-{10,11,.*}.Aug 12 2019, 7:49 PM

I've removed the account creation throttle. It's not hitted on warwiki (one of the most attacked wikis), and it seems it's blocking only real users now. I'll leave the global abuse filter change enabled, since it makes sense to have global filters running on warwiki.

@sbassett Could you review this, and close if you share my opinion that everything doable was done?

sbassett claimed this task.

@Urbanecm - this seem fine for now. I see $wgAccountCreationThrottle is back to normal and @Praxidicae's AF filters are still in place on those 4 projects, correct? If this specific attack (hard to tell them apart, really) heats up again, we can reopen this task, but I'll resolve for now. As a note: we're (Security-Team) still working on some StopForumSpam issues and getting that deployed to beta soon, along with other mitigations which we hope will be more effective in stopping problematic account creations.

Thanks @sbassett. Claiming this task, since I worked on it :).

Happy to help! Since there are no non-public info AFAICS, can this be published, or should this be PermanentlyPrivate ?

@Urbanecm - looks like other spambot incidents have been made public in the past, so I think it's fine to do so here. Nothing terribly secret on this particular task.

Urbanecm changed the subtype of this task from "Security Issue" to "Task".Aug 15 2019, 9:51 PM
Urbanecm changed the visibility from "Custom Policy" to "Public (No Login Required)".Aug 15 2019, 9:51 PM
Urbanecm changed the edit policy from "Custom Policy" to "All Users".

Done then @sbassett. PS: Is there a simpler way how to change task's subtype than abusing my acl*Batch-Editors power and submitting a bulk job with one task?

Chage Subtype in the Add Action... dropdown allows you to do so.

@Urbanecm - I'm not sure I even have perms to change subtypes, don't know much about that feature to be honest. I'd probably recommend reaching out to a Phab admin (@Reedy, @Aklapper, et al) to confirm.

You should have, if I can do so ;).

Given that this still appears to be the most recent spambot attack task, I thought I'd provide a small update here. I wrote some python that generates basic stats on new account creations and whether or not IPs associated with those users (as found within logstash) appear within StopForumSpam's various black lists. Here are some of the results for some of the recent, affected projects (e.g. T227416#5348535 etc):

ProjectDate RangeNew Acct CreationsNew Accts With IPs Within At Least One SFS File
es.wikiquotelast 30 days72845%
cy.wiktionarylast 30 days58393%
pt.wikiquotelast 30 days35363%
id.wikibookslast 30 days35049%
id.wikiquotelast 30 days19569%
species.wikimedialast 30 days111916%
gv.wikipedialast 30 days11428%
wikimania.wikimedialast 30 days84433%

One big caveat is that this is not a perfect way to calculate these stats, but they should, at the very least, provide relative indicators regarding SFS' potential to help combat spammy edits and new account creations. Although the mw extension is currently not capable of doing the latter, which would be one decent hurdle to overcome. After chatting with some Platform Engineering folks, It should be doable via a PreAuthenticationProvider similar to how ext:SpamBlacklist does it now.

2 1/2 months later; Is there anything actionable in this task to keep it open?

@Aklapper - this can be resolved. Any tracking tasks like this for an ongoing incident should most likely be resolved after a couple weeks of no on-task activity, and then a new task created to track the next incident. However, there's probably no great way to automate that at the moment.

@sbassett: Thanks for the quick reply. If you think that this should be resolved, please set the status of this task to "Resolved" via the Add Action...Change Status dropdown (or wondering if @Urbanecm wants to do that as the task assignee)? Thanks.

@Aklapper - I believe it is currently resolved?

/me facepalms... Not sure where my eyes were earlier today. Sigh. Sorry and thanks!