Hello! I would like to note that given we have account autocreations, it's not possible to do this as an isolated experiment. If we allow no-captcha account creations at mediawiki.org, it's then possible to get a no-captcha account at other projects by just letting it autocreate.

Can QuestyCaptcha have a rotating list of randomly selected phrases?

Could this be set up as an A/B test, with some accounts getting FancyCaptcha and others getting QuestyCaptcha?

Could QuestyCaptcha be used for account creation, but FancyCaptcha still be used for new/non-autoconfirmed accounts posting URLs?

In T299629#7636760, @Whatamidoing-WMF wrote:

Can QuestyCaptcha have a rotating list of randomly selected phrases?

Could this be set up as an A/B test, with some accounts getting FancyCaptcha and others getting QuestyCaptcha?

Could QuestyCaptcha be used for account creation, but FancyCaptcha still be used for new/non-autoconfirmed accounts posting URLs?

I think these all sound like potentially good ideas for a second phase of this experiment, but part of the motivation for this experiment, in its simplest form, is that it would be fairly lightweight to implement and monitor. Whereas experimenting with other captcha options involves quite a bit more work, and for which no team at the Foundation (or within the Community) has demonstrated recent interest in solving to completion (certainly not in the near-term), largely due to issues around resourcing, prioritization and technical expertise.

agree on success and abort criteria

make sure there's community support and enough volunteers to monitor, handle abuse filters and other protective mechanisms, and nuke the bots which get true while the filters are being tweaked

These are the trickiest two pieces of this plan, IMO. Given that we don't have complete metrics on captcha performance (as discussed within T241921 and T255208 - and can be a very difficult problem to solve for certain metrics), I think a series of simple, qualitative questions for those involved in the monitoring and management of this experiment might be the most feasible and efficient option as success/abort criteria.

And we definitely need buy-in from and coordination with a number of stewards and admins who can monitor and help resolve any issues that arise during the experiment. I'd imagine that could be organized somewhat organically by reaching out to various individuals directly and coordinating within more secure channels (hello, @Urbanecm).

Update: this is pending further internal review from the Security-Team, WMF-Legal and @Tgr.

Update: This work is still stalled on resources and identifying the answers to certain policy and legal questions. I do hope to advance a draft response to the Director of Security and WMF-Legal sometime this quarter to address these issues.

In T299629#7637269, @sbassett wrote:

And we definitely need buy-in from and coordination with a number of stewards and admins who can monitor and help resolve any issues that arise during the experiment. I'd imagine that could be organized somewhat organically by reaching out to various individuals directly and coordinating within more secure channels (hello, @Urbanecm).

What about beta cluster? We have spambots too.

In T299629#7859962, @AlexisJazz wrote:

What about beta cluster? We have spambots too.

We could potentially start there, though we'd still need real-world data on some of the larger projects as well.

In T299629#7869346, @sbassett wrote:

In T299629#7859962, @AlexisJazz wrote:

What about beta cluster? We have spambots too.

We could potentially start there, though we'd still need real-world data on some of the larger projects as well.

My thought is you add some form of negative captcha as with absolutely nothing this is almost guaranteed to fail. We set the anti-spam abuse filters to log-only which may help in analyzing the results and to prevent spambot IPs from being blocked. Disable the captcha. And see what amount of spam we get.

In T299629#7869964, @AlexisJazz wrote:

My thought is you add some form of negative captcha as with absolutely nothing this is almost guaranteed to fail. We set the anti-spam abuse filters to log-only which may help in analyzing the results and to prevent spambot IPs from being blocked. Disable the captcha. And see what amount of spam we get.

The problem with negative captchas though, is that they can catch a lot of accessibility tools like screen readers, etc. and falsely trap them. Really, that's just the crux overall with any kind of Turing test like captcha - the common tools many folks rely upon for accessibility and anonymity are the same tools malicious actors also use to carry out their nefarious efforts. If wholesale disabling FancyCaptcha on beta (or anywhere else) just seems like a bad idea to those in the community who are closest to mitigating spam/vandalism/harassment, then it might make sense to abandon this experiment, as FancyCaptcha would clearly be doing some good.

Another data point to consider is that, with StopForumSpam now enforcing on the beta cluster (T304111), we've seen 65,438 blocks and 20,163 exemptions over the last 30 days. That seems like a lot for something like beta cluster, from just one deny list, within a test environment that likely doesn't see near the potential abuse that enwiki and other large projects do.

In T299629#7897646, @sbassett wrote:

In T299629#7869964, @AlexisJazz wrote:

My thought is you add some form of negative captcha as with absolutely nothing this is almost guaranteed to fail. We set the anti-spam abuse filters to log-only which may help in analyzing the results and to prevent spambot IPs from being blocked. Disable the captcha. And see what amount of spam we get.

The problem with negative captchas though, is that they can catch a lot of accessibility tools like screen readers, etc. and falsely trap them. Really, that's just the crux overall with any kind of Turing test like captcha - the common tools many folks rely upon for accessibility and anonymity are the same tools malicious actors also use to carry out their nefarious efforts. If wholesale disabling FancyCaptcha on beta (or anywhere else) just seems like a bad idea to those in the community who are closest to mitigating spam/vandalism/harassment, then it might make sense to abandon this experiment, as FancyCaptcha would clearly be doing some good.

Another data point to consider is that, with StopForumSpam now enforcing on the beta cluster (T304111), we've seen 65,438 blocks and 20,163 exemptions over the last 30 days. That seems like a lot for something like beta cluster, from just one deny list, within a test environment that likely doesn't see near the potential abuse that enwiki and other large projects do.

I hadn't heard of StopForumSpam before, can I see the logs for betacluster anywhere?

FancyCaptcha stops hapless spiders that POST in hopes of accessing more information from stumbling into an account and spambots that are smart enough to POST but too dumb to even attempt to solve any captcha. Those would be stopped equally well by an image with undistorted text (which is still crap for screenreaders, though an alt could be provided as those bots aren't even trying to solve it anyway), asking "Who is the president of the US?" (which is also common but problematic for other reasons) or "Who is the founder of Wikipedia?" (and don't you dare enter Larry Sanger ;-) ) or simply "How much is five plus four?".

I think (but to be frank I have no stats to back up what I say) you need something to stop the very dumbest bots. Not much. On the server side, I suspect you could check if the user has completely filled out the account creation form in less than 3 seconds. If they did they are either not human or didn't read what they were submitting.

In T299629#7898178, @AlexisJazz wrote:

I hadn't heard of StopForumSpam before, can I see the logs for betacluster anywhere?

If you have access to https://beta-logs.wmcloud.org/ or /srv/mw-log on the beta cluster servers, they're in channel:StopForumSpam. I don't believe they're publicly accessible in any way though.

I think (but to be frank I have no stats to back up what I say) you need something to stop the very dumbest bots. Not much. On the server side, I suspect you could check if the user has completely filled out the account creation form in less than 3 seconds. If they did they are either not human or didn't read what they were submitting.

There's a few public grafana panels for this, which might be helpful:

1. captcha failure rates - https://grafana.wikimedia.org/d/000000370/captcha-failure-rates?orgId=1
2. auth metrics - https://grafana.wikimedia.org/d/000000004/authentication-metrics?orgId=1

There's probably something to the "not very smart bots" hypothesis in that there are quite a bit more captchas displayed than captchas which are, eventually, correctly or incorrectly solved. Though with just numbers, it's difficult to discern the ratio of "not very smart bots" and frustrated human users. Surely there is a mix of both, unfortunately, which was one of the motivations for running this experiment - to better determine what that might be.

In T299629#7899894, @sbassett wrote:

In T299629#7898178, @AlexisJazz wrote:

I hadn't heard of StopForumSpam before, can I see the logs for betacluster anywhere?

If you have access to https://beta-logs.wmcloud.org/ or /srv/mw-log on the beta cluster servers, they're in channel:StopForumSpam. I don't believe they're publicly accessible in any way though.

I think (but to be frank I have no stats to back up what I say) you need something to stop the very dumbest bots. Not much. On the server side, I suspect you could check if the user has completely filled out the account creation form in less than 3 seconds. If they did they are either not human or didn't read what they were submitting.

There's a few public grafana panels for this, which might be helpful:

1. captcha failure rates - https://grafana.wikimedia.org/d/000000370/captcha-failure-rates?orgId=1
2. auth metrics - https://grafana.wikimedia.org/d/000000004/authentication-metrics?orgId=1

There's probably something to the "not very smart bots" hypothesis in that there are quite a bit more captchas displayed than captchas which are, eventually, correctly or incorrectly solved. Though with just numbers, it's difficult to discern the ratio of "not very smart bots" and frustrated human users. Surely there is a mix of both, unfortunately, which was one of the motivations for running this experiment - to better determine what that might be.

I had already found that domain in T304111 but indeed I have no access.

I think we should accept that anyone who actually targets Wikimedia to spam will succeed. It took me two seconds with a search engine to find https://2captcha.com/ which offers to solve 1000 CAPTCHAs for 3 bucks or less. And they employ humans, so it's not even a false positive. Don't quit your day job though, they only pay $0.50/hour. Still beats the minimum wage in Sierra Leone, Liberia, Malawi, Sudan, Bangladesh and other fun countries.

But there are also bots that don't intend to spam, or at least don't intend to spam Wikimedia specifically. There must be a way to deter those.

Just brainstorming, can we ask the name of the project they are registering on? (maybe troublesome/much work for translation/confusing on lesser-known projects) Or maybe we could create a list of 100 common edible items (bread, rice, cheese, tomato, milk, salad, peanuts, apple, banana, ice cream, that's 10 already) and 100 common non-edible items (airplane, theater, socks, petrol, leather, jewelry, laptop, brick, rocket, cave) and ask the user to set 8 checkboxes for edible/non-edible items. That should be more accessible, right? This would give a 0.4% chance of randomly getting it right. Translation could largely be done by dictionary/Wikidata, https://www.wikidata.org/wiki/Q35509 for example.

Anyone pedantic enough to bring up Michel Lotito would probably just cause drama anyway. :-P