@Reedy: Hmm, what exactly is this task stalled on?

In T141490#6403690, @Aklapper wrote:

@Reedy: Hmm, what exactly is this task stalled on?

Lack of any consensus. Lack of decent enough metrics to even deploy the change and see what happens. It's been open 4 years at this point, and no clear path of moving forward for deploying any of the improvements proposed.

It'll probably get declined at some point

Mainly lack of metrics; lack of consensus is a consequence. So the path forward would be T255208: Catalog and evaluate methods of analysis for Wikimedia captcha performance or something similar - and eventually, some reporting mechanism that gives some reasonably reliable numbers on how well the captcha performs against humans vs. against bots. It's a few weeks of work, IMO, it just has been hotpotato'd around between various departments.

• Moved this bug report to its own item https://phabricator.wikimedia.org/T263223

I might be stating the obvious but a change like this shouldn't take 10 years to be deployed (with the last comment being more than three years ago). Is there anything I can do to get this off the ground? Should I just make the patches and get it deployed and check some (*waves at the air*) metrics? Should I just get T255208 done first? Anyone willing to help?

In T141490#9361722, @Ladsgroup wrote:

I might be stating the obvious but a change like this shouldn't take 10 years to be deployed (with the last comment being more than three years ago). Is there anything I can do to get this off the ground? Should I just make the patches and get it deployed and check some (*waves at the air*) metrics? Should I just get T255208 done first? Anyone willing to help?

Given the amount of time that has passed and the lack of conclusive data on how this might improve FancyCaptcha's efficacy while not further hindering accessibility, I'm not sure a simple code-cleanup and deploy would be the best option here. We definitely wouldn't want to introduce a worse captcha experience for project users at this point.

AIUI the fundamental blocker here is not being able to differentate between increased captcha failure rate for bots vs. increased captcha failure rate for humans. So we'd need captcha success rate stats (which we sort of have, but not very good ones) plus some sort of bot detection.

Also IMO the patch proposed here is a non-starter because it makes the captcha a lot harder to read, while the security gains would be limited at best. @Bawolff had some ideas for captchas which at least at first glance don't look harder for a human (see comments starting at T125132#4432800).

I feel like we're over emphasizing perfect stats here. Its not like the original captcha had extensive testing. Surely we can test how readable any given new captcha proposal is, by asking for a bunch of volunteers from the community to solve some captchas and see how they do.

I agree with Brian here, Captcha is by design hard to measure. If you have a way to detect bot request failure, then why are relying on captcha in the first place? just use that method to block the bots.

Sure, user testing works too. I'm not sure it's less effort. You can look at effect of captcha on known-human users (e.g. IPs from some insitutional range), split things by user agent etc.

You can look at effect of captcha on known-human users (e.g. IPs from some insitutional range)

Sounds difficult... what network can possibly be guaranteed not to have any bot on it?

I agree it _might_ be possible to pick some statistical proxy which would give us an idea of the impact. For example, what portion of the users pass the captcha and then get blocked, and perhaps more importantly how many capcha attempts were needed by users who did pass the captcha and did not get blocked after making an edit.

Examples of output of the patch I'm about to make

Change 990694 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/extensions/ConfirmEdit@master] Add negative kerning and lines to captcha

https://gerrit.wikimedia.org/r/990694

Change 990697 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] mediawiki: Use the new captcha

https://gerrit.wikimedia.org/r/990697

Change 990694 merged by jenkins-bot:

[mediawiki/extensions/ConfirmEdit@master] Add negative kerning and lines to captcha

https://gerrit.wikimedia.org/r/990694

FWIW, this is deployed in beta cluster as of now.

Change 990726 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/extensions/ConfirmEdit@master] captcha.py: Increase number of random lines in the text

https://gerrit.wikimedia.org/r/990726

^ This increases the number of lines, feel free to accept or abandon it. Some examples:

I haven't thrown this into an OCR to check its performance though.

The examples do seem like the lines are a bit low, maybe there should be some lines at the bottom and some near the top of the word

oh indeed, made some changes:

Is it better?

I think so, it looks more random to me anyhow (this is just gut feeling, i haven't done any testing)

Ran Tesseract Open Source OCR Engine v4.1.1 with Leptonica on 1000 generations and only on 71 it produced any output which mostly were garbage:

( CutsSauna-
Ceeuighv ews —
ashySears
U Serubpadgy —
we
newts julips
bongdoar_
<vadenkama
=
( pathghebe
wi
\sewersaez—
——
Ss
~ spagemary—
<= Gegn
“ Fenépunk—
-eedasonja-
—waveduery
=
(CupssnowY
~<prdnknodez——
a
a
Sakhaurged
c_ ahmeddears—
Cstudsrusts——

Only one case it produced the exact value, and if you strip out the garbage, it still doesn't go higher than five cases in total.

Change 990726 merged by jenkins-bot:

[mediawiki/extensions/ConfirmEdit@master] captcha.py: Increase number and position of random lines in the text

https://gerrit.wikimedia.org/r/990726

In T141490#9460756, @Ladsgroup wrote:

Only one case it produced the exact value, and if you strip out the garbage, it still doesn't go higher than five cases in total.

For comparison, the out of 20 mentioned above (F4313219), Tesseract correctly recognizes all twenty of them

Edit: That wasn't a fair comparison, With the old version can, the rate is 33% (https://bawolff.toolforge.org/captcha/setB/), with the newer one probably higher.

The options you give tesseracrt can make a big difference, in my old test i used -psm 13 -c tessedit_char_whitelist=abcdefghijklmnopqrstuvwxyz

We might want to consider making them not real words, to prevent spell checking attacks.

Fair, with that options, you get 7.8% accuracy.

In T141490#9460683, @Ladsgroup wrote:

oh indeed, made some changes:

Is it better?

I did some testing, and posted the results at T125132#9468294

Mentioned in SAL (#wikimedia-operations) [2024-01-25T17:30:28Z] <Amir1> deploying new captchas (T141490)

This is deployed.

Change 990697 merged by Ladsgroup:

[operations/puppet@production] mediawiki: Use the new captcha

https://gerrit.wikimedia.org/r/990697

In T141490#9489154, @Ladsgroup wrote:

This is deployed.

Resolvable for now then?

From an accessibility perspective, the proposed solution will decrease usability for people with blurred vision and dyslexia, potentially other users with visual impairments, but it does not drastically decrease the overall accessibility. That being said, I believe this is a fine solution for now.

@Bawolff How much does it matter if the font is FreeMonoBoldOblique or DejaVuSans? I see https://gerrit.wikimedia.org/r/c/operations/puppet/+/990697 where it was switched because the former font isn't installed on mwmaint hosts.

Should we look more into how to get that installed or is DejaVuSans fine?

fonts-freefont might be the package.

(Some more work is needed, changing the font if suitable, comms, measurements, etc. but doesn't need to happen in this ticket)

re: font choice. Visually, there is slightly more noise with the use of FreeMonoBoldOblique because of the serifs. This might have an effect on the ability to break it if that noise is helpful to make the CAPTCHA more difficult to break. From a readability perspective, DejaVuSans seems fine as that lack of additional serif noise may increase readability when all the characters are more close together with the noise of the abstract lines. But again, does that effect the ability to break it? Have these two fonts been tested against one another?

In T141490#9489169, @Dzahn wrote:

@Bawolff How much does it matter if the font is FreeMonoBoldOblique or DejaVuSans? I see https://gerrit.wikimedia.org/r/c/operations/puppet/+/990697 where it was switched because the former font isn't installed on mwmaint hosts.

Should we look more into how to get that installed or is DejaVuSans fine?

fonts-freefont might be the package.

FreeSans is more secure against more sophisticated attackers at the cost of somewhat lower readability (due to lower interletter spacing). DejaVu sans and FreeMonoBoldOblique are roughly the same level of security imo.

In T141490#9489166, @DTorsani-WMF wrote:

From an accessibility perspective, the proposed solution will decrease usability for people with blurred vision and dyslexia, potentially other users with visual impairments

I think the example in the task description oversells the readability of the current captcha (there is a random factor in how "hairy" it is and the more hairy ones can be quite difficult to read, even for average readers). Brian made a great dashboard where you can check out a more realistic set of examples: old captcha, new captcha.

The old captcha is hard to read in a certain way (characters can get quite distorted), the new captcha is hard to read in a different way (the characters aren't distorted but get squished together). I'm not sure what's the net effect is.

Fwiw: my intuition around different text captcha methods:

There are basically 3 steps in attacking a captcha - remove extranous elements, separate the word into individual letters, identify each letter. Thus those are the three knobs we can control to adjust difficulty.

• non letter elements (lines etc) - about equally hard for a computer and human. If they are obviously different then letters than both humans and computers can discard them. If the look the same as letters then hard for both. The best thing about this method is that off the shelf OCR software can't handle it well, so the attacker is forced to program their own thing, even if the algorithms involved aren't super complex. Thus we can get some benefit for very little drop in readability.
• overlapping letters - this is generally hard for computers and medium difficulty for humans (relatively speaking). This aims to prevent computers from seperating words into letters, so the entire captcha has to be taken as a whole. The algorithms involved in segmenting tend to be more complex, forcing the attacker to spend much more effort programming counter measures.
• distorting letter shapes. Generally hard for humans and easier for computers. This usually doesn't prevent a computer from segmenting a word into letters, and once the computer is dealing with it a letter at a time, things become much easier as the computer has only 26 choices and only needs to decide which of the 26 it is closest to, which computers can usually do with ease if it is all readable by humans. Additionally this part overlaps with OCR, so there are excellent off the shelf solutions.

Ok, thanks Bawolff, won't worry about the package then.

Is this (new?) captcha-related issue on beta cluster related? T355962

In T141490#9491493, @Dzahn wrote:

Is this (new?) captcha-related issue on beta cluster related? T355962

It's unrelated, the new captcha was deployed to beta cluster for weeks now. I know what's the root cause of that bug.

FYI, this rebroke captcha.py for pillow 10, by reintroducing getsize, see also T354099

Some stats (web only; event / minute, 3-hour rolling average, last 7 days compared with previous 7-day periods; for the last one % of failed captcha attempts from all captcha attempts, 6-hour rolling average):

successful registrations		link
successful captcha attempts		link
failed captcha attemps		link
captcha failure rate		link

Zero effect on captcha success, but captcha failures are almost cut in half, and the changes correlate well with the time of the deploy (16:30 on the 25th). What does that mean?

I can think of two explanations:

1. The average user finds the new captcha easier. We are down from a ~25% human failure rate to ~15%.
2. Some class of not very smart spambots that could OCR the captcha into a string in the past but with near-100% failure rate now has its OCR broken so hard that it doesn't even try to submit anything.

Either way, this seems like a positive change (although in the second case, not very consequential).

I did some measurements similar to what Tim did in T152219#3405800. I suggest reading that comment before continuing to read this.
Note that before the deploy, the measurement was for a week, but after the deploy, I have data for a day (and will repost once a full week passes).

Wikis failure ratio graph:

before deploy (for a week)	after deploy (for a day)

The plateau goes down from ~20% to 15-16% which to repeat Tim's prediction from seven years ago (T152219#3405800):

If we switched to a different CAPTCHA solution, we would want to see the height of the plateau remain the same, or be reduced.

(Why? The plateau basically represents large-enough wikis with mostly human activities)
Which is exactly what happened here, you could roughly estimate that we had 20% decrease in human failure (from roughly 20% to 16%). Any that is also aligned with what Tgr said in T141490#9491802.
Noting that it doesn't mean it's better for a11y, it's just easier to read for humans overall, it might make it easier for certain people while making it harder for other but the net change seems to be an improvement.

We know it's very likely human failure ratio going down because the slope for wikis with mostly spambot activities didn't move in a negative direction. It is too early to give an exact value for the derivative of the slope after 80% of failure rate but I'm not seeing that slope stopping at below 100% or having a lower derivative. I wait for a week for the exact estimate of impact on the spambots.

Here is list of large wikis failure ratio and the change:

I want to note that the wikis that had the highest redaction in failure rate were mostly wikis in languages that are not written with Latin scripts (Ukrainian, Persian, Korean, Thai, Arabic, Bangladeshi, Hindi, Russian). That would explain the massive redaction in human failure rate.

Mentioned in SAL (#wikimedia-operations) [2024-01-30T15:06:41Z] <claime> Manual run of mediawiki_job_generatecaptcha.service following timer failure - T141490

Results for a week are in:

Regarding impact on human failure rate

The plateau is at 16% which is lower than 22% on the old one, a 27% redaction on failure rate for humans.

Regarding impact on non-latin wikis:

On wikis that had more than 100 captcha attempt in the week of 26 Jan - 2 Feb, here is the result:

Difference for non-latin wikis is 9.3% redaction on average (it's not weighted). And for latin wikis it's 2.8% redaction in captcha failure rate (can be anything from non-native speakers or general improvement)

Given what we have, the two-tailed p-value for null hypothesis of latin or non-latin scripts having no impact on captcha failure rate is 0.000139 and as such is rejected. (Warning: My math/statistics is rusty)

Notes: I ignored multilingual wikis, considered sr and sh latin wikis and some other small stuff.

Regarding the impact on bots:

The slop on 80% and above has been cut to one third (from 0.03278 percent/captcha attempt to 0.00947 percent/captcha attempt) which is not what we expected, It can be many reasons, including that improvements on human captcha solving is masking the impact on the bots (as it's quite large) or the fact that there has been less captchas in total (bots not attempting?), etc. etc. I will need to investigate this more but any idea would be more than welcome.

I have some experience on the grey market, and I would assure you that these changes didn't make the captcha any stronger against machine OCR. There a lot of weaknesses remaining. Some of them are following.

• The random noise lines should not be grey, otherwise they can be easily discarded by a simplest contrast improvement using any graphic library.
• The operator of OCR can detect the font you use, then OCR crops every letter and matches it against existing library of letter images for this font type. The font type should be different for each letter to prevent that. The letter size also should vary.

In general, don't invent the bicycle, use open-source captcha solution with higher strength that are still easy to be solved by humans.

And when you do so, a part of the bot makers will simply forward these captchas to some real humans somewhere in India or China, they recognize 1000 captchas for 10 cents.

The real difference could only be made with Google reCaptcha or a non-evil alternative that uses IP-Address matching and browser fingerprinting to match the user against a blacklist.

I understand and agree abut this is just about raising the bar a little to avoid captcha being broken with off-the-shelf OCR with default settings. No matter what we do, it'll be some way around it. It's just improving one layer of defense out of many. Further improvements on captchas are being talked about at the moment. We will keep the community updated on those.