• When the user encounters the warning message for already having text in field of page body text and started editing, and if they want to run OCR:
  • There should be a pop-up that tells them in 2 lines:
    • "Replace with new text transcription?" (bigger text)
    • "You will lose the current page body text and it will be replaced with the newly transcribed text."

Label is appearing now on beta:

In T282760#7156232, @MusikAnimal wrote:

Yes! Done with https://github.com/wikimedia/wikimedia-ocr/pull/43. I went with moyen français (1400-1600) per https://fr.wikipedia.org/wiki/Moyen_fran%C3%A7ais

The language selector no longer lists equ.

Testing for this was done as part of T278935.

How Special:Contributions looks now:

In T281866#7148135, @dom_walden wrote:

Great. I think Sam just made https://github.com/wikimedia/wikimedia-ocr/pull/41

In T281866#7148133, @Daimona wrote:

In T281866#7148002, @dom_walden wrote:

In T281866#7142135, @Daimona wrote:

Should be fixed by https://github.com/wikimedia/wikimedia-ocr/commit/5013405a821b0875de15700ddab812eb273c13be, which I mistakenly just force-merged (sorry!)

@Daimona Does that regex need to include numbers as well now? We have just included the language ru-petr1708.

I'd say yes! I'm not sure if tesseract and google are following any standard with the language codes, so in doubt, it's probably a good idea to add whatever characters are currently used.

I have briefly tested this on all four wikis. I could see no major problems. We will await feedback.

This looks good to me. The tooltip and the title attribute no longer have extraneous white-space.

No code change here, so I will move straight to Done.

As this is a developer-facing change, I will move it straight into Done.

Thanks for looking into those for me @MusikAnimal.

In T281866#7142135, @Daimona wrote:

Should be fixed by https://github.com/wikimedia/wikimedia-ocr/commit/5013405a821b0875de15700ddab812eb273c13be, which I mistakenly just force-merged (sorry!)

Testing for this was done as part of T282760.

I compared our list from https://ocr-test.wmcloud.org/api/available_langs?engine=tesseract to what Tesseract claims to support in https://tesseract-ocr.github.io/tessdoc/Data-Files-in-different-versions.html.

1. No drop down menu to select OCR engine
2. Jargon output in Latin script - Test page - https://bn.wikisource.org/s/6bee

In T263617#7140598, @Urbanecm wrote:

@dom_walden I temporarily "fixed" this by manually creating /etc/cassandra/service-enabled, listed as ConditionPathExists in systemd file for cassandra. Then, starting cassandra manually worked as intended. Not sure a) is this file supposed to exist at beta b) if it is, did it exist before c) if it did, why it disappeared.

In T281866#7140922, @MusikAnimal wrote:

Sending this over to Dom in case he wants to revisit it, but effectively I think this task is invalid now that we only accept ISO 639-1 codes (with a few extras specific to Tesseract). If you agree please feel free to close as invalid.

We are seeing this problem again.

We have a new checkbox on Special:BlockList, checking it appears to remove all autoblocks from the table.

Special:BlockList with $wgEnablePartialActionBlocks = true:

@MusikAnimal @Samwilson Is there anything more to be done here? You can move it into Product sign-off or Done if not.

This code change has already been tested on beta here: T282080#7085163.

We now cache the extracted text, meaning subsequent (identical) requests complete within 100-300ms.

In T280953#7107854, @NRodriguez wrote:

Will wait to test on beta

While blocked only from sending Thanks, on a user_talk page with Flow/Structured Discussions enabled:

• "Thank" still shows next to other people's comments
• After clicking "Thank", see an error popup (Thank action failed (error code: blocked). Please try again.)
  
• After refreshing the page, "Thank" becomes "Thanked"
  • I think just in cookie/browser storage. After I cleared cookies and cache it became "Thank" again. But I will probably raise a separate bug.
    
• Also blocked when trying to submit Special:Thanks/Flow (which is the fallback if you don't have JS enabled)

This applies both to your own and to other people's user_talk pages, regardless of whether you are blocked from your own user_talk page.

At the moment, when we block users from sending thanks, we are blocking:

• Special:Thanks
• Special:Thanks/Log
• API:Thank (for both revisions and logs)
• We don't show the (thank) button on the page's revision history or Special:Log

I haven't tested this, but it is such a minor change I am just going to move it into done.

I have experimented with a number of books on beta in different languages. For example:

• https://en.wikisource.beta.wmflabs.org/w/index.php?title=Page:Wind_in_the_Willows_(1913).djvu/24&action=edit
• https://en.wikisource.beta.wmflabs.org/w/index.php?title=Page:%D0%A1%D0%BB%D0%BE%D0%B2%D0%B0%D1%80%D1%8C_%D0%94%D0%B0%D0%BB%D1%8F_1865-2.djvu/1&action=edit
• https://en.wikisource.beta.wmflabs.org/w/index.php?title=Page:%E0%A4%B9%E0%A4%BF%E0%A4%A8%E0%A5%8D%E0%A4%A6%E0%A5%80_%E0%A4%B5%E0%A4%BF%E0%A4%B6%E0%A5%8D%E0%A4%B5%E0%A4%95%E0%A5%8B%E0%A4%B7_%E0%A4%8F%E0%A4%95%E0%A4%B5%E0%A4%BF%E0%A4%82%E0%A4%B6_%E0%A4%AD%E0%A4%BE%E0%A4%97.djvu/1&action=edit
• https://en.wikisource.beta.wmflabs.org/w/index.php?title=Page:Algebraic_relations_between_certain_infinite_products.djvu/1&action=edit

(It does not seem to matter whether or not you are logged in).

Example entry in action=query&list=blocks&bkprop=id|user|by|timestamp|expiry|reason|flags|restrictions:

{
    "id": 22989,
    "user": "Drwpb",
    "by": "Dom walden",
    "timestamp": "2021-05-12T07:29:52Z",
    "expiry": "infinity",
    "reason": "",
    "allowusertalk": "",
    "partial": "",
    "restrictions": {
        "actions": [
            "upload",
            "move",
            "create"
        ]
    }
}

(More examples here: https://en.wikipedia.beta.wmflabs.org/w/api.php?action=query&list=blocks&bkprop=id|user|by|timestamp|expiry|reason|flags|restrictions&bklimit=20)

Do we also want to hide this behind the feature flag, so it does not show when action restrictions are disabled?

In T269341#6998282, @Niharika wrote:

This is a fatal error, and blocks completion of the request. This is in contrast to trying to block a nonexistent namespace, which just returns a warning but otherwise completes the request (ignoring the nonexistent namespace).

From the Slack discussion, @Tchanders reckons the risk for disruption is quite low. We should start a task to fix this in the future for the sake of consistency, however. @dom_walden could you start a task for that? Thank you!

I have been able to use this to successfully OCR files that were stored on a local mediawiki install.

Blocking someone from actions[1] and namespaces[2] which I then disable.

@STran @Tchanders Moving back into code review. There seems to be a CI test failure.

If it helps, I have been able to reproduce this on beta with 1000 votes: https://vote.wikimedia.beta.wmflabs.org/wiki/Special:SecurePoll/tally/632

When a user is blocked from creating new pages:

• They cannot create new pages via web or API (I tested several different namespaces and subpages)
  • This includes not being able to create their own User_talk page. I wonder if we should do the same thing for editing where they are allowed unless we explicitly say so.
• They can edit already existing pages
• They can move a page, even though I think technically this creates a new page (with the same name as the old page IIRC)
• They cannot upload a new file
• They can reupload an existing file
• They cannot successfully import (although they do not see a block error message)

I've tested some combinations of the Tesseract options via the UI. I see variations in the returned OCR text, so I guess that means the options are being passed to Tesseract.

In T281964#7063601, @MusikAnimal wrote:

The cache apparently wasn't cleared after T280213: Wikisource OCR: Accept Tesseract options on the API was deployed to staging. I did it manually and the second example now captures the error properly.

We (the engineers and I) appear to be getting emails whenever there is a 500 error on the test or production sites.

If you enter an invalid language (like in this link), you get a nice error message such as:

The following language is not supported by the OCR engine: blah

The languages we accept varies depending on whether you choose Google or Tesseract.

I tested how blocks look in Special:Log/block and action=query&list=logevents.

Moving to @Prtksxna if he wants to have a look.

I have been able to use the API to create blocks against the upload, move and create actions.

In T280214#7051507, @Samwilson wrote:

Better UI is merged (single input with tag-style language codes). Live on the test site (only, so far). @dom_walden, not sure if you want to have another look at this.

@HMonroy @NRodriguez For the Google OCR engine, we now only treat as valid what Google considers "Supported languages".

Blocking a user from uploading files blocks them from:

• Special:Upload
• API:Upload
• Visual Editor and Source Editor upload (although they appear to just use API:Upload)
• Upload Wizard (tested on Commons beta)

But not from any other action (that I tested).

Testing this has been done as part of T6995. See T6995#7056374.

In T231075#7051365, @Jmwpf wrote:

Hello,

I want to upload a translated svg but cannot and get this same exact error ("A file with this name already exists, and cannot be overwritten. If you still want to upload your file, please go back and use a new name. ..."), and may account is well over 4 days old.

I cannot reproduce the bug in the description anymore, via the web page or API. Now, we just return an empty text box.

I tested the behaviour of moving/renaming pages for a blocked user:

• If blocked from moving pages (either single or composite) they cannot move any page (via Special:MovePage or Api:Move)
• Otherwise, they cannot move to or from pages which they are blocked (I tested this for partial single and composite blocks, sitewide blocks I have tested a bit incidentally for T6995)

I get a very similar error message when suppress deleting a page. Should I raise a separate ticket?

I cannot reproduce this on https://en.wikipedia.beta.wmflabs.org MediaWiki 1.37.0-alpha (7ad7610) 05:22, 30 April 2021.

I am moving this along. This seems to be working fine.

You can see the effect of passing multiple languages here in the returned text:

• Single language (en): https://ocr-test.wmcloud.org/?image=https%3A%2F%2Fupload.wikimedia.org%2Fwikipedia%2Fcommons%2Fthumb%2F1%2F1e%2FThe_Book_of_Scottish_Song.djvu%2Fpage20-1024px-The_Book_of_Scottish_Song.djvu.jpg&langs%5B%5D=en&langs%5B%5D=&engine=google
• Multiple languages (en, sco): https://ocr-test.wmcloud.org/?image=https%3A%2F%2Fupload.wikimedia.org%2Fwikipedia%2Fcommons%2Fthumb%2F1%2F1e%2FThe_Book_of_Scottish_Song.djvu%2Fpage20-1024px-The_Book_of_Scottish_Song.djvu.jpg&langs%5B%5D=en&langs%5B%5D=sco&engine=google

In T281450#7044779, @RhinosF1 wrote:

Slight note, mobile view works fine

Tagging wikibase as they might know about it.

Some differences I am finding comparing Test (https://ocr-test.wmcloud.org) to Production (https://ocr.wmcloud.org):

• Support for columns appears to be better on Test. E.g. compare https://ocr-test.wmcloud.org/?image=https%3A%2F%2Fupload.wikimedia.org%2Fwikipedia%2Fcommons%2F2%2F20%2FENC_9-0379.jpg&langs[]=&engine=google to https://ocr.wmcloud.org/?image=https%3A%2F%2Fupload.wikimedia.org%2Fwikipedia%2Fcommons%2F2%2F20%2FENC_9-0379.jpg&lang=&engine=google
• You will also notice for the above image that Production interprets ſ correctly but Test turns it into f. On the other hand, Test is more likely to correctly recognise à. Production turns it into &.
• Test might have better support for ligatures, e.g. Sphæria in https://ocr-test.wmcloud.org/?image=https%3A%2F%2Fupload.wikimedia.org%2Fwikipedia%2Fcommons%2Fb%2Fb8%2FThe_Annals_and_magazine_of_natural_history_%25281841%2529_%252817788871594%2529.jpg&langs%5B%5D=&engine=google

(Bear in mind this is based on very little data. I suspect the above observations will not be true universally. The quality of the OCR tends to depend a lot on the quality of the image and of the original print.)

In T281093#7039851, @MusikAnimal wrote:

QA notes: There is no version number yet so this shows up as just the SHA at https://ocr-test.wmcloud.org/. The link is broken...

In T279556#7041767, @Tchanders wrote:

Should these wait for T280719 as well, or be fixed separately?

That's the same bug, should also wait for the design fix - maybe we should raise the priority of this. If it's too inconvenient for testing we could put up a quick patch to remove the checking/unchecking, and the checkbox just won't have any effect on sitewide blocks?

In T279556#7041556, @Tchanders wrote:

@dom_walden This is enabled on all beta sites except es.wikipedia - just in case you wanted to do any testing with it not enabled (will leave that up to you!)

We've also enabled the config on testwiki, but the code changes won't make it there until the next train, so you won't see any changes yet.

There's a known bug (comment from the patch on gerrit):

There's a small bug with this patch: when re-blocking a blocked user, the "Upload" checkbox will check/uncheck according to whether the editing radios are partial/sitewide, instead of respecting the parameters of the existing block. Fixing this is a bit fiddly (see how the createaccount checkbox works) and the fix would just be immediately removed once the design is updated: T280719.

I suggest we move forward without fixing this bug, then it will magically disappear once we do T280719 (which should be imminently). I think that would be preferable to holding this patch up.

In T263617#7037139, @Majavah wrote:

The mediawiki-config patch to switch over to sessionstore04 has not been merged or deployed yet, since I have not yet found anyone who can do that.

@Majavah This seems to be happening again. Looking on deployment-sessionstore04, however, Cassandra appears to be running:

dwalden@deployment-sessionstore04:~$ systemctl status cassandra.service
● cassandra.service - distributed storage system for structured data
   Loaded: loaded (/lib/systemd/system/cassandra.service; static; vendor preset: enabled)
   Active: active (running) since Tue 2021-04-27 06:42:42 UTC; 1h 48min ago
    Tasks: 61 (limit: 4718)
   Memory: 1.2G
   CGroup: /system.slice/cassandra.service
...

In T279556#7021891, @Tchanders wrote:

@dom_walden Ahead of us merging this patch, would it be possible for you to do some regression testing on PatchDemo with the feature flag switched off?

I tested a few images on https://ocr.wmcloud.org/ with Google and Tesseract. I didn't see any problems. It seemed pretty quick to me, but I don't have much for comparison.

In T279556#7021891, @Tchanders wrote:

@dom_walden Ahead of us merging this patch, would it be possible for you to do some regression testing on PatchDemo with the feature flag switched off?

In T275870#6989841, @Zdzislaw wrote:

@Samwilson It looks like the problem has been resolved. Thank you very much for your quick reaction and implementation of the solution (many people involved in promoting pl ws breathed a sigh of relief). We appreciate it.
I also checked the cases reported here - looks like zh is ok as well.

How the icons on the list page look:
Before:

I have been able to use the tesseract option to do OCR, for example.

In T279610#6994738, @MusikAnimal wrote:

So for this ticket, I suppose all we've done is better validate the URL that was given.

I cannot reproduce the bug in the description. When I try to submit an API block with a nonexistent or deleted page, the API returns:

{
    "error": {
        "code": "cant-block-nonexistent-page",
        "info": "You cannot block a user from Nonexistentpage because the page does not exist",
        "*": "See https://en.wikipedia.beta.wmflabs.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce> for notice of API deprecations and breaking changes."
    },
    "servedby": "deployment-mediawiki11"
}

I tested a few different types of election, including Range voting (plurality) with comment and Range voting (histogram range) with comment.

I have tested submitting translations in a couple of different languages and seeing the database and SecurePoll:<id>/msg/<lang> being appropriately updated.

In T275870#6978697, @Samwilson wrote:

... Anyway, how about we switch to using the title attribute instead, and check for whether it begins with [title]/?