@kaldari Nope, still seeing the same failure mode. It greys out the text in the editor and then throws an error in the JS console ala. An error occurred during ocr processing: /tmp/52004_6179/page_0199.tif.

In T265571#6549666, @Seudo wrote:

Apparently the HTML entities are fixed automatically in the English Wikisource (when I try in this book). ~~~~

This is a dup of T265571.

Not having access to T263371 it's hard to say anything intelligent about the specific issue, but…

Hmm. If one API endpoint returns unencoded text, then lower-level components seem unlikely culprits. If one API returns encoded text, then higher-level components seem unlikely. IOW: those results suggest to me that this is happening at the API layer.

In T256086#6515146, @Soda wrote:

… That isn't what this task is about... It's about deprecating a hook that specifically allows the proofreadpage extension to disable the javascript-based mobile editor. …

In T255345#6514822, @Soda wrote:

I'm not really sure if having the header and footer will help given that they will probably be in wikitext format and extracting it from the html would be easier than attempting to parse the wikitext.

In T256086#6484551, @ifried wrote:

… This would mean that, if no additional changes are made, wikis that use the ProofReadPage extension would not be able to disable the editor. Instead, all wikis with the extension would have the same, standard editor?

I didn't see the "Has somebody already looked at this edit?" aspect (i.e. mark as patrolled) in the above list. But maybe that's included in one of the existing bullet points?

@kamholz I don't understand what it is you're proposing to do here, nor see how it will have applicability outside just Balinese content. From whence comes #transliterate and what does it do? Why hard-code <br> inside ProofreadPage and provide two copies of the text? Why can this not be done with a normal template?

@kamholz You may possibly be interested in this, and there is some overlap with T259645.

In T259645#6362157, @kamholz wrote:

You have a good point about transclusion. I haven't looked into that yet, but presumably the pagelang should be applied there too?

How would this interact with the ability to set the language for a single Page: page? How about mainspace transclusions of Page pages?

@Aklapper Thanks, but as @TheDJ notes, going by that overview nobody owns multimedia features in WMF wikis now. That's a pretty sad state of affairs given how central multimedia is for almost all the projects (including Wikidata and whatever "Abstract" will end up as).

Just to note: Multimedia was tagged as a key player here, but that Phab team seems to have been archived. Who owns the components previously in that group's remit now?

In T258666#6330133, @daniel wrote:

… the import code will try to load the previous revision immediately after creating it. …

Presuming T212428 is an API race condition triggered by replication lag, this seems to be a different issue.

Note that T258666 looks like it could be an expression of this problem that was exacerbated by MediaWiki 1.36/wmf.1 (which is, I think, scheduled to hit the Wikipedias today). And as I commented there: I've never before seen this problem, and I've now seen 3 reports from 3 different projects in the last 24 hours. If so, this is not just a log spamming problem with the occasional user-visible weirdness any more.

The FileImporter issue is apparently T258666 and not obviously related.

I'd never seen the issue from T212428 (and I use FileImporter a lot), but in the last 24 hours I've seen this issue reported from enWS, jpWP, and by a Chinese user (not sure which project they call home) on Commons. If it's not actually more prevalent after deploying MediaWiki 1.36/wmf.1 that's a heck of a coincidence!

For the FileImporter issue mentioned on IRC see https://www.mediawiki.org/wiki/Topic:Vqmtib4ho61z4bji for two files that fail and the error message generated. That problem (and the possibly transient, possibly entirely unrelated ia-upload problem) were both reported in the last 24 hours; while the original issue in this task has apparently been happening for a month or thereabouts.

@Jdlrobson enwikisource has now switched over to a new version that's not reliant on the MF special casing. I haven't filed a separate Phab to disable it in config since it should drop by default in three days, but feel free to do so (or nudge me to do it) if you want it for tracking purposes.

Wouldn't it be easier and more robust to simply grab all the images directly on Commons? If MW doesn't already support JPEG2000 (.jp2; and I haven't checked) then either adding native support or transcoding to plain JPEG should be within the realm of reason (there's at least two OSS libraries for that), and a heck of a lot more robust then trying to dynamically fetch it from IA. We can create ProofreadPage Index: pages from individual images on Commons now, and if there was value in it I'm sure we could cook up some more fancy support (an index referencing a Category on Commons; or a virtual pseudo-file that just references those images; or built-in support for generating DjVus on demand; or …).

Sigh. Since GPU is either broken, or the correct module to pick there isn't mediawiki/core…

Possibly relevant to keep in mind when debugging this; PDF and DjVu files are not treated as just blobs of binary data on disk with just a little metadata in the db. Last I heard the hidden text layer gets extracted from the file, wrapped in an XML structure, and stuffed into a field in the database that is sized for metadata and can overflow. The more pages in a PDF, the more text, the more likely that's going to start playing a role. If this happens before the "upload" is considered complete, it might be a factor.

Everything Proofread Page knows would be useful to be able to access from JS: the associated Index: page, the page number, the previous and next pages, the values of the header and footer templates from the Index:, the CSS field from the Index: (quite possibly the other fields too, but I can't think of a use for them ottomh). Page name for Index and next/prev Page:, plus the {{{pagenum}}} are the obvious top priorities I'd say.

In T255073#6213215, @Billinghurst wrote:

I cannot say whether that was a global css statement or local. Either way, I didn't know think it should have display like that.

@Jdlrobson Any chance of a SWAT deploy on this?

T255073 potentially breaks tons of tools, gadgets, user scripts, etc. My call would be to hold until this is pinned down before rolling out to wikipedia.

This has broken things in awkward ways on English Wikisource (project-local javascript relies on only one such element existing). I've confirmed the same happens on Commons (Main Page). I've been told it only happens in Vector (but haven't verified yet).

In T199992#6178620, @Mahastama wrote:

We've stuck in how to make interface like this within the Wikisource:

where window A will send the material through Cakra API and the resulting OCR will be send into window B, just like in our server.

I'm going to go ahead and tag in the Readers Web guys on this, as it should be right up their alley; as input to the design process for Desktop Improvement if nothing else.

This is presumably not an issue that comes up often because it is rare that edit conflicts occur in ProofreadPage's Page: namespace pages. Each such page represents one single page of a physical book, and the editing done there is either transcribing the original text or correcting the wikitext to match the book, both tasks that are by nature solitary. It is relatively rare for more than one person to work on an entire book (everything covered by an Index:), and for a single Page: page it is exceedingly rare. I've seen this happen maybe twice in half a decade.

Ah, yes. I just ran across another instance of this. The problem is with extremely high-resolution (~4k x ~10k) JPEG-2000 files (particularly when combined with pathological codec settings during encoding) that exceed a hard limit in the Jasper library that GraphicsMagick uses. ImageMagick uses the openjpeg library, so it successfully deals with them, or you can use the utilities in the openjpeg library directly to convert these files.

Provided the "too many arguments" theory is correct, there are two obvious ways to work around it:

In T183087#6102416, @Krinkle wrote:

[…] When you use "Live preview", there will be new content every time you press "Preview changes". The hook from mediawiki.page.startup will run once when you open the edit page (and there is no content yet). The hook from live preview, will run every time for every preview.
[…]
I would welcome suggestions for how the documentation can be improved to summarise/clarify this understanding. The current phrase however, is not incorrect :)

In T244657#5995256, @JTannerWMF wrote:

Hey @ifried is this a challenge a lot of people are encountering?

Semantically the incomplete ref is part of the list, it just shouldn't get the backlink and marker. I don't think the <p> wrapper is important (in fact, it might be better without: cf. T49544). I think in general the short summary for these refs is that they 1) should be at the top of the list, and 2) should not have a backlink and marker. Everything else is implementation detail and practical considerations.

[ Apparently I didn't post this at some point back in January when I wrote it ]

@Tpt rtrim() removes trailing whitespace only, so this shouldn't affect leading newlines in the body of the page, should it?

@WMDE-Fisch Doesn't this just hide the problem? These newlines are coming from somewhere, and I can't think of a valid reason for them, so the root cause really should be fixed imo.

Since this seems to be essentially a dup of T242517 which was fixed by reverting; the part of the change that triggered this task was not an intended change in platform behaviour (it was an unintended side-effect of phan cleanup); and all the specific cases reported in this task have been resolved… I'm going to go ahead and "boldly" close this as resolved. @Ankry Please reopen if you disagree.

Could this be related to T241889?

Yes, I just retried handbookofnature00coms_0 and that is an instance of T215647:

Hmm. I grabbed the JP2s and ran gm on them, and the issue seems to be that these files, by virtue of being a pretty ridiculous resolution (6851x10000), exceeds a hard-coded limit in—in my case—the Jasper library that gm uses for JPEG-2000 support. I believe gm can be linked against different libraries for this (and the particular problem is a Jasper thing) which probably explains some variability in symptoms: different distributions will have linked different underlying libraries, and it may change with OS updates. I think this specific scan is best considered a pathological case.

Apt-ark uploaded this file to Commons manually (as File:Arkansas_Constitution_1874_(published_1913).pdf) so this task can probably be closed as stale now.

The logs (and other files) get deleted after some time so this now seems rather hard to debug (pro tip: upload the logs as attachments, or excerpt the relevant parts in a comment, to preserve them).

This is a duplicate of T215647.

This is almost certainly the same issue as T215647 (but that task has more log data).

@awight It seems all wikis were rolled back to wmf.16 (including the l10n cache) for other reasons, so we're now back to the status quo and <ref follow="foo"> works as normal again. As I understand it they haven't decided yet whether next week's deploy will be retrying wmf.18 or moving on to wmf.19, but from my (very limited) understanding we should not run into this particular issue again either way.

@awight Ok, so the summary seems to be: when the next planned "full scap sync" happens on Tuesday the l10n cache CDBs will get rebuilt with the correct messages, and either on Tuesday or Wednesday—depending on in which phase the caches get pushed out—the missing message key will become available on all the Group 1 wikis and this problem will go away. Is that roughly correct?

@awight Per Nemo above (and on IRC) there is some chaos surrounding this week's deploy train, and issues related to that which are keeping the key people busy. It sounds plausible that this and those issues may be related; but I have no details on what's going on beyond the task Nemo linked above (T233866, <s>which just looks like the deployment tracker to me</s> Nope, I see it's been updated with significant and relevant breakage.).

Just for completeness: I've verified that the problem occurs on plWS as well as enWS.

@Billinghurst I'm told there are other ongoing issues that may make getting the right eyes on this problem difficult with any urgency (cf. Nemo's message above). But as I recall you're a global interface admin: your thoughts on requesting the global interface admins on meta to mass add this interface message on all the affected projects? Details of what's needed in Thiemo's message above. We'd need to also remove it again once whatever is causing this is fixed, which I'm not sure whether the interface admin bit is sufficient for.

@thiemowmde Theory: the initial change removed the message, which removed it from translatewiki.net. The deploy started to Group 0 on Tuesday, at which point messages were sync'ed from translatewik. The patch was reverted on Wednsday before deploy to Group 1, but still with the translatewiki sync from Tuesday. Translatewiki was updated with the restored key, but that change won't be sync'ed back until the next scheduled deploy on Tuesday next.

Hmm. So essentially list-defined references, but with the list of named references defined on a separate wikipage rather than at the end of the same wikipage?

An attempt at some usage / requirements type notes… (and pinging @Tpt as they may wish to be aware of this task)

In T240858#5851392, @thiemowmde wrote:

I'm super confused right now.

Ok, I was hoping to get clarification on actual behaviour before drawing any conclusions, but since this is scheduled to go out today in the ongoing deploy of MediaWiki 1.35/wmf.18…

In T49544#5836366, @ssastry wrote:

Let us please not make changes like changing <span> to <div> without thinking through the implications.

In T49544#5834946, @thiemowmde wrote:

I was very specifically asking for an example of bad rendering in a browser.

In T49544#5834514, @thiemowmde wrote:

I feel the situation was a lot different back in 2013. It's 2020. We are not talking about XHTML any more. We use HTML 5, a "living standard". At this point, is this an academic discussion about the semantics of HTML, or does this have real-life consequences? If so, in which browsers? Is there a specific example we can look at? (Ideally with a revision ID and screenshot.)

In T49544#5507721, @beleg_tal wrote:

So... why are we not simply replacing the span with a div? We've shown several use cases where span causes problems, and I can't even think of any where div causes problems

@Tpt "No decode delegate for this image format" means the requisite image format library is not available (at runtime presuming gm is dynamically linked). First thing to check is whether the file in question is really JPEG-2000, or a mislabeled PNG/TIFF/PPM/whatever (in.ernet.dli is the Digital Library of India, and they have… uhm… original… ideas of how to encode their stuff; I often find PNG labelled as .jp2 or .jpg); and the second is to check "gm version" output for support for that file format.

Looks good to me!

@Umherirrender Many thanks!

@Mpaa By my assessment it cannot have been done deliberately, as the modified code will never work in the context it is evidently being used. When combined with the misleading commit message referencing Phan, my guess is that this was an attempt to fix a warning generated by the static analyzer that just failed to spot the need for armoring there.

See also T242422.

This problem coincides with deployment of MediaWiki 1.35/wmf.14 today, but I find nothing obviously connected in the changelog. A few changes mention touching calls to int(), but not in any context that would obviously be connected to this issue.

@bd808 Adding 5 specific Perl modules and an unrelated binary (Tesseract) to the generic image isn't really a dup of a moderately large new platform architecture change.

In T135963#5543136, @Bawolff wrote:

Ideally [talking to Flickr API] would be done through the server side to preserve user privacy.

@Koavf Which gadgets are enabled and which are on by default is up to each project.

In T228594#5753585, @Koavf wrote:

I clearly don't know everything that's involved but there is an OCR gadget that works on en.ws, so if that functional gadget can't just replace the standard OCR button in the interface somehow, I'm at a real loss.

@Tpt First: you're awesome! :)

@Koavf It's not that simple. Hit me up on my enWS user talk page if you want the nitty gritty details. The short version is that Phe's OCR gadget (which is what this task is about, and which is what I assume you mean by the "default") is based on Tesseract, but that's not where the problem is: it's somewhere in the custom code Phe has written to provide the interface to Tesseract or its interaction with the server infrastructure on Toolforge.

@Adithyak1997 Does this help at all?

In T228594#5752675, @Peteforsyth wrote:

Is that accurate?

There may or may not be a different manifestation of this problem in T240562.

It looks like this could possibly be a manifestation of T192866. In my testing, removing the text layer from a sufficient number of pages makes the problem disappear; but it does not appear removing any single (potentially problematic) page has any effect.

In T228594#5749785, @Koavf wrote:

I don't understand the distinction here.

@Tpt This is almost certainly not the same issue as T204020 (which is a dup of T219376, where some possible approaches to fix it are suggested). 20402 is basically the result of a naïve text layer extraction algorithm in MediaWiki-DjVu that fails badly when faced with unexpected data in a DjVu file, combined with the fact that the DjVuLibre tools will happily accept invalid input that it will choke on when asked to output it again. MediaWiki-DjVu can absolutely be modified such that it will fail gracefully in these cases. But, crucially, such DjVu files can be reliably identified (djvutxt --detail=page file.djvu | egrep "^failed"), and the example from this task does not exhibit that problem.

This is the same issue as T219376 which contains some possible approaches to fix this.

It looks like the mentioned javascript error in the console is gone now (as of 1.35.0-wmf.10 today; I can't recall if I checked on wmf.8 due to the recent hiccup in the deployment calendar), but the text layer is still not loading in the example page. Absent any console messages I have no idea how to debug this further. Are there server side logs that can be checked? A test system where it can be traced? Should I open a new task for that, and if so, with what tags or projects so that it has a chance to hit the right teams' radar?

@Aklapper See link in task description: https://en.wikisource.org/w/index.php?title=Page:Atalanta_-_Vol._2.djvu/18&action=edit&redlink=1

Incidentally, the correct place to solve this is in ProofreadPage because it has knowledge of what the next page in the sequence is. However, I suspect it would need facilities from core to do this intelligently: standard prefetching requires injecting a <link rel="prefetch" href="…" /> in the page, preferably before the browser sees it. ProofreadPage could of course "cheat" the way this gadget does (good version 1.0 / proof of concept / minimum viable product goal?), but that'd be kinda complicated, inelegant, overly specific, and prone to breaking over time.

Possibly related: T236469

This is a common operation on the Wikisources: what on a Wikipedia would be "a page" is on Wikisource "this collection of pages in multiple namespaces". Thus having a list of pages, possibly into the thousands (one per page of a book, for example), is a very common scenario. Based on recent experience on enWS it is even more common than mass deleting all contribs from a particular user (most of our recent spammers have constrained themselves to a single page).

Is it expected behaviour that mw.loader.using('mediawiki.RegExp', function() {…}) will fail silently?

Hmm. While DjVu and PDF (and, I think, TIFF) has explicit text layers; any kind of image can in principle contain text and could benefit from a structured way to store the OCR as actual text. We have oodles of images-of-text in JPEG, PNG, etc. formats in addition to the "book" formats (DjVu, PDF). Even book scans are not-infrequently uploaded as a couple hundred JPEGs gathered in a category (if we're lucky).