Hmm. As I recall, PRP uses a hard 1024px size for the "thumbnail" it requests. I am assuming this was a value picked as a sort of compromise between full fidelity to the user and various optimization concerns.

Hmm. Based on this and a few other recent failures, I'm starting to wonder if php-exec-command (which is the Command::exec(); wrapper ia-upload is using to execute binaries) is broken and returning "Command not found" for any non-zero exit status.

In T101075#6780815, @matmarex wrote:

… there's a difference in wikitext between an empty parameter and a not-provided-at-all parameter, …

In T257066#6843760, @sbassett wrote:

There is some progress being made on various protected tasks, …

So… we're currently waiting for a suitable volunteer to materialize out of thin air to address an issue whose details are not public for security reasons? And in the mean time we have many thousand broken pages across multiple projects and all we can do is bleed contributors in those areas?

In T274495#6822475, @Koavf wrote:

In T274495#6822466, @Xover wrote:

Absent specific proposals for better wording

I gave a proposal.

Absent specific proposals for better wording I think the status quo works well enough. Far from every ebook user has any conception of file formats, much less any idea what kind is best for their device, so giving them enough information suited to their frame of reference to make a sensible choice is a priority.

Let me throw an extra angel on the head of this needle: a user might conceivably want to export a work when currently on a wikipage in these namespaces, and a user might conceivably want to export a single page, as defined by a Page: wikipage, of a work.

I think that for any inherently paged format (like PDF), print should be a primary concern. For everything else we should nudge people to ePub where content can be dynamically reflowed. I have trouble imagining that a significant number of people actually print these onto dead trees, but that is the main rationale for the design of the PDF format the way it is.

Uhm. A5? Every printer in the world is designed for A4 (or its bastard offshoot, US Letter), and every sheet of printer paper sold ditto. The other sizes, including A5, are barely measurable in comparison. In fact, I think some of the B sizes may actually outsell A5 due to use in automated mass-mailings of various kinds.

No, please don't. Forcing links to open in a new tab or window to keep the user on your site is literally a dark pattern in web design. Users are quite capable of opening a link in a new tab if they want to, and, conversely, those users who have trouble with this are also apt to be confused by navigating multiple tabs or windows.

In T134469#6705758, @cscott wrote:

I bet something like __NO_P_WRAP__ would be fairly easy to support. Would it get enough adoption to get us closer to our goal of turning it off by default?

In T134469#2366099, @cscott wrote:

… In ten years, I'd love for us to be at the point where we don't do <p>-wrapping at all …

Yeah, daily would be better for newly added works. For changes to existing works the frequency could be much lower with not much problem I think. Alternatively new works could be manually triggered (we have lots of manual processes already) given an interface for it.

Oh, no, wait… I think I'm just being a dummy!

It definitely isn't working. On this page the paragraphs run together, but the output from djvutxt thefile.djvu -page=17 -detail=page is:

Hmm. $wgDjvuTxt is set in CommonSettings.php, so that should be ok.

Hmm. I didn't think there'd be any caching of this, but I may have misunderstood. It might also be that retrieveMetaData() is called once on upload rather than on demand as I'd assumed. And we need to check what $wgDjvuTxt is set to, since this whole block is only executed if that config var isset().

Just to add a perspective…

In T228594#6622243, @Ankry wrote:

In T228594#6622158, @Xover wrote:

Could you apply this diff

Done.

In T228594#6622087, @FreeCorp wrote:

In T228594#6615484, @Jan.Kamenicek wrote:

… every word is on a new line. …

Same feedback as @Jan.Kamenicek tonight, although it seemed to worked great a week ago.

In T228594#6621848, @Mpaa wrote:

@Xover, I think it is a misunderstanding
data.text.substring(0,5) != "<?xml" -> XML is accepted, if it is not XML, then is considered error.

In T228594#6620968, @Pols12 wrote:

…fallback to old OCR when got text is an error message instead of XML content:

function hocr_callback(data) {
	if ( data.error || data.text.substring(0,5)!="<?xml" ) {

Ok, I've now had some independent testing (Big big thank you to Jan!) that confirms the tweaked Gadget code now produces results that are at least within a reasonable distance of what it used to produce.

Ok, an update on the corrupted cache…

In T228594#6615608, @Xover wrote:

In T228594#6615484, @Jan.Kamenicek wrote:

… the [OCR] result is very poor, …: every word is on a new line.

This is a separate problem, and is most likely related to Tesseract being upgraded to 4.x.

In T228594#6615484, @Jan.Kamenicek wrote:

Unfortunately, the OCR does not work with any of these at all

In T228594#6614254, @Jan.Kamenicek wrote:

… I tested it now e. g. on Page:John_Huss,_his_life,_teachings_and_death,_after_five_hundred_years.pdf/122 and some other pages of the same book and it still does not work here :-(

In T228594#6613257, @kaldari wrote:

@Xover - What would be the effect of just deleting all the caches? Tesseract has been upgraded since most of those caches were generated anyway.

In T228594#6611611, @Pols12 wrote:

… still broken for some others, e.g. Page:Europe in China.djvu/422, Page:Europe in China.djvu/422 or Page:Plutarch's Lives (Clough, v.4, 1865).djvu/215.

The cache for a given work will be in a subdirectory of ~/cache/hocr/ created from the MD5 hash of the file's name (spaces replaced with underscores) concatenated with the invoking project's language code. So for Mexico_under_Carranza.djvu requested from English Wikisource, you can generate the hash with…

Ok, having gotten access to the project in connection with T265640 I've been trying to debug this a bit.

@JJMC89 Thanks!

@Candalua Thanks!

@Candalua That leaves you as the only admin on phetools with any likelihood of having the spare cycles to look at this (Phe and Tpt are highly unlikely to be available for this any time soon). Any chance you could poke around here a bit?

@Aklapper Indeed. Community-Tech was added as their Toolforge group account is one of the four accounts set as admin for the phetools Toolforge project.

In T244657#5995256, @JTannerWMF wrote:

… is this a challenge a lot of people are encountering?

@kaldari Nope, still seeing the same failure mode. It greys out the text in the editor and then throws an error in the JS console ala. An error occurred during ocr processing: /tmp/52004_6179/page_0199.tif.

In T265571#6549666, @Seudo wrote:

Apparently the HTML entities are fixed automatically in the English Wikisource (when I try in this book). ~~~~

This is a dup of T265571.

Not having access to T263371 it's hard to say anything intelligent about the specific issue, but…

Hmm. If one API endpoint returns unencoded text, then lower-level components seem unlikely culprits. If one API returns encoded text, then higher-level components seem unlikely. IOW: those results suggest to me that this is happening at the API layer.

In T256086#6515146, @Soda wrote:

… That isn't what this task is about... It's about deprecating a hook that specifically allows the proofreadpage extension to disable the javascript-based mobile editor. …

In T255345#6514822, @Soda wrote:

I'm not really sure if having the header and footer will help given that they will probably be in wikitext format and extracting it from the html would be easier than attempting to parse the wikitext.

In T256086#6484551, @ifried wrote:

… This would mean that, if no additional changes are made, wikis that use the ProofReadPage extension would not be able to disable the editor. Instead, all wikis with the extension would have the same, standard editor?

I didn't see the "Has somebody already looked at this edit?" aspect (i.e. mark as patrolled) in the above list. But maybe that's included in one of the existing bullet points?

@kamholz I don't understand what it is you're proposing to do here, nor see how it will have applicability outside just Balinese content. From whence comes #transliterate and what does it do? Why hard-code <br> inside ProofreadPage and provide two copies of the text? Why can this not be done with a normal template?

@kamholz You may possibly be interested in this, and there is some overlap with T259645.

In T259645#6362157, @kamholz wrote:

You have a good point about transclusion. I haven't looked into that yet, but presumably the pagelang should be applied there too?

How would this interact with the ability to set the language for a single Page: page? How about mainspace transclusions of Page pages?

@Aklapper Thanks, but as @TheDJ notes, going by that overview nobody owns multimedia features in WMF wikis now. That's a pretty sad state of affairs given how central multimedia is for almost all the projects (including Wikidata and whatever "Abstract" will end up as).

Just to note: Multimedia was tagged as a key player here, but that Phab team seems to have been archived. Who owns the components previously in that group's remit now?

In T258666#6330133, @daniel wrote:

… the import code will try to load the previous revision immediately after creating it. …

Presuming T212428 is an API race condition triggered by replication lag, this seems to be a different issue.

Note that T258666 looks like it could be an expression of this problem that was exacerbated by MediaWiki 1.36/wmf.1 (which is, I think, scheduled to hit the Wikipedias today). And as I commented there: I've never before seen this problem, and I've now seen 3 reports from 3 different projects in the last 24 hours. If so, this is not just a log spamming problem with the occasional user-visible weirdness any more.

The FileImporter issue is apparently T258666 and not obviously related.

I'd never seen the issue from T212428 (and I use FileImporter a lot), but in the last 24 hours I've seen this issue reported from enWS, jpWP, and by a Chinese user (not sure which project they call home) on Commons. If it's not actually more prevalent after deploying MediaWiki 1.36/wmf.1 that's a heck of a coincidence!

For the FileImporter issue mentioned on IRC see https://www.mediawiki.org/wiki/Topic:Vqmtib4ho61z4bji for two files that fail and the error message generated. That problem (and the possibly transient, possibly entirely unrelated ia-upload problem) were both reported in the last 24 hours; while the original issue in this task has apparently been happening for a month or thereabouts.

@Jdlrobson enwikisource has now switched over to a new version that's not reliant on the MF special casing. I haven't filed a separate Phab to disable it in config since it should drop by default in three days, but feel free to do so (or nudge me to do it) if you want it for tracking purposes.

Wouldn't it be easier and more robust to simply grab all the images directly on Commons? If MW doesn't already support JPEG2000 (.jp2; and I haven't checked) then either adding native support or transcoding to plain JPEG should be within the realm of reason (there's at least two OSS libraries for that), and a heck of a lot more robust then trying to dynamically fetch it from IA. We can create ProofreadPage Index: pages from individual images on Commons now, and if there was value in it I'm sure we could cook up some more fancy support (an index referencing a Category on Commons; or a virtual pseudo-file that just references those images; or built-in support for generating DjVus on demand; or …).

Sigh. Since GPU is either broken, or the correct module to pick there isn't mediawiki/core…

Possibly relevant to keep in mind when debugging this; PDF and DjVu files are not treated as just blobs of binary data on disk with just a little metadata in the db. Last I heard the hidden text layer gets extracted from the file, wrapped in an XML structure, and stuffed into a field in the database that is sized for metadata and can overflow. The more pages in a PDF, the more text, the more likely that's going to start playing a role. If this happens before the "upload" is considered complete, it might be a factor.

Everything Proofread Page knows would be useful to be able to access from JS: the associated Index: page, the page number, the previous and next pages, the values of the header and footer templates from the Index:, the CSS field from the Index: (quite possibly the other fields too, but I can't think of a use for them ottomh). Page name for Index and next/prev Page:, plus the {{{pagenum}}} are the obvious top priorities I'd say.

In T255073#6213215, @Billinghurst wrote:

I cannot say whether that was a global css statement or local. Either way, I didn't know think it should have display like that.

@Jdlrobson Any chance of a SWAT deploy on this?

T255073 potentially breaks tons of tools, gadgets, user scripts, etc. My call would be to hold until this is pinned down before rolling out to wikipedia.

This has broken things in awkward ways on English Wikisource (project-local javascript relies on only one such element existing). I've confirmed the same happens on Commons (Main Page). I've been told it only happens in Vector (but haven't verified yet).

In T199992#6178620, @Mahastama wrote:

We've stuck in how to make interface like this within the Wikisource:

where window A will send the material through Cakra API and the resulting OCR will be send into window B, just like in our server.

I'm going to go ahead and tag in the Readers Web guys on this, as it should be right up their alley; as input to the design process for Desktop Improvement if nothing else.

This is presumably not an issue that comes up often because it is rare that edit conflicts occur in ProofreadPage's Page: namespace pages. Each such page represents one single page of a physical book, and the editing done there is either transcribing the original text or correcting the wikitext to match the book, both tasks that are by nature solitary. It is relatively rare for more than one person to work on an entire book (everything covered by an Index:), and for a single Page: page it is exceedingly rare. I've seen this happen maybe twice in half a decade.

Ah, yes. I just ran across another instance of this. The problem is with extremely high-resolution (~4k x ~10k) JPEG-2000 files (particularly when combined with pathological codec settings during encoding) that exceed a hard limit in the Jasper library that GraphicsMagick uses. ImageMagick uses the openjpeg library, so it successfully deals with them, or you can use the utilities in the openjpeg library directly to convert these files.

Provided the "too many arguments" theory is correct, there are two obvious ways to work around it:

In T183087#6102416, @Krinkle wrote:

[…] When you use "Live preview", there will be new content every time you press "Preview changes". The hook from mediawiki.page.startup will run once when you open the edit page (and there is no content yet). The hook from live preview, will run every time for every preview.
[…]
I would welcome suggestions for how the documentation can be improved to summarise/clarify this understanding. The current phrase however, is not incorrect :)

In T244657#5995256, @JTannerWMF wrote:

Hey @ifried is this a challenge a lot of people are encountering?

Semantically the incomplete ref is part of the list, it just shouldn't get the backlink and marker. I don't think the <p> wrapper is important (in fact, it might be better without: cf. T49544). I think in general the short summary for these refs is that they 1) should be at the top of the list, and 2) should not have a backlink and marker. Everything else is implementation detail and practical considerations.

[ Apparently I didn't post this at some point back in January when I wrote it ]

@Tpt rtrim() removes trailing whitespace only, so this shouldn't affect leading newlines in the body of the page, should it?

@WMDE-Fisch Doesn't this just hide the problem? These newlines are coming from somewhere, and I can't think of a valid reason for them, so the root cause really should be fixed imo.

Since this seems to be essentially a dup of T242517 which was fixed by reverting; the part of the change that triggered this task was not an intended change in platform behaviour (it was an unintended side-effect of phan cleanup); and all the specific cases reported in this task have been resolved… I'm going to go ahead and "boldly" close this as resolved. @Ankry Please reopen if you disagree.

Could this be related to T241889?