In T244657#5995256, @JTannerWMF wrote:

Hey @ifried is this a challenge a lot of people are encountering?

Semantically the incomplete ref is part of the list, it just shouldn't get the backlink and marker. I don't think the <p> wrapper is important (in fact, it might be better without: cf. T49544). I think in general the short summary for these refs is that they 1) should be at the top of the list, and 2) should not have a backlink and marker. Everything else is implementation detail and practical considerations.

[ Apparently I didn't post this at some point back in January when I wrote it ]

@Tpt rtrim() removes trailing whitespace only, so this shouldn't affect leading newlines in the body of the page, should it?

@WMDE-Fisch Doesn't this just hide the problem? These newlines are coming from somewhere, and I can't think of a valid reason for them, so the root cause really should be fixed imo.

Since this seems to be essentially a dup of T242517 which was fixed by reverting; the part of the change that triggered this task was not an intended change in platform behaviour (it was an unintended side-effect of phan cleanup); and all the specific cases reported in this task have been resolved… I'm going to go ahead and "boldly" close this as resolved. @Ankry Please reopen if you disagree.

Could this be related to T241889?

Yes, I just retried handbookofnature00coms_0 and that is an instance of T215647:

Hmm. I grabbed the JP2s and ran gm on them, and the issue seems to be that these files, by virtue of being a pretty ridiculous resolution (6851x10000), exceeds a hard-coded limit in—in my case—the Jasper library that gm uses for JPEG-2000 support. I believe gm can be linked against different libraries for this (and the particular problem is a Jasper thing) which probably explains some variability in symptoms: different distributions will have linked different underlying libraries, and it may change with OS updates. I think this specific scan is best considered a pathological case.

Apt-ark uploaded this file to Commons manually (as File:Arkansas_Constitution_1874_(published_1913).pdf) so this task can probably be closed as stale now.

The logs (and other files) get deleted after some time so this now seems rather hard to debug (pro tip: upload the logs as attachments, or excerpt the relevant parts in a comment, to preserve them).

This is a duplicate of T215647.

This is almost certainly the same issue as T215647 (but that task has more log data).

@awight It seems all wikis were rolled back to wmf.16 (including the l10n cache) for other reasons, so we're now back to the status quo and <ref follow="foo"> works as normal again. As I understand it they haven't decided yet whether next week's deploy will be retrying wmf.18 or moving on to wmf.19, but from my (very limited) understanding we should not run into this particular issue again either way.

@awight Ok, so the summary seems to be: when the next planned "full scap sync" happens on Tuesday the l10n cache CDBs will get rebuilt with the correct messages, and either on Tuesday or Wednesday—depending on in which phase the caches get pushed out—the missing message key will become available on all the Group 1 wikis and this problem will go away. Is that roughly correct?

@awight Per Nemo above (and on IRC) there is some chaos surrounding this week's deploy train, and issues related to that which are keeping the key people busy. It sounds plausible that this and those issues may be related; but I have no details on what's going on beyond the task Nemo linked above (T233866, <s>which just looks like the deployment tracker to me</s> Nope, I see it's been updated with significant and relevant breakage.).

Just for completeness: I've verified that the problem occurs on plWS as well as enWS.

@Billinghurst I'm told there are other ongoing issues that may make getting the right eyes on this problem difficult with any urgency (cf. Nemo's message above). But as I recall you're a global interface admin: your thoughts on requesting the global interface admins on meta to mass add this interface message on all the affected projects? Details of what's needed in Thiemo's message above. We'd need to also remove it again once whatever is causing this is fixed, which I'm not sure whether the interface admin bit is sufficient for.

@thiemowmde Theory: the initial change removed the message, which removed it from translatewiki.net. The deploy started to Group 0 on Tuesday, at which point messages were sync'ed from translatewik. The patch was reverted on Wednsday before deploy to Group 1, but still with the translatewiki sync from Tuesday. Translatewiki was updated with the restored key, but that change won't be sync'ed back until the next scheduled deploy on Tuesday next.

Hmm. So essentially list-defined references, but with the list of named references defined on a separate wikipage rather than at the end of the same wikipage?

An attempt at some usage / requirements type notes… (and pinging @Tpt as they may wish to be aware of this task)

In T240858#5851392, @thiemowmde wrote:

I'm super confused right now.

Ok, I was hoping to get clarification on actual behaviour before drawing any conclusions, but since this is scheduled to go out today in the ongoing deploy of MediaWiki 1.35/wmf.18…

In T49544#5836366, @ssastry wrote:

Let us please not make changes like changing <span> to <div> without thinking through the implications.

In T49544#5834946, @thiemowmde wrote:

I was very specifically asking for an example of bad rendering in a browser.

In T49544#5834514, @thiemowmde wrote:

I feel the situation was a lot different back in 2013. It's 2020. We are not talking about XHTML any more. We use HTML 5, a "living standard". At this point, is this an academic discussion about the semantics of HTML, or does this have real-life consequences? If so, in which browsers? Is there a specific example we can look at? (Ideally with a revision ID and screenshot.)

In T49544#5507721, @beleg_tal wrote:

So... why are we not simply replacing the span with a div? We've shown several use cases where span causes problems, and I can't even think of any where div causes problems

@Tpt "No decode delegate for this image format" means the requisite image format library is not available (at runtime presuming gm is dynamically linked). First thing to check is whether the file in question is really JPEG-2000, or a mislabeled PNG/TIFF/PPM/whatever (in.ernet.dli is the Digital Library of India, and they have… uhm… original… ideas of how to encode their stuff; I often find PNG labelled as .jp2 or .jpg); and the second is to check "gm version" output for support for that file format.

Looks good to me!

@Umherirrender Many thanks!

@Mpaa By my assessment it cannot have been done deliberately, as the modified code will never work in the context it is evidently being used. When combined with the misleading commit message referencing Phan, my guess is that this was an attempt to fix a warning generated by the static analyzer that just failed to spot the need for armoring there.

See also T242422.

This problem coincides with deployment of MediaWiki 1.35/wmf.14 today, but I find nothing obviously connected in the changelog. A few changes mention touching calls to int(), but not in any context that would obviously be connected to this issue.

@bd808 Adding 5 specific Perl modules and an unrelated binary (Tesseract) to the generic image isn't really a dup of a moderately large new platform architecture change.

In T135963#5543136, @Bawolff wrote:

Ideally [talking to Flickr API] would be done through the server side to preserve user privacy.

@Koavf Which gadgets are enabled and which are on by default is up to each project.

In T228594#5753585, @Koavf wrote:

I clearly don't know everything that's involved but there is an OCR gadget that works on en.ws, so if that functional gadget can't just replace the standard OCR button in the interface somehow, I'm at a real loss.

@Tpt First: you're awesome! :)

@Koavf It's not that simple. Hit me up on my enWS user talk page if you want the nitty gritty details. The short version is that Phe's OCR gadget (which is what this task is about, and which is what I assume you mean by the "default") is based on Tesseract, but that's not where the problem is: it's somewhere in the custom code Phe has written to provide the interface to Tesseract or its interaction with the server infrastructure on Toolforge.

@Adithyak1997 Does this help at all?

In T228594#5752675, @Peteforsyth wrote:

Is that accurate?

There may or may not be a different manifestation of this problem in T240562.

It looks like this could possibly be a manifestation of T192866. In my testing, removing the text layer from a sufficient number of pages makes the problem disappear; but it does not appear removing any single (potentially problematic) page has any effect.

In T228594#5749785, @Koavf wrote:

I don't understand the distinction here.

@Tpt This is almost certainly not the same issue as T204020 (which is a dup of T219376, where some possible approaches to fix it are suggested). 20402 is basically the result of a naïve text layer extraction algorithm in MediaWiki-DjVu that fails badly when faced with unexpected data in a DjVu file, combined with the fact that the DjVuLibre tools will happily accept invalid input that it will choke on when asked to output it again. MediaWiki-DjVu can absolutely be modified such that it will fail gracefully in these cases. But, crucially, such DjVu files can be reliably identified (djvutxt --detail=page file.djvu | egrep "^failed"), and the example from this task does not exhibit that problem.

This is the same issue as T219376 which contains some possible approaches to fix this.

It looks like the mentioned javascript error in the console is gone now (as of 1.35.0-wmf.10 today; I can't recall if I checked on wmf.8 due to the recent hiccup in the deployment calendar), but the text layer is still not loading in the example page. Absent any console messages I have no idea how to debug this further. Are there server side logs that can be checked? A test system where it can be traced? Should I open a new task for that, and if so, with what tags or projects so that it has a chance to hit the right teams' radar?

@Aklapper See link in task description: https://en.wikisource.org/w/index.php?title=Page:Atalanta_-_Vol._2.djvu/18&action=edit&redlink=1

Incidentally, the correct place to solve this is in ProofreadPage because it has knowledge of what the next page in the sequence is. However, I suspect it would need facilities from core to do this intelligently: standard prefetching requires injecting a <link rel="prefetch" href="…" /> in the page, preferably before the browser sees it. ProofreadPage could of course "cheat" the way this gadget does (good version 1.0 / proof of concept / minimum viable product goal?), but that'd be kinda complicated, inelegant, overly specific, and prone to breaking over time.

Possibly related: T236469

This is a common operation on the Wikisources: what on a Wikipedia would be "a page" is on Wikisource "this collection of pages in multiple namespaces". Thus having a list of pages, possibly into the thousands (one per page of a book, for example), is a very common scenario. Based on recent experience on enWS it is even more common than mass deleting all contribs from a particular user (most of our recent spammers have constrained themselves to a single page).

Is it expected behaviour that mw.loader.using('mediawiki.RegExp', function() {…}) will fail silently?

Hmm. While DjVu and PDF (and, I think, TIFF) has explicit text layers; any kind of image can in principle contain text and could benefit from a structured way to store the OCR as actual text. We have oodles of images-of-text in JPEG, PNG, etc. formats in addition to the "book" formats (DjVu, PDF). Even book scans are not-infrequently uploaded as a couple hundred JPEGs gathered in a category (if we're lucky).

Ok, a couple of things here…

As the single community member (and admin) who expressed reservations there, I can also confirm that the discussion was a valid community consensus process; that the thread's closing was appropriate for enWS norms; that the closing comment accurately reflects the community's intent; and that this request is within the scope of that consensus. If needed we can get @Hesperian or @Mpaa (our bureaucrats) to confirm this here or on-wiki.

PS. Also: 13 years since this request was filed y'all! Probability of getting addressed tends to be inversely proportional to age of the request, so this is now the third least likely request to be addressed that I follow.

In T8808#1666317, @Billinghurst wrote:

Are we looking for a fourth option of "brand new name"?

Hmm. Perhaps this issue could also be solved by remex being intelligent about where this particular span belongs? At least inside a <tr>, or outside a <tr> but inside a <tbody>, the correct placement of it is inside the immediately preceding <td>. If remex knows the difference between ordinary page content and the stuff inside the ProofreadPage-specified interface file it should be feasible to do this at that layer with the existing infrastructure there.

In T228594#5478755, @Aklapper wrote:

[…] What happened or has changed that this task suddenly became "Unbreak Now!" priority, weeks after it was created?

Just an idle thought…

Now it's inserting extra whitespace before the close brackets of the citation, and it's still inserting duplicate links in |url= and |pages=. See https://en.wikipedia.org/w/index.php?title=Hamlet&diff=909523776&oldid=909485888&diffmode=source

Hmm. I'm not sure what's going on in Ineuw's browser to cause that ve.init undefined error, but I pretty sure the general OCR gadget issue that several people on enWS have reported is related to this console message (that I'm surprised Ineuw isn't seeing, but it may be being hidden by the ve.init issue):

[Error] Error: Syntax error, unrecognized expression: An error occurred during ocr processing: /tmp/52004_20706/page_0011.tif
	error (load.php:20:880)
	select (load.php:37:233)
	find (load.php:41:184)
	init (load.php:142:753)
	jQuery (load.php:2:505)
	hocr_callback (Script Element 1:594:825)
	fire (load.php:45:980)
	fireWith (load.php:47:174)
	done (load.php:126:628)
	(anonymous function) (load.php:130)

Which is an unhandled error emitted by this line (it looks like) of phetools.

Supporting move to "All wikis in a given language" is probably a pretty tall order since this was made as a one to one tool (and how would you merge multiple divergent edit histories?). But if we could get, say, FileExporter on Commons and FileImporter on English Wikisource, it would make it a lot easier when we have to repatriate books that are about to be deleted on Commons. That should be at least within the realm of a reasonable short term goal; and with that in place it may be possible to create user scripts or other external tools that can semi-automate the "All wikis in a given language" bit.

I cannot reproduce this either.

Some thoughts from a dabbler in user scripts on WMF wikis...

This looks to me like MediaWiki-DjVu is choking on the new file and thus never regenerating the thumbnails / page images. Not obvious that ProofreadPage is involved in this at all (except that this is where it becomes most visible). I downloaded the actual .DjVu from Commons and confirmed that the file indeed contains the fixed pages.

@eugrus The page numbers are generated by a community-maintained script, loaded from https://en.wikisource.org/wiki/MediaWiki:Common.js, that lives in https://en.wikisource.org/wiki/MediaWiki:PageNumbers.js. You will need various bits of supporting code from Common.js (in addition to loading the PageNumbers.js script itself) in order to get it to work.

From a technical perspective, the only clear advantage to the status quo would seem to be the OCR engine (ABBYY) that IA has access to but we (currently) do not.

All three text fields resize ok vertically here (they used to resize also horizontally, but I believe that was changed deliberately a while back). Their default size appear to be two lines.

Quick update from enWS: 1.33/wmf.12 is now live there and a quick check suggests it fixes the disappearing header and footer fields. IOW this probably confirms this was caused by T209939.

Quick update from enWS: 1.33/wmf.12 is now live there and a quick check suggests it fixes the disappearing horizontal rule. IOW this probably confirms this was caused by T209939.

Quick update from enWS: 1.33/wmf.12 is now live there and a quick check suggests it fixes the disappearing horizontal rule and the tiny/uneditable header and footer fields. No structured testing done, but it looks very promising.

Just to note for future reference, there have now been reports of this happening also in the browsers PaleMoon and WaterFox, who both seem to be Mozilla/Firefox forks, as well as Internet Explorer (unspecified version). My own tests suggest the "Horizontal layout" option has no effect either way on this (turned it off, no change), except in so far as it probably enables the "Show header and footer fields" option just above it (not sure what the default is for that). And if the header and footer fields aren't displayed at all you will of course not be able to reproduce any problems with them.

@ShakespeareFan00 The deployment that (aiui) rolls back this change isn't yet live on enWS. See most recent Tech News and the thread "Issue with the FI Template display" on the Scriptorium. Based on the schedule given in Tech News, the deployment should hit enWS at some point today (or possibly tomorrow, depending on how they schedule the rollout).

@stjn I did not say that <style> outside <head> is wrong (it isn't: <style> is allowed anywhere "flow content" is allowed).

Hmm. This looks like a pretty naive implementation. I had expected TemplateStyles to parse out any <templatestyles> elements and inserting them in the <head> element (or through ResourceLoader or something): expanding them inline is going to create loads of problems, of which this is a perfect example.

@Aklapper It seems very likely. Chromium is a fork of WebKit (the Safari engine) which exhibits the same problem (cf. the comments in the other task), and the timing (at least roughly) of this problem appearing coincides with the deployment of the changes there.

@EsmeShepherd The developers are aware of the problem you are experiencing, and the plan is (aiui) to fix it (by rolling back the changes that caused it) on 7 or 8 January. At that time, or shortly after, the header and footer fields should return to the size and behaviour they had previously.

@Tpt @TheDJ @Samwilson Looking quickly at the patch here, this is not something I imagine it's possible to do using only CSS changes. Flex box is a completely new layout model and is almost certain to require code (generated markup) changes and interact badly with other styling and scripts (including the markup produced by templates or hard coded in pages). As an example, imagine what changes would be necessary to let users switch back and forth between the two as skins. My gut feeling is that this would have to be a part of a "ProofreadPage UI 2.0" level effort in practice.

@Ankry The disappearing header and footer is more likely than not an issue only in Safari, or possibly on Retina-resolution devices (1pt = 4px, aka @2x, or higher), based on the reports I've seen (and it's probably some wonkyness related to font size: manually removing the font-size: 13px set on the text field using the web inspector makes the text field large enough to use).

@ShakespeareFan00: Either. Edit filters, aiui, can be managed more easily; but, based on my limited understanding, either should be able to solve this problem without actually creating an entirely new log.

Could not this (and would be better as) a javascript Gadget? It's just chaining together two existing operations so they can be triggered by a single button click. It would still need to be implemented, of course, but a Gadget sounds far less resource-heavy than changing the MW editing interface.

Could the log aspect perhaps be done using an edit filter? That wouldn't help retroactively (I don't think), but for all new edits they would be searchable.