Here two recent examples of IA Upload failures, recovered by xml2dsed:
https://commons.wikimedia.org/wiki/File:Caterina_da_Siena_%E2%80%93_Libro_della_divina_dottrina,_1912_%E2%80%93_BEIC_1785736.djvu

https://commons.wikimedia.org/wiki/File:Guidiccioni,_Giovanni_%E2%80%93_Rime,_1912_%E2%80%93_BEIC_1850335.djvu

Most of the recent failures are due this very same bug. See the log for any upload failure with a link for DJVU download.

I moved from plain use of _djvu.xml to the more complex _djvu.xml -> dsed conversion since dsed manipulation is really much more simple - the unique hard step being coordinate conversion. As soon as you get dsed format of OCR layer, you can use djvused routine, that is much faster and "elastic". I found too that some IA _djvu.xml are somehow bugged from origin, but that is possible to fix these bugs. I think that IA uses _djvu.xml just to get text coordinates needed to words search and highlight routine in its viewer, so that IA isn't so much interested into usability of _djvu.xml file to build a text layer into a djvu file.

@Alex_brollo I think this might be due to things like this in the djvu_xml file:

<WORD coords="x1,y2,x2,y2,dontcare">1</WORD>

where x1,x2,y1,y2 are all integers. If x1 == x2 || y1 == y2, the text layer will be corrupted according to djvused. It should be fairly easy to spot that and just drop the WORD if it happens. Or even set the co-ordinates to be he same as the prior/following WORD.

IA Upload failures come from a variety of _djvu.xml IA files, I suppose
that xml comes from _abbyy.gz. I noted two common mismatches in corrupted
xml files:

1. pages sometimes don't match with _jp2.zip list of images, since some

images are discarded;

2. xml file contains tags with no text or empty strings; such tags must be

removed.

Presently I'm not maintaining the script mentioned by Inductiveload, I
simply use Abbyy FineReader when IA Upload fails.

Alex brollo

Il giorno gio 11 giu 2020 alle ore 18:07 Inductiveload <
no-reply@phabricator.wikimedia.org> ha scritto:

Inductiveload added a comment. View Task
https://phabricator.wikimedia.org/T182778

@Alex_brollo https://phabricator.wikimedia.org/p/Alex_brollo/ I think
this might be due to things like this in the djvu_xml file:

<WORD coords="x1,y2,x2,y2,dontcare">1</WORD>

where x1,x2,y1,y2 are all integers. If x1 == x2 || y1 == y2, the text
layer will be corrupted according to djvused. It should be fairly easy to
spot that and just drop the WORD if it happens. Or even set the
co-ordinates to be he same as the prior/following WORD.

*TASK DETAIL*
https://phabricator.wikimedia.org/T182778

*EMAIL PREFERENCES*
https://phabricator.wikimedia.org/settings/panel/emailpreferences/

*To: *Inductiveload
*Cc: *Inductiveload, Samwilson, Ninovolador, Alex_brollo, Tpt