Ebook Exports: Formatting issues caused by the turning on of an option in calibre
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	ifried
	Jan 5 2021, 5:15 PM

Description

Background: There are some changes to the display of ebooks I believe caused by the turning on of an option in calibre (https://github.com/wsexport/tool/pull/297). Only affects formats which are produced by Calibre, like PDF, Mobi. Does not affect epubs. I believe the change was in order to improve performance of pdf generation.

Examples include:

The option tries to guess linebreaks in text, sometimes leading to undesirable results. Example below from https://de.wikisource.org/wiki/Bremens_edle_Tochter (https://wsexport-test.wmflabs.org/?lang=de&page=Bremens+edle+Tochter&format=pdf-a4&fonts=):
It tries to autodetect headlines. Example from https://en.wikisource.org/api/rest_v1/page/html/John_James_Audubon%2FChronology (https://wsexport-test.wmflabs.org/?lang=en&page=John_James_Audubon/Chronology&format=pdf-a4&fonts=):
It also does some nice things like adding italic text and making hyphenation consistent across the text. Example again from https://en.wikisource.org/wiki/John_James_Audubon/I. (https://wsexport-test.wmflabs.org/?lang=en&page=John_James_Audubon/I.&format=pdf-a4&fonts=):
It does a few other things as well, more details here https://manual.calibre-ebook.com/conversion.html#heuristic-processing. There are probably other differences I have not found yet.

Event Timeline

ifried created this task.Jan 5 2021, 5:15 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 5 2021, 5:15 PM

ifried updated the task description. (Show Details)Jan 5 2021, 5:17 PM

dom_walden updated the task description. (Show Details)Jan 5 2021, 5:18 PM

The revert of 297 is merged https://github.com/wsexport/tool/pull/301

dmaza claimed this task.Jan 14 2021, 11:17 PM

dmaza edited projects, added Community-Tech (Kanban-2020-21-Q3); removed Community-Tech.

dmaza moved this task from Ready 🎬 to In Development 💻 on the Community-Tech (Kanban-2020-21-Q3) board.

@dmaza Looks like this didn't get estimated yet. Any suggestion for the points it should be assigned? Thanks in advance.

In T271220#6752356, @ifried wrote:

@dmaza Looks like this didn't get estimated yet. Any suggestion for the points it should be assigned? Thanks in advance.

I would say 2 points ?

ifried set the point value for this task to 2.Jan 19 2021, 6:45 PM

I've managed to get the same visual results with and without --enable-heuristics by setting the following options --disable-delete-blank-paragraphs --disable-format-scene-breaks --disable-markup-chapter-headings --disable-unwrap-lines but I didn't get any gains in terms of performance. I tried multiple combinations and it seems that --disable-delete-blank-paragraphs is what slows things down but without it the PDF doesn't look right.
Anyhow, I've hit a wall and at this point this requires further exploration and maybe even look into calibre source to figure out what's going on and why generating the book goes from 10s to 60s in the presence of fonts.

To add more context, when adding the --enable-heuristics option to ebook-convert, the following book (Kapitanov_ključ) was exported to pdf in ~10s compared to ~60s without that option. And this is only an improvement when you are including fonts in the ebook, if no fonts are present the pdf is generated in ~10s with or without the option.

@ifried I don't really know where to go from here other than what I suggested above. Maybe we should move this into our backlog and possibly take a second look after we are done with our more pressing tasks

dmaza moved this task from In Development 💻 to Product sign-off 🤘 on the Community-Tech (Kanban-2020-21-Q3) board.Jan 21 2021, 12:26 AM

I'm removing the Community-Tech tag from this ticket. As a team, we have recently discussed this and determined that we will not be moving forward with this ticket. The only remaining work we do could do would be large & out of scope for our project (which will be closing soon), and there are no guarantees that it would result in us identifying the problem or what we can do about it. However, I will leave the ticket open, in case anyone else wants to investigate or dig into it in the future.

ifried removed a project: Community-Tech (Kanban-2020-21-Q3).Mar 17 2021, 11:23 PM

Restricted Application added a project: Community-Tech. · View Herald TranscriptMar 17 2021, 11:23 PM

ifried removed the point value for this task.Mar 17 2021, 11:23 PM

ifried removed a project: Community-Tech.

@dmaza: Removing task assignee as this open task has been assigned for more than two years - See the email sent to task assignee on Feburary 22nd, 2023.
Please assign this task to yourself again if you still realistically [plan to] work on this task - it would be welcome! :)
If this task has been resolved in the meantime, or should not be worked on by anybody ("declined"), please update its task status via "Add Action… 🡒 Change Status".
Also see https://www.mediawiki.org/wiki/Bug_management/Assignee_cleanup for tips how to best manage your individual work in Phabricator. Thanks!

Ebook Exports: Formatting issues caused by the turning on of an option in calibreOpen, Needs TriagePublicActions

Description

Event Timeline

Ebook Exports: Formatting issues caused by the turning on of an option in calibre
Open, Needs TriagePublic
Actions