Indeed, this bug is related to c1f80efe05349cc39cb82dc9347b0a7699b11dfd (T304303): the live preview feature relies on the JS API allowing to fetch the current content of the editing area. To fix T183950 we dropped the hack inside of ProofreadPage that provides not only the page body but also the proofreading level and the page header/footer as page content. So, after this change, only the page body is fed to the live preview feature, hence the bug. A possible solution might be to add back the "complete page content" hack inside of ProofreadPage but some signficant work would be required to make features like find/replace work with this hack.

I have written an implementation draft: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikisource/+/793867

A fix has just been merged https://github.com/wikimedia/ws-export/pull/397 with a nicer follow up https://github.com/wikimedia/ws-export/pull/398

rel=prefetch has been added targetting the previous and next pages to speed up bulk proofreading according to T230689

@Xover Sadly there is not hope for it to be pushed today except if someone is managing to follow the emergency deployment process: https://wikitech.wikimedia.org/wiki/Deployments/Emergencies

High priority: breaks a part of the basic Wikisource proofreading workflow.

@Inductiveload Thank you!

I wrote a change that should fix the bug https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ProofreadPage/+/751946/

The bug happend again on the new disk with the "exhibitorsherald2*" files. I removed them to put the tool back on. We should maybe prevent files >1GB to be downloaded and processed.

@Xover Thank you so much for having found this issue. It was under my nose all this time.

@Mahir256 The change I wrote is in fact buggy. It's looking for a field named "toc" and not a field which data field is "toc". I'm writing a fix right now. Sorry for that.

@Soda do you have a HiDPI screen? If it's the case, I believe modern browsers will download and use the bigger image that is provided by the srcset attribute before the initialization of Openseadragon. Then Openseadragon is initialized and use the regular size image. It's what happens with my 4K screen configured with 2x scaling.

Sorry, I made a confusion between two different numerical systems: the ICU "beng" numerals are defined as the 10 digits "০১২৩৪৫৬৭৮৯" and, so, are not using base 16 as the currency numerals requested by this task. The base 10 numerical system is properly implemented and available now, the base 16 numerals are not yet availlable and not provided by ICU.

@Inductiveload I believe @Bodhisattwa says that the "beng" option is not visible in the pagelist editing widget.

@Thadguidry @BenAtOlive It's a great idea to move the conversation somewhere else. Oxigraph has a discussion space and a Gitter channel.

@BenAtOlive If I understand correctly, you are asking if Oxigraph can be used to build a distributed SPARQL database with reader nodes and writer nodes? Its definitely possible to reuse Oxigraph components for that. For example I am currently moving the SPARQL parser out of Oxigraph and I plan to add general SPARQL query optimizations to it. However, building a query evaluator against a distributed storage where data are not in the same machine as the one executing (or at least scheduling) the query is a quite different task as building a query executor against a single node database. Access latencies are easily an order of magnitude higher with a remote store and it changes quite a lot of tradeoffs. So, yes for making the part of Oxigraph that are easy to reuse reusable but sadly I am not sure that naively spliting the query evaluator out of Oxigraph would provide a good query evaluator on top of a distributed storage.

The change fixing this issue is now merged.

It's a great idea! I just believe P2860 (cites work) should be removed from the list. It's definitely not a classifying property.

It is now fixed. There was some broken job data in the file system because of the full disk problem. I have opened T292365 to prevent it to happen again.

It's because the disk is currently full. I have removed a few huge books from the job queue and rebooted the server. However it does not seems to be enough to bring the website back.

@Mpaa @Dvorapa @Dalba Sorry for distrubing you. Is this problem fixed now?

@Ladsgroup Thank you! So what would you think about MCR for storing the "main" data and some change tags as "secondary" data for queryability and recent changes filtering? This way we would get the best of both worlds.

Thank you @Tgr and @Ladsgroup for your feedbacks!

@Tgr @Legoktm Sorry for bothering you. I believe you where at some point involved in with change tags system. Do you think our plan seems sensible?

@Inductiveload Great question! We might start now. Maybe with oldwikisource?

In T290243#7328312, @Dovi wrote:

This still doesn't explain how the fields were there earlier, as the current text shows (which cannot be edited).

@Aklapper Thank you! Most of Wikisource now use that is an improved version of Mediawiki:Proofreadpage_index_attributes. MediaWiki:Proofreadpage_index_data_config is defined in the Hebrew Wikisource.

@Thadguidry Great! Thank you!

@Thadguidry Hi! It's a great idea to improve the roadmap. Thank you for pushing me to do that! I have updated the GitHub issues and milestones and added a link to them in the README:
Each milestone should now have a short description and a set of linked issues. Most of the issues have been connected to a milestones.
I am a bit afraid of adding a ROADMAP file to the repository because it is yet another source of truth to maintain.
To also keep a single source of truth I made the choice to not expand the milestone description too much and keep the detailed discussions inside of issues. Here is [one about the storage backend and Sled]( https://github.com/oxigraph/oxigraph
/milestones?direction=desc&sort=completeness&state=open). However, the milestone+connected issues descriptions should allow to make sense of the roadmap. Feel free to ping me if you believe it's not clear enough.

Thank you @So9q for opening this task!

I'm working on a fix. Should be ready soon. Sorry for the troubles.

Fixed and deployed.

Sorry, I should not have closed this issue. The change that should fix this problem is not merged yet: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ProofreadPage/+/704564
So, it makes sense that the problem does not seem solved on Wikisource.

How widespread is the impact of this bug remaining unfixed? If it's a nitch use and we can fix it with a backport in the short term then I wouldn't roll back. On the other hand, if this is causing some fallout elsewhere or triggering major production breakage then definitely roll back.

Yes. To make it work properly "data": "toc" should be added to the relevant field in Mediawiki:Proofreadpage_index_data_config.

@DannyS712 Thank you! I have understood why e.g. https://phabricator.wikimedia.org/diffusion/EPRP/browse/master/includes/Pagination/Pagination.php$25 is not flagged by these sniffs. It's because abstract methods are not covered by the lint: https://github.com/wikimedia/mediawiki-tools-codesniffer/blob/5f813f727ee47d5c90981b44e82f0c5a0263e933/MediaWiki/Sniffs/Commenting/FunctionCommentSniff.php#L198

Thank you @Ankry, @Xover @Inductiveload! So I guess rel="preload" seems the way to go for the next page image. What about using rel="prefetch" for the previous one? This way, it gets a good chance to be loaded too but with less priority. What do you think about it?

@Ankry Thank you! So, I guess that the main goal is to make sure that the image thumbnail is already generated when a Page: page is displayed.

If I remind correctly preload is to ask the browser to aggressively fetch a content for the current page rendering and prefetch is to ask the browser to load the content for future use in the navigation.

But aiui the granularity would be per-project and not per-index. How would a project start experimenting with this or avoid the need for a big bang migration? Use two toc fields in the index, one for the traditional way and one for a "Wikidata-based toc"? That would necessitate PRP differentiate between "present" and "non-empty" for the index data field tagged with "data": "toc", I think. That sounds a little more complicated, but is perhaps not too much so?

Thank you @Mahir256 and @Xover for this discussion.

The "Source" tab is currently inserted using javascript ("modules/article/ext.proofreadpage.article.js"). It might be possible to use the same trick for Minerva. But migrating everything to PHP would be nicer.

@Tpt Is there a technical reason for this?

Done and deployed

Sorry for stepping in. A maybe relevant task when designing a new zooming system: T43614

@He7d3r @GOIII are you fine with me closing this task as "declined" or do you want to move this forward, maybe by getting feedback from the Wikisource community?

After thinking a little bit (and asking my wife who does not contribute to Wikisource), I have a feeling using an indicator is maybe not highlighting enough the proofreading level information: it's the first thing people are often looking for if they go to a page: page. But I am not an UX design expert so maybe an indicator is good enough.

Thank you for pushing this. I am a bit afraid it might create a backlash with some contributors feeling that the proofreading level is not highlighted enough. It might be nice to start a discussion on the biggest Wikisource communities. A first step is maybe to move the quality level from the <pages> tag and in the Index namespaces to an indicator.

This is presumably very nearly T239033, since if the extension can add a "Source" link for a mainspace page user of <pages/>, it can do it in any other namespace?

@Inductiveload Yes, I guess it should be also done in onSkinTemplateNavigation. The tricky part is that the "source" link is not known from the current page title but from the parser output, and onSkinTemplateNavigation do not have access to the parser output. It's why I have not done the migration to server side yet. There is maybe a good solution, I have not taken much time to look for an alternative yet.

@Tpt, is that fix complete or is there more that needs to be considered.

@Inductiveload https://gerrit.wikimedia.org/r/672760 generated "null pointer exceptions" during Page: pages rendering if the page was not already connected to an index (trace in T278379). Antoine and I reverted the change to fix this error. It was the simplest way we found to fix this error quickly on Wikisource, sorry. I have started to write a change that should fix this problem: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ProofreadPage/+/674691/1/includes/Page/PageContent.php
Feel free to integrate it into your change and push it again to gerrit.
Sorry again the quick revert.

@Tpt: hmm, but how will you then get the prp-pagequality-N qualityN classes applied? Or do that manually?

It should be automatically as part by the LinkRenderer: the classes are added by a linker hook.

@Inductiveload I believe we could switch <pagelist> rendering to outputting HTML. $parser->getLinkRenderer()->makeLink allows to easily reuse MediaWiki linking system.

@Inductiveload Yes, ProofreadPageInit.php is a good place for that.

Not sure if that's a use case that is needed?

@Inductiveload I believe it should be better to use templatestyle because it does a lot of nice and useful CSS transformation and sanitization.

@Aklapper @jayantanth Sorry for this problem. It's now fixed. I have changed the "index template" to allow @Tshrinivasan to play with Index: template rendring using Lua modules.

Why is MOBI described as being for Calibre? I would've thought EPUB would be closer to Calibre's "default" format.

@Urbanecm Sure. Here it is: https://wikisource.org/wiki/Wikisource:Scriptorium#MediaWiki_Wikisource_extension_installation

@Yash9265 wrote a change that removes navigation links when transcluding Special:IndexPages. I believe this problem is now solved.

I believe we want to display the export links only in namespaces where ProofreadPage allows the transclusion to happen and displays its navigation links.
For now it's only the main namespace but some Wikisource requested to be able to use it in other namespace (T53980).
Possible way to fix both problems at the same time is maybe create a per-wiki config inside of ProofreadPage and reuse it for the Wsexport link.

We don't show the links on https://wikisource.org. I don't know if this matters. I don't know if users use that site.

The change adding "thai" numerals support is now deployed on Wikisource.

Sorry, I confused this task with an other one.

My 2 cents: I like the idea of using Calibre ePub to ePub to do the file split in case of too big files. The current implementation in Wsexport is very bad. An other option would be to fix it inside of Wsexport by having a look of what Calibre is actually doing internally.

So, the question is: Can the ProofreadPage extension be smarter about where the pagenum template is inserted to avoid putting it in "dead" table space?

Thank you! It looks like a great plan!