Page MenuHomePhabricator
Create Task
Maniphest T177993

Article concatenation fails on large books
Closed, InvalidPublic
Actions

Description

Screen Shot 2017-10-11 at 12.52.24 PM.png (172×539 px, 18 KB)

Reducing the number of articles in the book using the book creator from 60 to 25 seems to allow the book to render.
This raises serious questions about whether our concatenation method will work at scale.

Note, that the limit seems to be on size of articles not number.
A similar book with 26 pages can be rendered:
https://en.wikipedia.beta.wmflabs.org/wiki/User:Jdlrobson/Books/The_big_book

Gergo has unanswered comments on the following patches (which maybe helpful for this task, but we should get to them in eventually):

  1. https://gerrit.wikimedia.org/r/#/c/381277/
  2. https://gerrit.wikimedia.org/r/#/c/381133/

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolved• JKatzWMFT150871 [EPIC] (Proposal) Replicate core OCG features and sunset OCG service
 InvalidNoneT186740 [EPIC] It should be possible to print a book using the Proton service
 InvalidNoneT178095 [EPIC] Fix problems with the PHP concatenation special page service
 ResolvedPRODUCTION ERRORphuedxT177988 Warning: Invalid argument supplied for foreach() in /srv/mediawiki/php-1.31.0-wmf.2/extensions/Collection/includes/DataProvider.php on line 152
 InvalidJdlrobsonT175868 Deploy and test new book rendering (Remex + Electron)
 ResolvedNoneT175882 Requesting access to scb* and pdfrender-admin for tgr
 InvalidJdlrobsonT178036 Book rendering database query error
 ResolvedJdlrobsonT178289 Restrict access to the testing URL for the new book rendering workflow
 StalledNoneT174670 Remove banner from saved books
 InvalidNoneT171832 Deploy new book renderer to all projects
 DuplicateNoneT171833 Deploy new book renderer to all projects side by side with OCG
 DeclinedNoneT173018 Add an option in Special:Book to download PDFs generated by ElectronPdfService
 InvalidNoneT173015 Use PDF post-processing service to generate final PDF
 InvalidNoneT173579 Expose PDF post-processing scripts as a stateless web service
 Resolved• bmansurovT171965 [Spike - 8 hours] How should the PDF post-processing script be exposed for use by Extension:Collection
 InvalidovasilevaT171960 Create a library to post-process PDF and add page numbers and table of contents
 ResolvedovasilevaT168871 Introduct toc with page numbers during pdf post-processing
 ResolvedJdlrobsonT176463 [Spike 8hrs] Investigate libraries for post-processing without non-JS dependencies
 ResolvedpmiazgaT171838 Build out article concatenation according to requirements for books
 ResolvedovasilevaT175856 Implement changes to article concatenation based on books requirements
 InvalidNoneT177993 Article concatenation fails on large books
 ResolvedovasilevaT177994 Book generation fails for articles with '/' character in title
 InvalidNoneT177996 Article concatenation not resilient to curl errors
 InvalidNoneT182230 [Spike] Explore ways of creating a stateless web service in Python
 Resolved• dpatrickT173014 Security review of pdfrw

Event Timeline

Jdlrobson created this task.Oct 11 2017, 7:54 PM
Jdlrobson removed the point value for this task.Oct 11 2017, 8:06 PM
Jdlrobson mentioned this in T175868: Deploy and test new book rendering (Remex + Electron).Oct 11 2017, 8:30 PM
Tgr subscribed.Oct 11 2017, 8:57 PM

Looking at logstash, there is a too-many-titles-for-query API error (I did point that out in code review but it got ignored), a bunch of other errors caused by that, and a (probably unrelated) memcached error (which helpfully just says "SERVER ERROR"). I did not think of memcached size limits which is probably what we are hitting here; the HTML part of the book might need to be split up into many cache entries or use something other than memcached.

Jdlrobson moved this task from Incoming to 2017-18 Q2 on the Web-Team-Backlog board.Oct 11 2017, 10:05 PM
ovasileva renamed this task from [subtask] Article concatenation fails on large books to Article concatenation fails on large books.Oct 12 2017, 5:28 PM
ovasileva removed a project: Readers-Web-Kanbanana-Board-Old.
Jdlrobson added a parent task: T178095: [EPIC] Fix problems with the PHP concatenation special page service.Oct 12 2017, 5:30 PM
Jdlrobson moved this task from 2017-18 Q2 to Upcoming on the Web-Team-Backlog board.Oct 13 2017, 12:13 AM
• bmansurov added a comment.Oct 13 2017, 12:28 PM

@Tgr fyi I saw your comments on my 2 patches, but the patches got merged before I could follow up on them. I think it's important I should follow up on them. Not sure if doing that work fits our priorities at this time, however.

phuedx added a comment.Oct 13 2017, 1:20 PM

@bmansurov: Could you add details about those comments and what the suggested fixes are to the description of this task?

• bmansurov updated the task description. (Show Details)Oct 13 2017, 1:23 PM
Jdlrobson moved this task from Upcoming to Needs Prioritization on the Web-Team-Backlog board.Oct 18 2017, 11:08 PM
Jdlrobson changed the task status from Open to Stalled.Oct 30 2017, 11:27 PM

Until we have time to talk through T178095

Jdlrobson moved this task from Needs Prioritization to Tracking on the Web-Team-Backlog board.Feb 7 2018, 7:20 PM
Jdlrobson edited projects, added Web-Team-Backlog (Tracking); removed Web-Team-Backlog.
Jdlrobson added a parent task: T186740: [EPIC] It should be possible to print a book using the Proton service.Feb 7 2018, 7:23 PM
• bmansurov unsubscribed.Feb 7 2018, 8:03 PM
ovasileva closed this task as Invalid.Apr 9 2018, 2:42 PM

Closing as per T184772#4116906. Pediapress will be taking on rendering of PDF books.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL