Memory issues for IA-upload when converting large files to djvu
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	Billinghurst
	Mar 25 2017, 9:20 AM

Description

When trying to upload and convert the work Armagh clergy and parishes it failed after converting individual pages

armaghclergypari00lesl Armagh clergy and parishes.djvu In progress View log

log ...

[2017-03-18 09:50:19] LOG.INFO: Creating DjVu for armaghclergypari00lesl from Jp2 [] []
[2017-03-18 09:50:19] LOG.INFO: Saving IA metadata to /mnt/nfs/labstore-secondary-tools-project/ia-upload/ia-upload/jobqueue/armaghclergypari00lesl/metadata.json [] []
[2017-03-18 09:50:20] LOG.INFO: Downloading armaghclergypari00lesl/armaghclergypari00lesl_djvu.xml [] []
[2017-03-18 09:50:25] LOG.INFO: Downloading armaghclergypari00lesl/armaghclergypari00lesl_jp2.zip [] []
[2017-03-18 09:52:30] LOG.INFO: Unzipping /mnt/nfs/labstore-secondary-tools-project/ia-upload/ia-upload/jobqueue/armaghclergypari00lesl/armaghclergypari00lesl_jp2.zip [] []
[2017-03-18 09:53:19] LOG.DEBUG: Zip file extracted to /mnt/nfs/labstore-secondary-tools-project/ia-upload/ia-upload/jobqueue/armaghclergypari00lesl/armaghclergypari00lesl_jp2 [] []
[2017-03-18 09:53:19] LOG.INFO: Processing JP2 files [] []
[2017-03-18 09:53:19] LOG.INFO: Converting 518 individual JP2s to DjVus [] []

...

[2017-03-18 10:21:08] LOG.INFO: Merging all DjVu files to /mnt/nfs/labstore-secondary-tools-project/ia-upload/ia-upload/jobqueue/armaghclergypari00lesl/armaghclergypari00lesl.djvu [] []
[2017-03-18 10:21:30] LOG.INFO: Modifying DjVu XML file /mnt/nfs/labstore-secondary-tools-project/ia-upload/ia-upload/jobqueue/armaghclergypari00lesl/armaghclergypari00lesl_djvu.xml to add /mnt/nfs/labstore-secondary-tools-project/ia-upload/ia-upload/jobqueue/armaghclergypari00lesl/armaghclergypari00lesl.djvu [] []
[2017-03-18 10:21:33] LOG.INFO: Merging modified XML into full DjVu file [] []

@Samwilson said that it was a memory issue.

Related Objects

Mentioned In: T173908: job-queue too long
T159796: IA Upload: "Failed to get specified page" DjVu generation error

Event Timeline

Billinghurst created this task.Mar 25 2017, 9:20 AM

Restricted Application added a project: Internet-Archive. · View Herald TranscriptMar 25 2017, 9:20 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Billinghurst moved this task from Backlog to IA Upload on the All-and-every-Wikisource board.Mar 25 2017, 9:21 AM

It's looking like items with more than about 400 pages (the one mentioned here is 518 pages) run into this problem. It correctly converts each individual page, and then merges them into one single DjVu file, but when it comes time to update the OCR data from the XML, a guru meditation happens.

@Samwilson Will increasing the -mem for the job help? https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Grid#Allocating_additional_memory

• DannyH moved this task from New & TBD Tickets to Needs Discussion on the Community-Tech board.Apr 4 2017, 11:25 PM

• DannyH triaged this task as Medium priority.Apr 4 2017, 11:31 PM

Let's up the memory to 3 gigs and put a 500 page limit on the script.

kaldari lowered the priority of this task from Medium to Low.Apr 4 2017, 11:33 PM

kaldari set the point value for this task to 3.

kaldari moved this task from Needs Discussion to Up Next (June 3-21) on the Community-Tech board.

I've increased the memory allowance to 3 GB and File:Armagh clergy and parishes.djvu has been uploaded. There will be others that will still fail; I'll keep an eye on the queue and see what the page count is.

didn't have a text layer once uploaded to Commons :-(

Bodhisattwa subscribed.Apr 7 2017, 12:45 PM

Samwilson mentioned this in T159796: IA Upload: "Failed to get specified page" DjVu generation error.May 11 2017, 1:11 AM

Samwilson edited projects, added IA Upload; removed All-and-every-Wikisource.May 25 2017, 11:29 AM

kaldari edited projects, added Community-Tech; removed Community-Tech-Sprint.Jun 6 2017, 11:23 PM

kaldari moved this task from Up Next (June 3-21) to Needs Discussion on the Community-Tech board.Jun 20 2017, 11:14 PM

kaldari moved this task from Needs Discussion to Up Next (June 3-21) on the Community-Tech board.Jun 20 2017, 11:38 PM

Samwilson mentioned this in T173908: job-queue too long.Sep 12 2017, 12:09 PM

Samwilson merged a task: T173908: job-queue too long.

Samwilson added a subscriber: VDeFrancescaBEIC.

Maybe we could put in a sanity check to prevent people from trying to upload files that are above a certain really large size to keep people from running into the memory limit (at least for now).

kaldari edited projects, added Community-Tech-Sprint; removed Community-Tech.Dec 6 2017, 12:25 AM

Some recent *jp2.zip file sizes:

Size (MB)	Page count
99.7	230	https://commons.wikimedia.org/wiki/File:Rovetta_-_Baby_e_tiranni_minimi.djvu
406.6	386	https://commons.wikimedia.org/wiki/File:A_daughter_of_the_rich,_by_M._E._Waller.djvu
147	833	https://commons.wikimedia.org/wiki/File:Annali_della_Scuola_Normale_Superiore_vol._VI_1889.djvu
450.3	523	https://commons.wikimedia.org/wiki/File:Ganita_Sara_Sangraha_-_Sanskrit.djvu

I've added a warning (GH37) for when it's over 600 MB or 800 pages, but not prohibiting the upload, because it looks like it runs out of memory more often in processing either the scans (converting and resizing to jpg) or merging the OCR XML. Both of those things happen later, and so we can't warn the user at job-creation time.

Maybe this will be sufficient.

I'll encourage people to report bugs when they find hung jobs.

Samwilson removed a project: Internet-Archive.Dec 7 2017, 4:48 AM

PR37 deployed.

I think this can be closed; the other memory-related issues are being tracked separately, or if they're not then specific issues will be opened for them.

kaldari moved this task from Needs Review/Feedback to Q1 2018-19 on the Community-Tech-Sprint board.Dec 9 2017, 2:48 AM

• DannyH edited projects, added Community-Tech; removed Community-Tech-Sprint.Dec 19 2017, 12:54 AM

• DannyH moved this task from Up Next (June 3-21) to Archive on the Community-Tech board.Dec 19 2017, 1:14 AM

Memory issues for IA-upload when converting large files to djvuClosed, ResolvedPublic3 Estimated Story PointsActions

Description

Related Objects

Event Timeline

Memory issues for IA-upload when converting large files to djvu
Closed, ResolvedPublic3 Estimated Story Points
Actions