Page MenuHomePhabricator

Cache problems with new Index pages in Wikisource
Open, Needs TriagePublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

  • Upload a new pdf or djvu file to Commons
  • Create an index page in Wikisource

Example Index: https://en.wikisource.org/w/index.php?title=Index:Mizo_Chanchin_-_Liangkhaia.pdf&action=edit
Example file: https://commons.wikimedia.org/wiki/File:File:Mizo_Chanchin_-_Liangkhaia.pdf

What happens?: Instead of generating the pagelist the red message "Error: Invalid interval" appears.

This can be usually solved by purging the cache of the file both in Commons and in Wikisource, but new and unexperienced editors usually do not know how to do it, and so fixing these cache problems is really needed.

Event Timeline

This is a fairly common issue with media from Commons and is not really a PRP issue itself. PRP requires metadata about the media to be available and correctly yields errors when it cannot get such. The typical workaround it to issue a purge on the media on Commons forcing an update there and all sites using Commons to remote host media.

@Uzume As I have mentioned in the original description, experienced editors can solve it by purging the pages, but newbies and other unexperienced editors, some of which even do not know what cache is, get stuck in that moment and some of them might simply leave the unfriendly site. Wikimedia projects must strive to be newbies friendly.

@Jan.Kamenicek Yes, but what exactly do you want PRP to do about that? About the only thing it might be able to do would be to improve the error messaging under those circumstances. PRP has no control over how media are hosted, remotely on Commons or otherwise. The problem description does define such things but it does not really specify what you want to happen. Also this issue is current tagged/attached to PRP so presumably you are looking for a solution in that space but short of potentially improving some error messaging there is really nothing PRP can do about the situation.

There is no way for PRP to generate a pagelist without determining the "interval" based upon the media metadata and there is no way to get the media metadata without the media being locally cached properly something which PRP has no direct control over (and thus it cannot possibly hope to fix).

I do not know ''what exactly'' can be done about that. I just know that it was not happening and now it is happening, which means that something must have been changed, which means that it must be possible to be fixed. And because it affects newcomers who we need, it should be fixed rather earlier than later.

It actually has been happening for a rather long time but the frequency has certainly increased as Commons has grown. To actually fix it, the issue is a Commons issue and not a PRP one and thus should get retagged/reassigned there. There are numerous tickets about mentioning things like PDF/DjVu having 0x0 size that are essentially the same thing. This seems to just be the latest duplicate of such.

MatthewVernon subscribed.

This isn't a swift issue; I think it's in fact related to the thumbnailing process, so I'll shuffle it to the thumbor queue.

Not a Thumbor issue as far as I'm aware, but a problem with how MediaWiki/PdfHandler generates and stores the file metadata. The thumbnails themselves are usually fine if you manually construct the URL.

https://commons.wikimedia.org/wiki/File:The_English_Reports_v2_1901.pdf was uploaded at 16:21 2 Jan 2025. It currently (17:20) shows 0 × 0 pixels, the file page displays a broken PDF icon, and the row in the image table (via the WMCS replicas) is

img_nameimg_sizeimg_widthimg_heightimg_metadataimg_bitsimg_media_typeimg_major_mimeimg_minor_mimeimg_description_idimg_actorimg_timestampimg_sha1
The_English_Reports_v2_1901.pdf2624162352101616250OFFICEapplicationpdf365472352.033302120250102162124cry12bq1sp6zfrcgs2cwgyp7qh74vtp

Manually constructing https://upload.wikimedia.org/wikipedia/commons/thumb/5/5d/The_English_Reports_v2_1901.pdf/page1-250px-The_English_Reports_v2_1901.pdf.jpg results in the expected thumbnail. Testing locally, both commands in https://phabricator.wikimedia.org/diffusion/EPHD/browse/master/scripts/retrieveMetaData.sh successfully retrieve metadata from the file.

$ pdfinfo -enc 'UTF-8' -meta "The_English_Reports_v2_1901.pdf"                           
<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?>
<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="Adobe XMP Core 9.1-c001 79.675d0f7, 2023/06/11-19:21:16        ">
   <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
      <rdf:Description rdf:about=""
            xmlns:dc="http://purl.org/dc/elements/1.1/"
            xmlns:pdf="http://ns.adobe.com/pdf/1.3/"
            xmlns:xmp="http://ns.adobe.com/xap/1.0/"
            xmlns:xmpMM="http://ns.adobe.com/xap/1.0/mm/">
         <dc:format>application/pdf</dc:format>
         <dc:creator>
            <rdf:Seq>
               <rdf:li/>
            </rdf:Seq>
         </dc:creator>
         <dc:title>
            <rdf:Alt>
               <rdf:li xml:lang="x-default">The English Reports</rdf:li>
            </rdf:Alt>
         </dc:title>
         <pdf:Producer>Adobe Acrobat Pro (64-bit) 24 Paper Capture Plug-in</pdf:Producer>
         <xmp:ModifyDate>2025-01-02T09:43:58-05:00</xmp:ModifyDate>
         <xmp:CreateDate>2025-01-02T09:43:58-05:00</xmp:CreateDate>
         <xmp:MetadataDate>2025-01-02T09:43:58-05:00</xmp:MetadataDate>
         <xmpMM:DocumentID>uuid:a1d2bf19-20bc-4708-a8a1-c339b9693b15</xmpMM:DocumentID>
         <xmpMM:InstanceID>uuid:fc170d18-4225-41b7-9b50-00e6f971658c</xmpMM:InstanceID>
      </rdf:Description>
   </rdf:RDF>
</x:xmpmeta>
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                           
<?xpacket end="w"?>
$ pdfinfo -enc 'UTF-8' -l 9999999 "The_English_Reports_v2_1901.pdf" | head -n 25         
Title:           The English Reports
Author:          
Producer:        Adobe Acrobat Pro (64-bit) 24 Paper Capture Plug-in
CreationDate:    Thu Jan  2 09:43:58 2025 EST
ModDate:         Thu Jan  2 09:43:58 2025 EST
Custom Metadata: no
Metadata Stream: yes
Tagged:          no
UserProperties:  no
Suspects:        no
Form:            AcroForm
JavaScript:      no
Pages:           1349
Encrypted:       no
Page    1 size:  488.16 x 780 pts
Page    1 rot:   0
Page    2 size:  435.36 x 717.12 pts
Page    2 rot:   0
Page    3 size:  452.4 x 721.2 pts
Page    3 rot:   0
Page    4 size:  440.4 x 713.76 pts
Page    4 rot:   0
Page    5 size:  442.8 x 718.32 pts
Page    5 rot:   0
Page    6 size:  420.48 x 713.28 pts

I wonder if there's a problem getting the file into Shellbox?

I don't have enough information to know if this task is a duplicate of T299521: PDF file has 0x0 image size in Commons after uploading a new version while the page number is correct, but it sounds related at least.

The problem is getting more and more urgent. While experienced editors know how to workaround it by purging caches of various affected pages, newbies get confused and frustrated, sometimes uploading duplicate files suffering the same problem, etc. Wikisource cannot afford losing new users, I do ask for solving this bug urgently.

@Aklapper : I increased the priority because it is of high priority and because it urgently needs WMF programmers to start working on fixing this, half a year after the task was created.

@Jan.Kamenicek Ah. Increasing priority does not magically make bored developers appear (plus it does not need "WMF" programmers). Please see https://www.mediawiki.org/wiki/Bug_management/Development_prioritization - thanks.

I proposed the following in a parallel request T390603. But it was ignored and the request was closed.

  • Is it possible to categorize PDFs with such an error that persists for more than 1 day or more than a week?
  • And is it possible to configure the bot so that it tries to process such PDF again? And also purge the Index page in Wikisource that uses this file, since an error remains there after purge the file page on Commons.

A year has passed. The problem remains: panic among new users who want to work on Wikisource but are unable to do so because of this error, which undermines their trust in the project.

I proposed the following in a parallel request T390603. But it was ignored and the request was closed.

  • Is it possible to categorize PDFs with such an error that persists for more than 1 day or more than a week?
  • And is it possible to configure the bot so that it tries to process such PDF again? And also purge the Index page in Wikisource that uses this file, since an error remains there after purge the file page on Commons.

A year has passed. The problem remains: panic among new users who want to work on Wikisource but are unable to do so because of this error, which undermines their trust in the project.

Since T425717, new Index pages with this problem will be tracked in MediaWiki:Proofreadpage invalid interval category. I tried to run a bot to do the purging and null edits in those indexes but for some reason I couldn't replicate the same effect as doing it "by hand",

Thank you, this is very useful! But I can't find this category:

A PDF might work fine on Commons but have a bug in Wikisource, as I've reported in other bug reports. But, I also can't find this category on ru.Wikisource.