Page MenuHomePhabricator
Create Task
Maniphest T298417

Undeleted djvu files show incorrect metadata: 0x0 size, no page number info
Open, Needs TriagePublicBUG REPORT
Actions

Description

After undeletion of djvu files deleted in 2015 they become useless: 0x0 size, no page number information/
eg.
https://commons.wikimedia.org/wiki/File:Skibinski_pamietnik0001.djvu
https://commons.wikimedia.org/wiki/File:Le_Goffic_-_Le_Crucifi%C3%A9_de_Kerali%C3%A8s.djvu

Details

SubjectRepoBranchLines +/-
file: Move json metadata storage out of LocalFilemediawiki/coremaster+173 -98
maintenance: Add support for oldimage table metadata refreshmediawiki/corewmf/1.38.0-wmf.16+26 -16
maintenance: Add support for oldimage table metadata refreshmediawiki/corewmf/1.38.0-wmf.13+26 -16
maintenance: Add support for oldimage table metadata refreshmediawiki/coremaster+26 -16
Customize query in gerrit

Related Objects

Event Timeline

Ankry created this task.Jan 1 2022, 1:22 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 1 2022, 1:22 AM
Ankry added a comment.Jan 1 2022, 1:45 AM

One more example:
https://commons.wikimedia.org/wiki/File:PL_Kieszonkowy_s%C5%82ownik_j%C4%99zyk%C3%B3w_%C5%82aci%C5%84skiego_i_polskiego._Cz._1,_S%C5%82ownik_%C5%82aci%C5%84sko-polski.djvu
(this one has been deleted in 2018)

Ankry added a comment.Jan 1 2022, 1:54 AM

Same problem with unhiding a file version that was hidden in August 2021:
https://commons.wikimedia.org/wiki/File:PL_Miriam_-_U_poet%C3%B3w.djvu

Anoop changed the subtype of this task from "Task" to "Bug Report".Jan 1 2022, 2:13 AM
Anoop added a project: MediaWiki-DjVu.
Peachey88 added a project: MediaWiki-File-management.Jan 1 2022, 2:27 AM
Ankry updated the task description. (Show Details)Jan 1 2022, 2:42 AM
Maintenance_bot added a project: Commons.Jan 1 2022, 2:46 AM
Yann mentioned this in T298423: DjVu thumbnails not displayed on Wikisource.Jan 1 2022, 12:29 PM
Peachey88 merged a task: T298423: DjVu thumbnails not displayed on Wikisource.Jan 1 2022, 2:41 PM
Peachey88 added a subscriber: Yann.
Umherirrender subscribed.Jan 1 2022, 2:48 PM
In T298417#7592371, @Ankry wrote:

Same problem with unhiding a file version that was hidden in August 2021:
https://commons.wikimedia.org/wiki/File:PL_Miriam_-_U_poet%C3%B3w.djvu

This is affected by the missing split of big metadata for the filehistory (oldimage table).

The original task description is affected by the missing split of big metadata for the file archive (filearchive table)

The work was done as part of T275268, but the refreshImageMetadata.php only running for the current file versions

Ankry added a comment.Jan 1 2022, 7:24 PM
In T298417#7592558, @Umherirrender wrote:
In T298417#7592371, @Ankry wrote:

Same problem with unhiding a file version that was hidden in August 2021:
https://commons.wikimedia.org/wiki/File:PL_Miriam_-_U_poet%C3%B3w.djvu

This is affected by the missing split of big metadata for the filehistory (oldimage table).

The original task description is affected by the missing split of big metadata for the file archive (filearchive table)

The work was done as part of T275268, but the refreshImageMetadata.php only running for the current file versions

This is a big problem for all Wikisources as we need to restore a bunch of deleted files or revisions on the Public Domain Day each year, and missing metadata (mainly the numbed of pages) would break hundreds ot thousands existing pages that rely on it.
How can we ensure that an undeleted file will have correct metadata? It seems we need to delay the Public Domain Day until this is fixed.

Ankry added a comment.EditedJan 1 2022, 8:53 PM

An attempt to reupload (even the failed one) restores the metadata in Commons:
https://commons.wikimedia.org/wiki/File:Skibinski_pamietnik0001.djvu
But this does not restore the metadata in Wikisource:
https://pl.wikisource.org/wiki/Plik:Skibinski_pamietnik0001.djvu
so cannot be used as an ugly workaround

Umherirrender added subscribers: Ladsgroup, Krinkle, tstarling.Jan 2 2022, 7:18 PM
In T298417#7592643, @Ankry wrote:

An attempt to reupload (even the failed one) restores the metadata in Commons:
https://commons.wikimedia.org/wiki/File:Skibinski_pamietnik0001.djvu
But this does not restore the metadata in Wikisource:
https://pl.wikisource.org/wiki/Plik:Skibinski_pamietnik0001.djvu
so cannot be used as an ugly workaround

A upload stores the metadata in the new way and it works after that, but that sound not like a good idea.

The main task was T192866, I add some reviewer from there to this task.

Matlin subscribed.Jan 2 2022, 7:26 PM
Umherirrender added a comment.Jan 2 2022, 7:30 PM

The api output of the metadata still contains xml key, but https://gerrit.wikimedia.org/r/c/mediawiki/core/+/740320/ removes the interpretation of that key

https://commons.wikimedia.org/w/api.php?action=query&prop=imageinfo&iiprop=metadata&titles=File:Skibinski_pamietnik0001.djvu

"pages": {
    "42588070": {
        "pageid": 42588070,
        "ns": 6,
        "title": "File:Skibinski pamietnik0001.djvu",
        "imagerepository": "local",
        "imageinfo": [
            {
                "metadata": [
                    {
                        "name": "xml",
                        "value": "<?xml version=\"1.0\" ?>\n<!DOCTYPE DjVuXML PUBLIC \"-//W3C//DTD DjVuXML 1.1//EN\" \"pubtext/DjVuXML-s.dtd\">\n<mw-djvu><DjVuXML>\n<HEAD></HEAD>\n<BODY><OBJECT
Ladsgroup added a comment.Jan 3 2022, 5:06 AM

I will look into this. I suspected this would become an issue but I found only 100 deleted djvu files in the db back then. Maybe I misread something. Anyway. I will take a look.

Ladsgroup added a comment.Jan 3 2022, 8:14 AM

So running the refresh metadata script on them fixes the issue (Skibinski_pamietnik0001.djvu is fixed now both on commons and plwikisource). I don't know if it's possible to do this on all of deleted files. I will take a look.

Xover subscribed.Jan 3 2022, 8:30 AM
In T298417#7593376, @Ladsgroup wrote:

… I found only 100 deleted djvu files in the db …

I don't have actual data, but my gut feeling is that this number must be off by at least an order of magnitude. Between files that are deleted for the usual reasons (mostly copyright), there must be several thousand files that have been transwikied to Commons from a Wikisource; and all of these are candidates for being restored for various reasons. A prime example being that Commons' copyright policy requires works to be in the public domain in both its country of origin and in the US, while the Wikisourcen may only consider US status. Undeleting these locally when they get deleted on Commons isn't uncommon. Undeleting files on Commons when the relevant copyright expires can't be all that rare an event either, but I don't follow the relevant fora so I could be way off on that.

Ladsgroup added a comment.Jan 3 2022, 8:38 AM
wikiadmin@10.64.0.220(commonswiki)> select count(*) from filearchive where fa_media_type = 'OFFICE' limit 5;
+----------+
| count(*) |
+----------+
|    92509 |
+----------+
1 row in set (37.633 sec)

wikiadmin@10.64.0.220(commonswiki)> select count(*) from filearchive where fa_media_type = 'OFFICE' and fa_major_mime = 'image' limit 5;
+----------+
| count(*) |
+----------+
|      736 |
+----------+
1 row in set (3.213 sec)

It doesn't matter. I will write a patch to run this on both filearchive and oldimage table.

gerritbot added a comment.Jan 3 2022, 10:32 AM

Change 751079 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] maintenance: Add support for oldimage table metadata refresh

https://gerrit.wikimedia.org/r/751079

gerritbot added a project: Patch-For-Review.Jan 3 2022, 10:32 AM
Ladsgroup added a comment.Jan 3 2022, 10:38 AM

This patch and running it would fix some of the issues but not all. For "filearchive" table fixes I need to find a separate way. In the meantime, any file that has been undeleted and now has issue, just list it here and I run the script on it.

Ankry added a comment.Jan 3 2022, 4:52 PM
In T298417#7593773, @Ladsgroup wrote:

This patch and running it would fix some of the issues but not all. For "filearchive" table fixes I need to find a separate way. In the meantime, any file that has been undeleted and now has issue, just list it here and I run the script on it.

There is still problem with the 2nd version of
https://pl.wikisource.org/wiki/Plik:PL_Miriam_-_U_poet%C3%B3w.djvu
I would like to make it the top version but I am afraid to break things that depend on its metadata.

Ladsgroup added a comment.Jan 4 2022, 2:31 AM

Don't worry on that, just make it top version and then let me know to run the script (until I make the patch merged and deployed)

gerritbot added a comment.Jan 4 2022, 10:01 PM

Change 751079 merged by jenkins-bot:

[mediawiki/core@master] maintenance: Add support for oldimage table metadata refresh

https://gerrit.wikimedia.org/r/751079

Maintenance_bot removed a project: Patch-For-Review.Jan 4 2022, 10:10 PM
ReleaseTaggerBot added a project: MW-1.38-notes (1.38.0-wmf.17; 2022-01-10).Jan 4 2022, 11:00 PM
gerritbot added a comment.Jan 5 2022, 1:34 AM

Change 751526 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@wmf/1.38.0-wmf.16] maintenance: Add support for oldimage table metadata refresh

https://gerrit.wikimedia.org/r/751526

gerritbot added a project: Patch-For-Review.Jan 5 2022, 1:34 AM

Change 751527 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@wmf/1.38.0-wmf.13] maintenance: Add support for oldimage table metadata refresh

https://gerrit.wikimedia.org/r/751527

gerritbot added a comment.Jan 5 2022, 2:05 AM

Change 751527 merged by jenkins-bot:

[mediawiki/core@wmf/1.38.0-wmf.13] maintenance: Add support for oldimage table metadata refresh

https://gerrit.wikimedia.org/r/751527

Stashbot added a comment.Jan 5 2022, 2:09 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-05T02:09:08Z] <ladsgroup@deploy1002> Synchronized php-1.38.0-wmf.13/maintenance/refreshImageMetadata.php: Backport: [[gerrit:751527|maintenance: Add support for oldimage table metadata refresh (T298417)]] (duration: 01m 08s)

gerritbot added a comment.Jan 5 2022, 2:09 AM

Change 751526 merged by jenkins-bot:

[mediawiki/core@wmf/1.38.0-wmf.16] maintenance: Add support for oldimage table metadata refresh

https://gerrit.wikimedia.org/r/751526

Maintenance_bot removed a project: Patch-For-Review.Jan 5 2022, 2:10 AM
Stashbot added a comment.Jan 5 2022, 2:11 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-05T02:11:19Z] <ladsgroup@deploy1002> Synchronized php-1.38.0-wmf.16/maintenance/refreshImageMetadata.php: Backport: [[gerrit:751526|maintenance: Add support for oldimage table metadata refresh (T298417)]] (duration: 01m 07s)

Stashbot added a comment.Jan 5 2022, 2:13 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-05T02:13:23Z] <Amir1> running foreachwikiindblist all maintenance/refreshImageMetadata.php --force --verbose --mediatype=OFFICE --oldimage (T298417)

ReleaseTaggerBot edited projects, added MW-1.38-notes (1.38.0-wmf.13; 2021-12-13); removed MW-1.38-notes (1.38.0-wmf.17; 2022-01-10).Jan 5 2022, 3:00 AM
Ladsgroup added a comment.Jan 6 2022, 4:28 AM

So the oldimage clean up has been done and you should now be able to see non-current metadata. Next problem is for deleted files, It's already in a much smaller scale.

Ankry added a comment.Jan 9 2022, 8:53 PM
In T298417#7594875, @Ladsgroup wrote:

Don't worry on that, just make it top version and then let me know to run the script (until I make the patch merged and deployed)

There is problem with this file, after unhidint/restoring its oldest version:
https://commons.wikimedia.org/wiki/File:PL_Nowele_obce_(antologia).djvu

Ladsgroup added a comment.Jan 10 2022, 8:20 AM

Done.

Ankry mentioned this in T299521: PDF file has 0x0 image size in Commons after uploading a new version while the page number is correct.Jan 19 2022, 1:54 PM
Ankry added a comment.Jan 25 2022, 11:22 PM
In T298417#7593773, @Ladsgroup wrote:

This patch and running it would fix some of the issues but not all. For "filearchive" table fixes I need to find a separate way. In the meantime, any file that has been undeleted and now has issue, just list it here and I run the script on it.

It seems the just restored
https://commons.wikimedia.org/wiki/File:Fairy_Tales_for_Worker%27s_Children.djvu
also hits this problem.

Ladsgroup added a comment.Jan 26 2022, 1:01 PM

Refreshed. There are around 700 deleted djvu files atm. I will check how it can be fixed.

Bugreporter mentioned this in T301291: PDF and Djvu files on Commons failed to be processed (no thumbnails, zero pages) but otherwise valid.Feb 9 2022, 1:10 AM
Krinkle unsubscribed.Feb 28 2022, 11:01 PM
Umherirrender unsubscribed.Mar 15 2022, 10:21 PM
gerritbot added a comment.Apr 1 2022, 3:17 PM

Change 776214 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] file: Move json metadata storage out of LocalFile

https://gerrit.wikimedia.org/r/776214

gerritbot added a project: Patch-For-Review.Apr 1 2022, 3:17 PM
gerritbot added a comment.Apr 7 2022, 1:28 AM

Change 776214 merged by jenkins-bot:

[mediawiki/core@master] file: Move json metadata storage out of LocalFile

https://gerrit.wikimedia.org/r/776214

Maintenance_bot removed a project: Patch-For-Review.Apr 7 2022, 1:30 AM
Peachey88 mentioned this in T331194: Problem with DjVuHandler?.Mar 4 2023, 10:35 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL