Page MenuHomePhabricator
Create Task
Maniphest T428162

Make declarations for Category:Items with VRTS permission confirmed
Closed, ResolvedPublic
Actions

Description

Category: Category:Items with VRTS permission confirmed
Estimated number of files: 1,837,414
Version of commonsdb-commons-supplier: 0.1.15, 0.1.17

Result

Files processed: ?
Declarations made: 223,588
Declarations failed: ? (? %)
Files already declared (skipped): ? (? %)

Total time: h:mm:ss (? days)
Average time per file: ? s

Prepare

Files processed: 4,328,754
Files already declared (skipped): 1,457,316 (33.7 %)

Total time: h:mm:ss
Average time per file: ? s

Related Objects

Event Timeline

Sebastian_Berlin-WMSE created this task.Jun 4 2026, 1:00 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 4 2026, 1:00 PM
Sebastian_Berlin-WMSE mentioned this in T427033: ☂ Make five million declarations.Jun 4 2026, 1:00 PM
Sebastian_Berlin-WMSE added a comment.Jun 9 2026, 7:48 AM

I accidentally stopped the prepare job as I was doing other things. It was restarted right away.

Sebastian_Berlin-WMSE added a project: User-Sebastian_Berlin-WMSE.Jun 15 2026, 7:32 AM

I started the declaration now. It looked like there were some issues, possibly due to memory limits. I raised the memory for the job to 3 GB.

The prepare job is still going, 1,403,450 files prepared at the moment. I don't think this should cause any issues with the declarations. It should run all the files that were prepared at the time of starting. When it's done it should be possible to run the rest in another job.

Sebastian_Berlin-WMSE moved this task from Backlog to This Week on the User-Sebastian_Berlin-WMSE board.Jun 15 2026, 7:32 AM
Sebastian_Berlin-WMSE moved this task from Backlog to In progress on the WMSE-CommonsDB-registry board.
Sebastian_Berlin-WMSE added a comment.Jul 1 2026, 4:18 PM

There has been a repeated error with file File:Bernhard Richter.jpg.

2026-07-01 09:07:46,559;__main__;INFO;Starting on file: 'File:Bernhard Richter.jpg'.
2026-07-01 09:07:46,647;urllib3.connectionpool;DEBUG;https://commons.wikimedia.org:443 "POST /w/api.php HTTP/1.1" 200 None
2026-07-01 09:07:46,713;urllib3.connectionpool;DEBUG;https://commons.wikimedia.org:443 "POST /w/api.php HTTP/1.1" 200 None
2026-07-01 09:07:46,715;file_fetcher;INFO;Downloading file: 'Bernhard_Richter.jpg'
2026-07-01 09:07:46,775;urllib3.connectionpool;DEBUG;https://commons.wikimedia.org:443 "POST /w/api.php HTTP/1.1" 200 1118
2026-07-01 09:07:46,823;urllib3.connectionpool;DEBUG;https://upload.wikimedia.org:443 "GET /wikipedia/commons/2/2b/Bernhard_Richter.jpg HTTP/1.1" 200 20450
2026-07-01 09:07:46,825;PIL.TiffImagePlugin;DEBUG;tag: Make (271) - type: string (2) Tag Location: 22 - Data Location: 122 - value: b'EASTMAN KODAK COMPANY\x00'
2026-07-01 09:07:46,825;PIL.TiffImagePlugin;DEBUG;tag: Model (272) - type: string (2) Tag Location: 34 - Data Location: 144 - value: b'KODAK DX3500 DIGITAL CAMERA\x00'
2026-07-01 09:07:46,825;PIL.TiffImagePlugin;DEBUG;tag: Orientation (274) - type: short (3) - value: b'\x01\x00'
2026-07-01 09:07:46,825;PIL.TiffImagePlugin;DEBUG;tag: XResolution (282) - type: rational (5) Tag Location: 58 - Data Location: 172 - value: b'\x00\x00\xc01\x00\x00\x00\x01'
2026-07-01 09:07:46,825;PIL.TiffImagePlugin;DEBUG;tag: YResolution (283) - type: rational (5) Tag Location: 70 - Data Location: 180 - value: b'\x00\x00\xc01\x00\x00\x00\x01'
2026-07-01 09:07:46,825;PIL.TiffImagePlugin;DEBUG;tag: ResolutionUnit (296) - type: short (3) - value: b'\x02\x00'
2026-07-01 09:07:46,825;PIL.TiffImagePlugin;DEBUG;tag: DateTime (306) - type: string (2) Tag Location: 94 - Data Location: 188 - value: b'2002:10:15 14:57:59\x00'
2026-07-01 09:07:46,825;PIL.TiffImagePlugin;DEBUG;tag: YCbCrPositioning (531) - type: short (3) - value: b'\x01\x00'
2026-07-01 09:07:46,826;PIL.TiffImagePlugin;DEBUG;tag: ExifIFD (34665) - type: long (4) - value: b'\xd0\x00\x00\x00'
2026-07-01 09:07:46,826;thumbnail_generator;INFO;Generating thumbnail from image: '/tmp/tmp7sal3u11/Bernhard_Richter.jpg'.
2026-07-01 09:07:46,826;PIL.TiffImagePlugin;DEBUG;tag: Make (271) - type: string (2) Tag Location: 22 - Data Location: 122 - value: b'EASTMAN KODAK COMPANY\x00'
2026-07-01 09:07:46,827;PIL.TiffImagePlugin;DEBUG;tag: Model (272) - type: string (2) Tag Location: 34 - Data Location: 144 - value: b'KODAK DX3500 DIGITAL CAMERA\x00'
2026-07-01 09:07:46,827;PIL.TiffImagePlugin;DEBUG;tag: Orientation (274) - type: short (3) - value: b'\x01\x00'
2026-07-01 09:07:46,827;PIL.TiffImagePlugin;DEBUG;tag: XResolution (282) - type: rational (5) Tag Location: 58 - Data Location: 172 - value: b'\x00\x00\xc01\x00\x00\x00\x01'
2026-07-01 09:07:46,827;PIL.TiffImagePlugin;DEBUG;tag: YResolution (283) - type: rational (5) Tag Location: 70 - Data Location: 180 - value: b'\x00\x00\xc01\x00\x00\x00\x01'
2026-07-01 09:07:46,827;PIL.TiffImagePlugin;DEBUG;tag: ResolutionUnit (296) - type: short (3) - value: b'\x02\x00'
2026-07-01 09:07:46,827;PIL.TiffImagePlugin;DEBUG;tag: DateTime (306) - type: string (2) Tag Location: 94 - Data Location: 188 - value: b'2002:10:15 14:57:59\x00'
2026-07-01 09:07:46,827;PIL.TiffImagePlugin;DEBUG;tag: YCbCrPositioning (531) - type: short (3) - value: b'\x01\x00'
2026-07-01 09:07:46,827;PIL.TiffImagePlugin;DEBUG;tag: ExifIFD (34665) - type: long (4) - value: b'\xd0\x00\x00\x00'
2026-07-01 09:07:46,828;PIL.TiffImagePlugin;DEBUG;tag: Make (271) - type: string (2) Tag Location: 22 - Data Location: 122 - value: b'EASTMAN KODAK COMPANY\x00'
2026-07-01 09:07:46,828;PIL.TiffImagePlugin;DEBUG;tag: Model (272) - type: string (2) Tag Location: 34 - Data Location: 144 - value: b'KODAK DX3500 DIGITAL CAMERA\x00'
2026-07-01 09:07:46,828;PIL.TiffImagePlugin;DEBUG;tag: Orientation (274) - type: short (3) - value: b'\x01\x00'
2026-07-01 09:07:46,828;PIL.TiffImagePlugin;DEBUG;tag: XResolution (282) - type: rational (5) Tag Location: 58 - Data Location: 172 - value: b'\x00\x00\xc01\x00\x00\x00\x01'
2026-07-01 09:07:46,828;PIL.TiffImagePlugin;DEBUG;tag: YResolution (283) - type: rational (5) Tag Location: 70 - Data Location: 180 - value: b'\x00\x00\xc01\x00\x00\x00\x01'
2026-07-01 09:07:46,828;PIL.TiffImagePlugin;DEBUG;tag: ResolutionUnit (296) - type: short (3) - value: b'\x02\x00'
2026-07-01 09:07:46,829;PIL.TiffImagePlugin;DEBUG;tag: DateTime (306) - type: string (2) Tag Location: 94 - Data Location: 188 - value: b'2002:10:15 14:57:59\x00'
2026-07-01 09:07:46,829;PIL.TiffImagePlugin;DEBUG;tag: YCbCrPositioning (531) - type: short (3) - value: b'\x01\x00'
2026-07-01 09:07:46,829;PIL.TiffImagePlugin;DEBUG;tag: ExifIFD (34665) - type: long (4) - value: b'\xd0\x00\x00\x00'
2026-07-01 09:07:46,835;thumbnail_generator;DEBUG;Thumbnail generation done.
2026-07-01 09:07:46,835;iscc_generator;INFO;Generating ISCC from image: '/tmp/tmp7sal3u11/Bernhard_Richter.jpg'.
2026-07-01 09:07:46,906;exiv2;ERROR;Directory Iop with 1024 entries considered invalid; not read.
2026-07-01 09:07:46,906;exiv2;ERROR;Directory Iop with 1024 entries considered invalid; not read.
2026-07-01 09:07:46,915;exiv2;WARNING;iconv: Invalid argument (errno = 22) inbytesleft = 1
terminate called after throwing an instance of 'Exiv2::Error'
  what():  Cannot convert text encoding from 'UCS-2LE' to 'UTF-8'
Aborted (core dumped)

This error is not recoverable and a new job needs to be started, but it'll never get past this file.

As a quick fix I added the string "FILE-CAUSES-ERROR-SEE-T428162" as the CID in the database for this file. This should mean that when it's encountered next time it will (incorrectly) be counted as already declared and skipped.

Sebastian_Berlin-WMSE changed the task status from Open to Stalled.Jul 6 2026, 7:40 AM
Sebastian_Berlin-WMSE moved this task from This Week to Waitin' on the User-Sebastian_Berlin-WMSE board.

This now runs out of memory. While it would be possible to continue a bit it's not certain that the whole batch would finish with the resource on Toolforge. There are several ways of working around this, for instance:

  1. Tagging the prepared declarations in the database in "chunks" (of say 100,000 each). Running each of them should be fine memory-wise. This would also have the advantage of not needing to re-check all the failed declarations it a batch needs to restart. However it would add a lot more jobs to the already long job list, which makes it more cumbersome.
  2. Implement chunking large declaration lists when declarations are made. This would require a bit of code restructuring. Things could get a bit tricky to make sure that things are logged correctly.

For now we've decided to put this batch on hold and run smaller ones that don't run into the same issues.

Sebastian_Berlin-WMSE moved this task from In progress to Blocked on the WMSE-CommonsDB-registry board.Jul 6 2026, 7:40 AM
Sebastian_Berlin-WMSE mentioned this in T431685: Handle files that fail repeatedly.Jul 9 2026, 10:41 AM
Sebastian_Berlin-WMSE updated the task description. (Show Details)Jul 27 2026, 8:31 AM
Sebastian_Berlin-WMSE mentioned this in T433222: Limit memory usage for large batches.Jul 27 2026, 8:43 AM
Sebastian_Berlin-WMSE moved this task from Waitin' to Done on the User-Sebastian_Berlin-WMSE board.Jul 27 2026, 8:50 AM

This one wasn't finished because of the issues mentioned above. It wasn't easy to get useful statistics from the logs due to repeated attempts.

Sebastian_Berlin-WMSE closed this task as Resolved.Jul 27 2026, 8:50 AM
Sebastian_Berlin-WMSE claimed this task.
Sebastian_Berlin-WMSE moved this task from Blocked to Done on the WMSE-CommonsDB-registry board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits