Page MenuHomePhabricator
Create Task
Maniphest T351628

Run the maintenance script scanning images in mediamoderation_scan on a test wiki
Closed, ResolvedPublic2 Estimated Story Points
Actions

Description

Once created, the maintenance script in T351399: Create a maintenance script to automatically scan files listed in mediamoderation_scan should be run daily first on a nominated wiki (likely testwiki). This is to validate the script works for T351544: [Sub EPIC] Start test wiki scan.

Acceptance criteria

Related Objects
Search...

Event Timeline

• Dreamy_Jazz created this task.Nov 20 2023, 10:58 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 20 2023, 10:58 AM
• Dreamy_Jazz added a parent task: T351544: [Sub EPIC] Start test wiki scan.Nov 20 2023, 11:00 AM
• Dreamy_Jazz added a subtask: T351399: Create a maintenance script to automatically scan files listed in mediamoderation_scan.
• Dreamy_Jazz added a subtask: T353441: Collect statsd metrics of requests per day to PhotoDNA and response times from the PhotoDNA.Dec 14 2023, 12:11 PM
• Dreamy_Jazz removed a subtask: T353441: Collect statsd metrics of requests per day to PhotoDNA and response times from the PhotoDNA.Dec 14 2023, 12:31 PM
• Dreamy_Jazz assigned this task to Tchanders.Dec 19 2023, 6:51 PM
• Dreamy_Jazz added a comment.Dec 19 2023, 6:54 PM
This comment was removed by • Dreamy_Jazz.
Tchanders added a comment.Dec 19 2023, 6:57 PM

We ran this on testwiki today. Observations:

  • Script ran without errors, took about 2 hours for tens of thousands of images
  • There were some successful calls to the PhotoDNA API ("No match" printed below)
  • There were failures for most of the images, with different errors

Sample output:

SHA-1 tuhoup8ncvtu8un6frtuwlr5zqzqqhb: Scan failed.

SHA-1 tul6ljtuslab5hhnmtf8nogoo9lp6tw

...Could not transform file Hhh.jpeg: Error creating thumbnail: An unknown error occurred.

SHA-1 tul6ljtuslab5hhnmtf8nogoo9lp6tw: No match.
SHA-1 tulcj38atgajh5obu2wwtpb40yxdw9k

...Could not transform file Silva_of_North_America,_The._Volume_XI,_Coniferæ_(pinus)_-_DPLA_-_3f6e22ceec126c964f4707deece445ac_(page_13).jpg: ArchivedFile instances cannot be processed yet.

SHA-1 tulcj38atgajh5obu2wwtpb40yxdw9k: Scan failed.

SHA-1 tultbn4e1bju9fz94kcrl27f5gxsm01: No match.
SHA-1 tunw9bm8gk70wwdjuv8x4zr7n9woyvl

...Could not transform file The_Town_Crier,_v.12,_no.41,_Oct._13,_1917_-_DPLA_-_d69fb7db787c39d76e58a4e4454c8fcd_(page_13).jpg: ArchivedFile instances cannot be processed yet.

SHA-1 tunw9bm8gk70wwdjuv8x4zr7n9woyvl: Scan failed.

SHA-1 tuokfhthmhaprvcv0nksoyj3i452j5o

...Could not transform file 1213.jpg: ArchivedFile instances cannot be processed yet.

SHA-1 tuokfhthmhaprvcv0nksoyj3i452j5o: Scan failed.

SHA-1 tuqwwiql91wowkznx1y6wunh48l9u7s

...Could not transform file Wachusett_Reservoir,_view_of_reservoir_from_Sawyer's_Mills,_looking_east,_Boylston,_Mass.,_Apr._9,_1908_-_DPLA_-_befa3ffd3592977341054853c6c389dd.jpg: ArchivedFile instances cannot be processed yet.

SHA-1 tuqwwiql91wowkznx1y6wunh48l9u7s: Scan failed.

SHA-1 tv0irne7l2qiax4tiigvvthffk9x1mw

...Could not transform file Seepage_and_return_waters_-_DPLA_-_282e74c1dfd04ae1d2cf3b74515c6367_(page_21).jpg: ArchivedFile instances cannot be processed yet.

SHA-1 tv0irne7l2qiax4tiigvvthffk9x1mw: Scan failed.

SHA-1 tv0qp81ibzekubvbz5175oajcoxbj9x

...Could not transform file Pacific_Builder_and_Engineer,_v._8,_no._33,_Aug._7,_1909_-_DPLA_-_6f745d578d37c1699a12b506a12ea7f4_(page_25).jpg: ArchivedFile instances cannot be processed yet.

SHA-1 tv0qp81ibzekubvbz5175oajcoxbj9x: Scan failed.

SHA-1 tv2hxda3j19nx1bn7yl4zh9k3kja272

...Could not transform file Through_the_leaves,_1926_-_DPLA_-_e7dae0ac19473d8460df97cf7359562f_(page_437).jpg: ArchivedFile instances cannot be processed yet.

SHA-1 tv2hxda3j19nx1bn7yl4zh9k3kja272: Scan failed.

SHA-1 tv47ovq5bqd1jbipsddmvxir9ps0516

...Could not transform file Seattle_Mail_and_Herald,_v._9,_no._5,_Dec._23,_1905_-_DPLA_-_183d4ac73582c57cdc7147a2fea17c24_(page_5).jpg: ArchivedFile instances cannot be processed yet.

SHA-1 tv47ovq5bqd1jbipsddmvxir9ps0516: Scan failed.

SHA-1 tv4utmc0nmgsnjmh55i8yfp2dfvl2mx

...Could not transform file Through_the_leaves,_1926_-_DPLA_-_e7dae0ac19473d8460df97cf7359562f_(page_482).jpg: ArchivedFile instances cannot be processed yet.

SHA-1 tv4utmc0nmgsnjmh55i8yfp2dfvl2mx: Scan failed.

SHA-1 tv66vcq2h0zbdr5httokefbkut6xoeg

...Could not transform file Seepage_or_return_waters_from_irrigation_-_DPLA_-_36a56740a529045f6a038f0e63d92c3f_(page_11).jpg: ArchivedFile instances cannot be processed yet.

SHA-1 tv66vcq2h0zbdr5httokefbkut6xoeg: Scan failed.

SHA-1 tvawzi05vccatj6igwaojbke4tbd0jp

...Could not transform file Pacific_Builder_and_Engineer,_v._6,_no._26,_Jun._27,_1908_-_DPLA_-_79922daab0905ccb1cdec4e72c14631d_(page_18).jpg: ArchivedFile instances cannot be processed yet.

SHA-1 tvawzi05vccatj6igwaojbke4tbd0jp: Scan failed.

SHA-1 tvedgatrnmv394spmmdxtxsz1iz4p5a

...Could not transform file Moreno_picante.jpeg: ArchivedFile instances cannot be processed yet.

...Could not transform file Contreiras.jpeg: ArchivedFile instances cannot be processed yet.

SHA-1 tvedgatrnmv394spmmdxtxsz1iz4p5a: Scan failed.

SHA-1 tvfiihgnil49gnzheoaoq9wwbd8jkbt

...Could not transform file Through_the_leaves,_1926_-_DPLA_-_e7dae0ac19473d8460df97cf7359562f_(page_475).jpg: ArchivedFile instances cannot be processed yet.

SHA-1 tvfiihgnil49gnzheoaoq9wwbd8jkbt: Scan failed.

SHA-1 tvfj81svmjyb7rqvh0dscvx4j4w84lr

...Could not transform file TheBiographersTale.jpg: ArchivedFile instances cannot be processed yet.

SHA-1 tvfj81svmjyb7rqvh0dscvx4j4w84lr: Scan failed.

SHA-1 tvidppl3j4oe3b4m934j2g2y7qv1d67

...Could not transform file Silva_of_North_America,_The._Volume_VII,_Lauraceæ_-_juglandaceæ_-_DPLA_-_1a8247cc44d2801629a5ce30cd495c48_(page_226).jpg: ArchivedFile instances cannot be processed yet.

SHA-1 tvidppl3j4oe3b4m934j2g2y7qv1d67: Scan failed.

SHA-1 tvk09b899v2enavwjv5jw5hdn3o11k2

...Could not transform file On_the_measurement_and_division_of_water_-_DPLA_-_7ed979f6ca86142436705b6b2ccdf3d3_(page_25).jpg: ArchivedFile instances cannot be processed yet.

SHA-1 tvk09b899v2enavwjv5jw5hdn3o11k2: Scan failed.

SHA-1 tvlhowqudxwrk2zpdh7pvbgsr5ksc67

...Could not transform file Wachusett_Department,_Clinton_Sewerage_filter-beds,_showing_inlet_and_method_of_distribution,_Clinton,_Mass.,_Sep._24,_1909_-_DPLA_-_737f581df50c31b66b86c71bb02887cd.jpg: ArchivedFile instances cannot be processed yet.

SHA-1 tvlhowqudxwrk2zpdh7pvbgsr5ksc67: Scan failed.

SHA-1 tvofotx8nr8qinpwa7qefn8pw8rradz

...Could not transform file The_Daily_Express._(San_Antonio,_Tex.),_Vol._43,_No._105,_Ed._1_Tuesday,_April_14,_1908_-_DPLA_-_4c7ce210b65ceaa32d7d60f56ecfecc4_(page_11).jpg: ArchivedFile instances cannot be processed yet.

SHA-1 tvofotx8nr8qinpwa7qefn8pw8rradz: Scan failed.

SHA-1 tvpi2d0xer6gpmzf0y87f1r1tqu7ry5

...Could not transform file D._M._Bradley_to_W._A._Harrington,_4_August_1857_-_DPLA_-_aac2e4abb9d48b05f78dc6d067abb12f_(page_1).jpg: ArchivedFile instances cannot be processed yet.

SHA-1 tvpi2d0xer6gpmzf0y87f1r1tqu7ry5: Scan failed.

SHA-1 tvpkk8oz0drsn3i6xby6g1p8kbsc8k0

...Could not transform file Pacific_Builder_and_Engineer,_v._9,_no._23,_Jun._4,_1910_-_DPLA_-_730e2c0882f6cb4733ddc74e282991b2_(page_34).jpg: ArchivedFile instances cannot be processed yet.

SHA-1 tvpkk8oz0drsn3i6xby6g1p8kbsc8k0: Scan failed.

SHA-1 tvqjmtps5g14crur1589q5nxtndg5sq

...Could not transform file Pacific_Builder_and_Engineer,_v._11,_no._18,_May_6,_1911_-_DPLA_-_d2ca917f7c169df9fd46512408f318a6_(page_2).jpg: ArchivedFile instances cannot be processed yet.

SHA-1 tvqjmtps5g14crur1589q5nxtndg5sq: Scan failed.

SHA-1 tvt9m06enc1t079san27kwu17nfk0ly

...Could not transform file Spme_tripod.jpeg: ArchivedFile instances cannot be processed yet.

SHA-1 tvt9m06enc1t079san27kwu17nfk0ly: Scan failed.

SHA-1 tvttf282fe8w2ew87kxtc9za4kbr6u0

...Could not transform file Silva_of_North_America,_The._Volume_IV,_Rosaceæ_-_saxifragaceæ_-_DPLA_-_058565fcdae9ec2e2c9812e6a56602a9_(page_119).jpg: ArchivedFile instances cannot be processed yet.

SHA-1 tvttf282fe8w2ew87kxtc9za4kbr6u0: Scan failed.

SHA-1 tvuyhxxjuqxuyqg9oly00n26dgitcs6

...Could not transform file Dc_three_1.svg: ArchivedFile instances cannot be processed yet.

...Could not transform file File-Dc_three_1.svg: ArchivedFile instances cannot be processed yet.

SHA-1 tvuyhxxjuqxuyqg9oly00n26dgitcs6: Scan failed.

SHA-1 tvv0p0zxuzxaxyyjf1d451f31zzhnra: No match.
SHA-1 tvv98omdw18mkqtvy16qf23tx3frt3u

...Could not transform file CIMG5101.jpg: Error creating thumbnail: An unknown error occurred.

SHA-1 tvv98omdw18mkqtvy16qf23tx3frt3u: Scan failed.

SHA-1 tvz9yep5co3so2tnznd53a84doudefo

...Could not transform file A._S._Bartlett_to_Curtis_c-o_Kimball_Jewett_and_Co,_30_August_1847_-_DPLA_-_51b317a7c8566131199e144268882930_(page_3).jpg: ArchivedFile instances cannot be processed yet.

SHA-1 tvz9yep5co3so2tnznd53a84doudefo: Scan failed.

SHA-1 tw0heomwah1ku01slcwipj7wvqp441v

...Could not transform file Seattle_Mail_and_Herald,_v._4,_no._41,_Aug._24,_1901_-_DPLA_-_bed7c8d6ac8f4bbd8386f273b4b184e9_(page_1).jpg: ArchivedFile instances cannot be processed yet.

SHA-1 tw0heomwah1ku01slcwipj7wvqp441v: Scan failed.

SHA-1 tw3i65hmhxq9clxy0kzlb5wu5b8drg1

...Could not get file contents for 0.868354067718666.png

SHA-1 tw3i65hmhxq9clxy0kzlb5wu5b8drg1: Scan failed.

SHA-1 tw4g73knirmtwyt5mli64v5ybkyvmps

...Could not transform file Mturk_test_image_9.jpg: ArchivedFile instances cannot be processed yet.

SHA-1 tw4g73knirmtwyt5mli64v5ybkyvmps: Scan failed.

SHA-1 tw5jboqzkoa59alzc5o5lxdozk2agik

...Could not transform file The_Town_Crier,_v.9,_no.37,_Sep._12,_1914_-_DPLA_-_84f09212db8a93d3ba69d66b7f92ae48_(page_4).jpg: ArchivedFile instances cannot be processed yet.

SHA-1 tw5jboqzkoa59alzc5o5lxdozk2agik: Scan failed.

SHA-1 twajz1qzkn12n36dzquf31iwngr56e7

...Could not transform file Novems_test_file.png: Error creating thumbnail: An unknown error occurred.

SHA-1 twajz1qzkn12n36dzquf31iwngr56e7: Scan failed.

SHA-1 twfbsk825sje18d6ju9zq3brl6mmrwt

...Could not transform file Making_better_decisions-_2003_Colorado_alfalfa_variety_performance_trials_-_DPLA_-_a0cda018c27c80e1ee5eb729b2a3176b_(page_9).jpg: ArchivedFile instances cannot be processed yet.

SHA-1 twfbsk825sje18d6ju9zq3brl6mmrwt: Scan failed.

SHA-1 twfizyywxkcfpf1j7dj0w1dwg8b4uvo

...Could not transform file St-_Paul's_Cemetery_2012-09-21_14-15-56.jpg: Error creating thumbnail: An unknown error occurred.

SHA-1 twfizyywxkcfpf1j7dj0w1dwg8b4uvo: Scan failed.

SHA-1 twgxh7yid3spyr3m021213qj6sw4sbm

...Could not transform file Silva_of_North_America,_The._Volume_XI,_Coniferæ_(pinus)_-_DPLA_-_3f6e22ceec126c964f4707deece445ac_(page_173).jpg: ArchivedFile instances cannot be processed yet.

SHA-1 twgxh7yid3spyr3m021213qj6sw4sbm: Scan failed.

SHA-1 twifg77fvqqkmjwruecq407wpgpt4af

...Could not transform file Testfile_1362033755.jpg: Error creating thumbnail: An unknown error occurred.

SHA-1 twifg77fvqqkmjwruecq407wpgpt4af: Scan failed.

SHA-1 twinj6ussv0haah4s4rv1pcctciuf0l

...Could not transform file History_of_agriculture_in_Colorado_-_DPLA_-_2b43f71755ee0ed5205c41c4bb669456_(page_219).jpg: ArchivedFile instances cannot be processed yet.

SHA-1 twinj6ussv0haah4s4rv1pcctciuf0l: Scan failed.
• Dreamy_Jazz added a project: Trust and Safety Product Sprint (Sprint Lute (11th Dec. - 22nd Dec. 2023)).Jan 4 2024, 8:55 AM
• Dreamy_Jazz set the point value for this task to 2.
• Dreamy_Jazz moved this task from Priority Backlog to Needs review on the Trust and Safety Product Sprint (Sprint Lute (11th Dec. - 22nd Dec. 2023)) board.

This is completed, and now in "Needs review" so that we can inspect the logs and make fixes to the script before potentially re-running the script.

Tchanders moved this task from Needs review to Done on the Trust and Safety Product Sprint (Sprint Lute (11th Dec. - 22nd Dec. 2023)) board.Jan 5 2024, 3:12 PM
Tchanders added a subscriber: kostajh.

The script was run a couple more times yesterday, and some resulting bugs fixed.

@Dreamy_Jazz @kostajh Can this be closed?

kostajh closed this task as Resolved.Jan 5 2024, 3:34 PM
In T351628#9437732, @Tchanders wrote:

The script was run a couple more times yesterday, and some resulting bugs fixed.

@Dreamy_Jazz @kostajh Can this be closed?

Yes, I think so.

• Dreamy_Jazz closed subtask T351399: Create a maintenance script to automatically scan files listed in mediamoderation_scan as Resolved.Jan 8 2024, 4:14 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL