In T325811#8508994, @Aklapper wrote:

different at that time (move from Gerrit to Phab Differential instead of GitLab).

As I understood from the Avoiding arcanist paragraph, also this repo should be deleted from Phabricator Diffusion in terms of this process. Am I right?

In T325811#8508994, @Aklapper wrote:

Arcanist can be removed. That change was merged 6.5y ago and plans were different at that time (move from Gerrit to Phab Differential instead of GitLab).

Hello @greg
I saw you were the author of the change that introduced Arcanist into the thumbor-plugins project.
As I noticed Arcanist hasn't been actively used since 2019. Now, all the code review process is on Gerrit. Shall we need to support .arcconfig, .arclint in the project since it isn't used? Or is it the strict rule to have and maintain it in the repo?
@hnowlan maybe you have some ideas about it.
Please feel free to share your thoughts.

In T253280#8500398, @Tgr wrote:

T325948: Provide an API for accessing alt text (and possibly other structured data) stored with MediaWiki file uploads and T325949: Allow access to (some) structured image metadata across wikis is my vague idea of how to access Commons structured file metadata during thumbnail rendering. (The use case I'm personally interested in is alt text, but as I said above I think it is a fairly generic problem.)

The ticket was moved to Blocked on the Thumbor Migration board since the ticket requires decisions from the WMF management/engineering staff related to MediaWiki core, Structured Data, etc.

The ticket was moved to Blocked on the Thumbor Migration board since the ticket requires decisions from the WMF management/engineering staff related to MediaWiki core, Structured Data, etc.

The ticket was moved to Blocked on the Thumbor Migration board since the ticket requires decisions from the WMF management/engineering staff related to MediaWiki core, Structured Data, etc.

@TheDJ hello.
Thank you for the really good catch.
Does it happen on the current production version of Thumbor?

Hello @Jeff_G. It was interesting to work on this ticket because it was not really obvious what is going on with the mentioned files. I found nine buggy files.
I tested all files mentioned in the description of the ticket and the following link which goes to Commons_talk:CropTool.
I figured out that all thumbnails that we can't see even the preview fail with the next error on the production version of Thumbor -> RGB color space not permitted on grayscale PNG. I mean the thumbnails that look as in the screenshot below.

It seems that I found where is the trick. As it turned out the failed SVG file has a small body, as a result, the source in the prepare_source function can be created from a buffer and the buffer is a byte string. So it requires opening a file to write bytes.
Other files I took to test this bug have a bigger body, so the source was taken from the context and there is no need to create a temporary file and write to this file in the prepare_source function.

@hnowlan has checked the mentioned file on the staging and he didn't get any error as on the current production version of Thumbor. Since we will soon replace the current Thumbor production version(Debian Stretch/Python2.7) with the current staging one(Debian Buster/Python3.7), this ticket can be moved to done.

In T311004#8456746, @TheDJ wrote:

File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/imagemagick/imagemagick.py", line 62, in create_image
  temp_file.write(buffer)

I got the same error when I tried to thumbnail this file on the production version of Thumbor with Debian:Stretch and Python2.7 locally.
Here is the error message from libvips which causes the break of the code -> vipspng: profile 'icc': 'RGB ': RGB color space not permitted on grayscale PNG vips2png: unable to write.

In T311004#8453837, @Vlad.shapik wrote:

I just found that the mentioned file can be thumbnailed only if the name doesn't have any round brackets. I was able to reproduce it locally.

I have uploaded a TIF file to commons.wikimedia.beta.wmflabs.org with round brackets in the filename -> https://commons.wikimedia.beta.wmflabs.org/wiki/File:67-0372-C_Bann_Van_Hddds_(left)_and_Lai_Tio_Truffn.tif. And everything is ok. I believe my suspicion of round brackets in the filename is wrong.

@TheDJ thank you for your research. I am looking into it.
I just found that the mentioned file can be thumbnailed only if the name doesn't have any round brackets. I was able to reproduce it locally.
I successfully got an image for localhost:8800/thumbor/unsafe/800x/67-0372-C_Tran_Van_Huong_left_and_Mai_Tho_Truyen.tif
but
It failed for localhost:8800/thumbor/unsafe/800x/67-0372-C_Tran_Van_Huong_(left)_and_Mai_Tho_Truyen.tif
67-0372-C_Tran_Van_Huong_(left)_and_Mai_Tho_Truyen.tif is the original name of the mentioned file in the description of the ticket. The only difference is that the word left is enclosed in round brackets in the original filename.
Python 3.7 says that the file is not found.

In T256959#8371255, @gerritbot wrote:

Change 853402 had a related patch set uploaded (by Vlad.shapik; author: Vlad.shapik):

[operations/software/thumbor-plugins@master] Add ability to specify a DPI value for PDF

https://gerrit.wikimedia.org/r/853402

In T47212#8440100, @gerritbot wrote:

Change 863399 had a related patch set uploaded (by Vlad.shapik; author: Vlad.shapik):

[operations/software/thumbor-plugins@master] WP: Add ability to specify filters such as sharpening and etc. for TIFF format

https://gerrit.wikimedia.org/r/863399

I worked within the Per-file control ticket. I researched the following subtasks: PNG thumbnail..., TIF less focused..., Allow PDF's to be rendered... I found several useful comments and links to the related Phab tickets.
There is an idea to create a new URL scheme for getting thumbnails. From my point of view, it is not a blocker for us to add an opportunity to configure creating thumbnails, but the status of this ticket should be controlled since a new URL scheme can influence the logic of per-file control for thumbnail generation parameters based on the plan below. This comment describes a valuable approach of using structured data and checking the header which Thumbor will process and decide what to do with one or another user config parameter.
Here is the short plan which can help to add per-file control for some thumbnail generation parameters:

1. The community should define a standard for how file configs information is stored. As mentioned by @Tgr it could be a new property that needs to be proposed on Wikidata or could use some existing property.
2. We also need a Mediawiki side logic which will detect when some config parameter of a file is changed in structured data, and push the info to a Swift header.
3. This information would have to be stored in Swift alongside the original object as a special header, as access to the database from the thumbnailing layer is undesirable for security reasons.
4. It should be also possible to configure only per file not per thumbnail because it helps to do this without inflating storage needs. There needs to be a mechanism to keep those particular flags in sync in Swift. Switching the flag on and off would need to purge all thumbnails. Once one of the flags exists in Swift, it's trivial for Thumbor to generate thumbnails accordingly.
5. And at the end, Thumbor should check the header and apply a config parameter appropriately.

It seems like a lot of work should be done there.

The DPI value will need to be stored with other image meta-data somehow, hence the suggestion of storing ti as a property or field.

The front-end of this task is basically a matter of querying the stuctured data for the DPI value that should be used. The data is stored in a json file, but there are functions to get the information you need, like the Page Banner extension uses.

Hello @AntiCompositeNumber.
I am working on the BE solution to make it possible to configure the DPI option for PDF files. It is the sub-ticket T256959 of this one.
As I understood correctly you proposed that we have these configuration options already put in the structured data tab on the file page. And then thumbor will take possible configuration options and use them to generate the file on a case-by-case basis.
Did I get your point? Could you please share your ideas with me?

Hello @ShakespeareFan00
I am working on this ticket now. Could you please share with me the location of a 'property' for the File:'s entry in the image links table which you mentioned as a possible way to implement the DPI option in the FE?
And also will you be able to describe the idea of a 'modifier' template for use on sites like Wikimedia Commons and Wikisource?
For now, I just can't understand the connection between these two steps because I do not how the DPI option should look on these sites. Maybe you could give me some examples of already implemented options on these sites or something like that?
I just need to know if the front-end team(for now I don't know who will implement it in the FE) can use my BE solution to implement this configurator on the file page, for instance, that's why I am asking.

In T323114#8410298, @hnowlan wrote:

This appears to be fixed. The issue relates to us calling Tornado's set_header with a string that contains non-ascii characters. Encoding these strings to be utf-8 bytestrings fixes the issue.

In T318406#8360326, @gerritbot wrote:

Change 841183 merged by jenkins-bot:

[operations/software/thumbor-plugins@master] Provide additional tests to cover errors caused by wrong engine commands

https://gerrit.wikimedia.org/r/841183

In T318548#8329085, @dom_walden wrote:

So, I think the difference I am seeing is because I had not set RESPECT_ORIENTATION = True in my local thumbor config file. That option is set on beta. The default for that option is False.

When I add that option to my local thumbor config, the 4 files mentioned in the description are rendered in the correct orientation.

@roman-stolar @Vlad.shapik please double-check, but I think this bug is a false alarm.

The code coverage has increased from 75% to 76%. It means that 11 lines of errors related code have been covered by tests.

In T313016#8172824, @dom_walden wrote:

@pwangai and I have been reviewing the code coverage. We found we could run coverage run -m pytest || coverage html where you can see line-by-line code coverage.

The areas we seem to be missing code coverage:

• PoolCounter (https://wikitech.wikimedia.org/wiki/Thumbor#Poolcounter-based; code wikimedia_thumbor/poolcounter/__init__.py)
• Swift (code wikimedia_thumbor/loader/swift/__init__.py, wikimedia_thumbor/result_storage/swift/swift.py)
• Error/exception handling
• Several files in wikimedia_thumbor/logging/
• Several files in wikimedia_thumbor/error_handlers/
• wikimedia_thumbor/storage/request/request.py (not sure what this does)
• wikimedia_thumbor/core/importer.py (not sure what this does)
• wikimedia_thumbor/core/context_importer.py (not sure what this does)

Testing PoolCounter and Swift might not be possible without getting these services running locally. This might be hard, particularly with Swift.

In T252719#8158515, @ori wrote:

Hi, I'm trying to understand https://gerrit.wikimedia.org/r/c/operations/software/thumbor-plugins/+/800170/ a bit better.

• What parts of this change are coming from upstream and what parts are new?
• How did https://gerrit.wikimedia.org/r/c/operations/software/thumbor-plugins/+/489022/6 ended up getting included, when it has not (AFAICT) been merged?
• Did the new code get thoroughly reviewed? I only looked at one file, tests/integration/test_swift.py, and it looks like the change made at least some of the test code unreachable -- adding assert False to mock_get_object() does not result in a test failure.

Hi @ori.
I will create a new patch with reverted changes related to https://gerrit.wikimedia.org/r/c/operations/software/thumbor-plugins/+/489022/6

The results of the investigation:

1. In general current tests work as expected and it doesn't seem that they are written wrong.
2. Regarding the integration testing, most of the tests use one test case for testing - run_and_check_ssim_and_size. The number of testing parameters in this function can be extended which could bring better test cases. Additionally, I’ve noticed that “tests_3d.py” is not an online test, it has a local URL. And what's more, it can be as one of the variants to create more tests using different filters:(parameter), format:(parameter), etc of the URL for the Thumbor service to get a converted image.
3. As for unit tests, it also can be taken into consideration because very few functions of the source code are used in the existing tests. For now, it is only an 80% assumption which has appeared after working with tests. But the picture will be clearer after the code coverage check.