Page MenuHomePhabricator

Embed TinyRGB color profile when JPG EXIF Color Space = sRGB but no profile embedded
Closed, ResolvedPublic

Description

Last year, the thumbnailling process was enhanced (TinyRGB support for JPG uploads) to replace the standard sRGB embedded color profile with the TinyRGB (aka C2) profile from Facebook. This is because it is smaller and also Free. See also Thumbnails should embed tinyRGB instead of sRGB when they include that ICC profile.

However Facebook created that profile to solve a different problem -- to deal with images that lack an embedded color profile. Although the W3C standard expects browsers to assume sRGB when an embedded profile is missing (from JPG, PNG, CSS, etc) in fact all browsers (by default) will just assume the user's monitor profile in this case. While that is reasonable if the monitor is calibrated to sRGB colorspace, it is a bad assumption for anyone using a wide-gamut monitor. The result is that users with such monitors see gaudy over-saturated images on the web for sRGB images that lack an embedded color profile.

The problem is widespread since pretty much all JPG's created by cameras and smartphones lack an embedded color profile. Often it is only JPGs created by professional image software (Photoshop, Lightroom, etc) that are correctly saved with an included profile. There is further confusion since the problem is often described as affecting "untagged images". The JPG EXIF data has a ColorSpace tag which one might think would help. Its only valid values are 1=sRGB and 65535 = Uncalibrated. However all web browsers completely ignore this tag. See My Browser Test Page. The problem is really images that are sRGB but do not embed the sRGB color profile.

Additionally, many non-professional image programs strip off all color profile and EXIF data when saving JPGs. This leads to images that are almost certainly sRGB but lack any identifying attributes.

Although it is possible to configure Firefox to obey the W3C standard for untagged/unprofiled images (see above test page for details) this is not the out-of-the box experience, and all other browsers fail. For this reason, Facebook created TinyRGB so they could embed a color profile into every JPG at minimal cost. All desktop browsers respect the embedded color profile and display the image correctly, even on wide-gamut monitors.

I therefore propose a change to the TinyRGB code created by Giles. In that code, he examined the embedded color profile for the Device Model Desc attribute == IEC 61966-2.1 Default RGB colour space - sRGB. If an exact match is found, then the profile is replaced with TinyRGB. I propose instead two criteria must be met:

  1. Check the EXIF ColorSpace=sRGB or the tag is missing (which can be done with EXIFTOOL at the same time as getting any profile information).
  2. Check the profile Device Model Desc == IEC 61966-2.1 Default RGB colour space - sRGB or the profile is missing altogether

In the case of existing functionality, the additional check for the EXIF tag should be harmless as all valid JPGs will have that tag set to sRGB if they have gone to the trouble of embedding an sRGB profile.

Images that do not use sRGB but also fail to include an embedded profile (typically AdobeRGB color space) are also a problem, but far less common. There may be ways to identify and patch those up, but that could be a different lower-priority task.

This feature would only benefit thumbnailed images (which includes those displayed at fixed size on the image-description page). It doesn't alter the full-sized JPG. Another option would be to embed a profile at upload time when one is missing. It is the approach taken by many image hosting websites (Smugmug, Flickr) to ensure all their images look great. But it does change the file, which we might want to avoid, and it doesn't help with content already uploaded.

Btw, thumbnails a cached. How long is the cache held for and what would cause new thumbnails to be generated for existing images?

[Edit - added the following]
Steps to Reproduce:

  1. Get yourself a wide-gamut desktop monitor and configure both it and your operating system display profile to be AdobeRGB or native profile of the monitor. If you've done this correctly then go to this test page and section "How far from sRGB is your display color gamut?". You should see a different between the super-saturated ProPhotoRGB color swatches and the plain old sRGB color swatches. If you don't then you won't see the issue. You may also want to look at the "How does your browser interpret untagged images and page elements?" section to see how different the effect of including a profile can be.
  2. View an image on Commons that has no embedded profile via the thumbnailer. See My Browser Test Page and the section "Images without profiles". You can use Chrome, Firefox or Opera but do not use Internet Explorer or Edge since those are just broken wrt monitor color management.

Current Outcome:
The pencils on the left will appear more saturated than those in the section above. This is because they are being rendered with your wide-gamut monitor profile (AdobeRGB) when they are in fact just sRGB images.
Expected Outcome:
The pencils on the left should appear identical to those in the section above (assuming the cached thumbnails are re-generated).

Event Timeline

Colin created this task.May 5 2016, 3:26 PM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptMay 5 2016, 3:26 PM

Thanks for reporting this. Editing the task description to split into "steps to reproduce", "current outcome" and "expected outcome" sections is welcome for better readability. :)

Restricted Application added projects: Multimedia, Commons. · View Herald TranscriptMay 5 2016, 3:28 PM
Restricted Application added a subscriber: Steinsplitter. · View Herald Transcript
Colin updated the task description. (Show Details)May 5 2016, 3:49 PM
Restricted Application added subscribers: Poyekhali, Matanya. · View Herald TranscriptMay 5 2016, 3:49 PM
Colin updated the task description. (Show Details)May 5 2016, 3:51 PM
Colin updated the task description. (Show Details)May 5 2016, 3:59 PM

Change 307511 had a related patch set uploaded (by Matthias Mullie):
Embed TinyRGB color profile when JPG EXIF Color Space = sRGB but no profile embedded

https://gerrit.wikimedia.org/r/307511

I submitted a patch that - I believe - does what is suggested here:

  • Unless EXIF:ColorSpace is "sRGB" or missing, the file will not be touched (this is the case for e.g. AdobeRGB, whose EXIF:ColorSpace is "Uncalibrated")
  • Unless ICC_Profile:DeviceModelDesc is "IEC 61966-2.1 Default RGB colour space - sRGB" or missing, the file will be not be touched

If any of EXIF:ColorSpace or ICC_Profile:DeviceModelDesc indicates sRGB, the TinyRGB profile will be added.

If both EXIF:ColorSpace and ICC_Profile:DeviceModelDesc are missing, TinyRGB will also be added.
As I understood it, this is what we want because, according to w3c, browsers should assume sRGB in such cases.
Browsers, however, don't seem to properly do so, causing the image to be displayed differently across different monitors - this is why we'd be forcing the images to be sRGB.
Is this really what we want in this case, since we have no indication or guarantee that sRGB is what the image is supposed to be?

Gilles added a subscriber: Gilles.Aug 31 2016, 1:29 PM

Yeah ok that makes sense. It is true that most images produced without any of that data are likely to have assumed sRGB when they produced those files.

As for your earlier question about thumbnail generation, thumbnails are currently generated on demand and cached forever. You'd have to purge all those files if you wanted the existing thumbnails renewed, which is impractical. New thumbnails will run your code as they are generated.

Since I'm writing code for a new production thumbnail generation system based on Thumbor, I will create a follow-up task to reproduce the same logic as your code into the Wikimedia Thumbor plugins.

Colin added a comment.Aug 31 2016, 2:28 PM

I don't think the logic is quite correct. It is assuming that if DeviceModelDesc is missing then there is no profile. This is not a safe assumption. Both the TinyRGB and AdobeRGB sample JPG files lack this field, for example. Details of the ICC profile spec can be found

http://www.color.org/icc_specs2.xalter

and

file:///C:/Downloads/JWG7N0187_ISO-WD-20677_24-05-16.pdf

There are some tags that are derived from the header, so must be present (though could be blank) and there are also some tags that are mandatory. The best example for our purpose seems to be ProfileDescription. If this is missing, then no profile is present. In the examples used for testing, the values for this field are "Adobe RGB (1998)", "c2" and "sRGB IEC61966-2.1". It may be worth examining a variety of sRGB profile images to confirm that this name is always used for sRGB images. If so then it may be a better field to check than DeviceModelDesc.

Also, matthiasmullie's description with the bullet-points describes his fix but the subsequent if any/if both sentences do not. For example, if the ColorSpace=sRGB but the Profile is present and not sRGB then the file is untouched. In other words, it isn't sufficient for the ColorSpace tag to indicate sRGB. And this is desirable -- a non-sRGB profile should be retained regardless of contradictory ColorSpace tags.

Wrt the question "is this really what we want". The fact is that without an embedded colour profile, the actual colours rendered on the user's screen are not determined. Wc3 claim the browser should assume sRGB but they don't (for various reasons). So it is rendered without calibration to any profile and merely whatever the user's monitor is configured for. Different people will see different things. At least with this change, everyone will see the same thing. And 99.9% of the time, it was an sRGB image anyway.

It is a shame there is no way to "purge" a cached thumbnail on request. This could mean that users may see different colours (or saturations of colours) depending on what size thumbnail they view (based on what was cached before or after the change). Is there no way, for a given file, to request the thumbnails be regenerated?

You can have a given file's thumbnails regenerated by purging the file. I just meant that this can't be done on a large scale for all thumbnails. But you can absolutely fix all thumbnails for a given problematic file once the new code is in.

AdobeRGB images indeed lack ICC_Profile:DeviceModelDesc, but should have EXIF:ColorSpace set to "Uncalibrated", so they won't be touched.
But I agree that it would be better if we could get the profile more reliably than is now the case with the (sometimes missing) ICC_Profile:DeviceModelDesc.
Any case against using ProfileDescription instead? Was there a particular reason we currently use ICC_Profile:DeviceModelDesc?

Colin added a comment.Sep 5 2016, 5:22 PM

I can only guess that it wasn't appreciated that DeviceModelDesc wasn't mandatory. The previous code only engaged when it found a match, so if we had an sRGB profile without that field, then the worst that could happen was we used a larger sRGB rather than the smaller TinyRGB. But the new logic engages when a field is not found (with the assumption that this means there is no profile present at all). And the worse that could happen is that we replace a non-sRGB profile with TinyRGB and the colours are changed.

So it is much more important to be confident that the lack of a field means no profile. And the spec says that ProfileDescription is a required field. The actual string in ProfileDescription is slightly different to DeviceModelDesc, but should still be a reliable way of identifying sRGB.

While non sRGB profiles like AdobeRGB and ProPhotoRGB should imply EXIF:ColorSpace is set to Uncalibrated, browsers just seem to ignore that field completely. If a profile is present, it supersedes all the EXIF colour information wrt how to render the image.

Gilles added a comment.Sep 5 2016, 6:45 PM

ProfileDescription is definitely better. I think we could simply switch to using that to start with? This is what I've been using for my Thumbor-based rewrite of the thumbnailing code.

Gilles added a comment.Sep 6 2016, 1:03 PM

@Colin are you happy with the current state of the patch? Namely, if my understanding is correct:

  • If a low quality thumbnail is requested (this is a feature we use for the mobile site when served in regions with poor internet performance), only apply TinyRGB if sRGB was there and it makes the file lighter. That's what we had already, albeit now looking at the more reliable ProfileDescription field.
  • If a regular thumbnail is requested, replace the heavy sRGB by TinyRGB when found. If no profile is included and ColorSpace is either sRGB or missing, apply TinyRGB.
  • When a different profile is found, leave it as-is.
Colin added a comment.Sep 6 2016, 3:38 PM

Giles, I'm not very familiar with gerrit or php but think I'm looking at the latest code. I assume

$profiles[] = '-';

adds '-' to the array along with the sRGB profile name, thus making "profile field not found" another reason to include TinyRGB.

I haven't examined all the test cases. Your description above sounds good.

To be honest, the current situation with mobile browsers is that the profile is ignored (see my test page with a mobile browser). So really there is no point in sending any profile if one is looking to save bytes. If one cared to get the colours right on such a device, then we'd want to transform the minority of images with non-sRGB profiles into the sRGB colourspace (which is a more expensive operation, for the server, than calling EXIFTOOL) and then send them all without profiles. And strip off some unnecessary EXIF fields too. But doing such a thing is really a task for another ticket, and you'd need to consider the legacy of cached images once mobile devices finally starting doing colours correctly and internet connections improve. I'm not really that enthusiastic about doing that unless one really really was desperate to save bytes.

You mention the "mobile site when served in regions with poor internet performance". Does that mean they have different cached thumbs to users browsing on a mobile in the UK/USA, say? Does it know whether the user is browsing over WIFI or mobile data?

I'm sure that mobile devices will start taking colours seriously when some of them adopt wide-gamut and HDR displays to match the very latest video standards. Because if they don't then people buying such devices will complain about gaudy photographs.

Thanks again for implementing this. People using wide-gamut displays will be grateful. I only wish more camera manufacturers would embed profiles for their out-of-camera JPGs, but I think they are still stuck thinking when memory cards only held a few photos.

Actually, wide gamut displays are coming to mobile. Apple's iPhone 7 supports DCI-P3 both in its cameras and its display. I don't know if JPGs taken with it will embed that profile by default. I'll be getting one of these phones soon and will check for myself. In the Android world this is nothing new, with old Galaxy Note models already claiming the ability to display 97% of the Adobe RGB spectrum: https://en.wikipedia.org/wiki/AMOLED

There is a difference, however, between what the phone can do in its photo app and what it will do in browsers. I have no idea if any of these phones do "the right thing" in their browsers.

This rollout of wide gamut mobile support is why I've never pursued the obvious idea of stripping color profiles wholesale for mobile, as I anticipated this technology was coming.

For the low-bandwidth option, though, it would make sense to do so, as we've clearly decided to trade poorer thumbnail quality for faster page load already. This is something I can look at next quarter.

As for your question regarding wifi, etc. the way I believe this was solved is to prompt users to switch to the high quality version of the page if they wish to, in a sticky fashion. It's just the low bandwidth experience by default in areas where we know that the average visitor has very poor bandwidth. Those lucky enough to live in those areas and have good bandwidth can opt into the regular version of the mobile site with "high quality" thumbnails. We aimed for a better default, but people can opt in/out if they want to.

Colin added a comment.Sep 8 2016, 2:17 PM

Interesting. I wonder what % of DCI-P3 the phone display supports. Apparently the latest imac supports close to 100%. However, I read:

http://www.colourspace.xyz/the-new-apple-imac-and-the-dci-p3-colour-gamut/

that DCI-P3 in in some ways weaker than AdobeRGB so may be less suitable for photographers who print. I also note the claim that Safari 9 correctly handles untagged images on a wide-gamut browser -- which is the issue we are dealing with here. If true, then we only have IE, Edge, Chrome and out-of-the-box Firefox to get in line! I would also be interested if folk who have current apple browsers and phones could update my browser test page on Commons if required.

I suspect the iphone7 JPGs will have sRGB colourspace and i would be interested to know if they bother to embed a colour profile. I reckon DCI-P3 is for watching movies, not taking photos. If they took photos in DCI-P3 then anyone looking at them on an unmanaged browser or program would see very dull colours. But perhaps there will be an option.

I'm pretty sure I've read that Rec.2020 (fully wide gamut) requires 10-bit to avoid posterising so is not suitable for 8-bit JPGs and requires 10-bit workflow to manage. It would be lovely to see a modern 10-bit replacement for JPG, but all the work in this area seems to concern movies.

Thanks for explaining how the low-bandwidth version works. For saving bandwidth, I would have thought you could remove pretty much all the EXIF, any embedded thumbnail JPG and the colour profile. Some files accumulate quite a lot of EXIF cruft after they have been through Lightroom, Photoshop or even just with lots of camera manufacturer data.

Gilles added a comment.Sep 8 2016, 5:05 PM

Yes, Safari has been known to handle color profiles correctly for a very long time.

As for selective EXIF stripping, it's a long standing request that I've implemented in my Thumbor-based rewrite of the thumbnailing pipeline. We'll retain Artist, Copyright and whatever fields the Commons community wants to remain in thumbnails. These fields are usually reasonable length-wise and only weigh a few bytes. The reason why we didn't strip everything before is that people were against losing all attribution. Hence why we needed to wait for a way to do selective filtering.

Gilles added a comment.Sep 8 2016, 5:08 PM

Looking at your test page on OS X:

  • On Firefox the 4 images with profiles look the same, they seem a bit oversaturated. The 2 images without profiles look the same and not saturated
  • On Chrome all 6 images look the same, not saturated (they look identical to images without a profile on firefox)
  • On Safari, same as Chrome
Colin added a comment.Sep 8 2016, 6:07 PM

Giles, perhaps you could post your findings on my test page (start a discussion on Talk if you like). I'm confused about your results, and need more info about your setup/monitor/any calibration/profiling you have done. Makes me wonder if you have set your OS to a profile that does not actually correspond with your monitor (or accidentally changed the mode on your monitor). Anyway, this is getting off topic.

This discussion has gone slightly off-topic :)

If I read it correctly, the patch in its current state (as described by @Gilles in T134498#2610900) adequately addresses this ticket.
Right? Or did I miss another detail?

Colin added a comment.Nov 16 2016, 8:46 AM

I believe it does. Thanks.

dr0ptp4kt triaged this task as Normal priority.May 1 2017, 3:17 PM

Change 307511 merged by jenkins-bot:
[mediawiki/core@master] Embed TinyRGB color profile when JPG EXIF Color Space = sRGB but no profile embedded

https://gerrit.wikimedia.org/r/307511

matthiasmullie closed this task as Resolved.May 4 2017, 8:08 AM