josch/img2pdf

Fork 10

Images with ICC profile silently converted to sRGB profile #83

New issue

Closed

opened 2021-04-25 19:58:53 +00:00 by josch · 0 comments

josch commented

2021-04-25 19:58:53 +00:00

Owner

By jbarlow83 on 2020-07-07T07:04:20.047Z

If a color image has a color profile, img2pdf will silently discard it instead of preserving it.

The example image is a PNG set to the ProPhoto RGB profile, with the profile embedded. Pillow is capable of extracting it.

The PDF produced by img2pdf will have the profile dropped, which means it is assumed to be sRGB. As a result, it will look like "washed-out purple" rather than "fuschia-pink".

Also included is a PDF repaired with pikepdf to include the appropriate color profile. This is for the case of a paletted (/Indexed) image.

pink-palette-icc

pink-palette-icc-img2pdf.pdf

pink-palette-icc-correct.pdf

By josch on 2020-08-06T22:15:26.567Z

I think this is a duplicate of #65

I just rebased the patch onto recent img2pdf (especially with the new test suite) and am checking:

    if 'icc_profile' in imgdata.info:
        iccp = imgdata.info.get('icc_profile')

I'm still struggeling with testing this though. With the help of @iwata74 I'm now able to use imagemagick to compare the pdf as rendered by ghostscript like so:

$ img2pdf png_with_icc.png -o out.pdf
$ gs -dQUIET -dNOPAUSE -dBATCH -sDEVICE=png16m -r96 -sOutputFile="out-gs.png" out.pdf
$ convert -profile reverse.icc png_with_icc.png png_reversed.png
$ compare -metric AE out-gs.png png_reversed.png

Under Debian unstable I created png_with_icc.png with /usr/share/color/icc/sRGB.icc embedded in it and used /usr/share/color/icc/ghostscript/srgb.icc as reverse.icc in the commands above.

Strangely enough, this procedure only works when rendering the pdf as created by img2pdf with ghostscript. If I use mupdf, poppler or pdfimages, I get slightly different results which are not exact anymore.

I'm also puzzled by the fact that I cannot distinguish your two files pink-palette-icc-img2pdf.pdf and pink-palette-icc-correct.pdf from each other -- which pdf viewer are you using?

My first try in implementing this is in commit f0b57985ee but as you can see, compare_poppler, compare_mupdf and compare_pdfimages_png are commented out because I could not get them to compare exactly. Do you know what the problem could be?

By josch on 2020-08-06T22:16:50.069Z

Mentioned in issue #65

By jbarlow83 on 2020-08-07T08:35:07.665Z

If you can't distinguish the files your PDF viewer or system is not color managed. The difference should be pretty stark.

Here is how it looks to me:

I double-checked them in Acrobat and macOS Preview. Firefox cannot tell the difference (it does not have color managed PDF, but does have color managed images).If it's still an issue I believe there are test ICC profiles that exchange RGB->GBR or something, so it's immediately obvious if the color is wrong. I could see about regenerating with those.

When comparing, one or more of those programs may not apply color profiles correctly or might round them differently when they are applied, because it can be a lossy conversion.

By josch on 2020-08-07T10:52:03.343Z

Thanks a lot for going through the trouble of posting a screenshot. Indeed I see a difference but to my eyes, it's really minor, so it would not surprise me, if I am just either really bad at distinguishing these colors or my laptop monitor is really bad or both. XD But of course what counts is not that I am able to distinguish these but that the testsuite is able to distinguish these.

The idea of using a test profile sounds very intriguing and very useful. Then even people with bad eyes and/or monitors like me would be able to easily spot a problem. :)

I found this, for example and that looks useful, but unfortunately, the used ICC profile is proprietary. Do you know of some test profiles under a free license? Or maybe it's really simple to auto-generate a test profile?

By josch on 2020-08-07T17:39:38.923Z

I was now playing around with channel-swapping ICC profiles and noticed, that my image viewer (feh) ignores ICC profiles embedded in PNG images -- no wonder I never saw a difference! Firefox seems to be okay with ICC profiles in png images, but ignores them in PDF files. My normal PDF viewer (mupdf) also ignores embedded profiles while evince takes them into account. What a mess! So no wonder nothing worked for me if my normal image viewer as well as my normal pdf viewer both are ignorant about ICC profiles.

Thanks for the color-swapping tip -- that made me finally find out all this. :)

By josch on 2020-08-07T18:19:04.893Z

@jbarlow83 sorry to bother you again, but I somehow have to make sure that I'm not crazy. Take these two images:

The first image has no embedded ICC profile. The second one contains a profile that switches colors. In firefox, those two PNG images, indeed look different, so it seems the profile embedded in the second gets respected. But if I open the second image (the one with the profile) with display from imagemagick, then it looks exactly like the first. Even worse:

compare -metric AE icc.png no-icc.png null:
0

So even imagemagick claims that those two images are absolutely identical even though they should not be. How am I supposed to write proper tests for PNG images and PDF files if the tooling is absolutely incapable of working with ICC profiles? You write that you are working with Acrobat and MacOS preview but I'm afraid that's a no-go for the testsuite of img2pdf. Do you have any suggestions for the tools I should use to properly work with images and PDFs containing ICC profiles?

Or maybe I'm making a mistake with imagemagick? I find it extremely weird, that the most popular image manipulation software cannot handle this?

By jbarlow83 on 2020-08-08T09:15:49.856Z

Well, if imagemagick were competent at everything, img2pdf wouldn't need to exist, would it? :)

I think imagemagick ignores the color profile in some cases.

But I think your tests are "backwards" too: it would be easier to have images with different numerical values, that with the correct color profile will present as the same wavelengths of light. You want to switch the colors, and apply a profile that switches them back to normal.

Try these ones, which are permissively licensed and could be included in your test suite. They conveniently have text that becomes visible only if something is wrong:
http://displaycal.net/icc-color-management-test/

By josch on 2020-08-09T20:19:53.856Z

@jbarlow83 thank you for your input!

The original issue got fixed with the 0.4.0 upload, right?

Actually I would prefer it, if there existed other software that was able to losslessly turn a raster graphic into a pdf, because then I'd have to do less work. That's why in the README I compare img2pdf to other software. I'm not writing software just to stay busy -- I have enough better things to do. The only reason I maintain img2pdf is, because I do not see any other software that is able to do the same thing, sadly. In an ideal future, imagemagick would be able to do the same thing img2pdf is able to do and then I could finally put img2pdf into the attic.

Thanks for the additional resource containing ICC profiles (and images) for testing purposes, but even though they are distributed under a free license, to me they are still binary blobs and even a binary blob under a free license, does not give me the freedom to change and improve it. So instead I investigated how to create my own color channel switching ICC profiles from scratch in commit 11907242a5 -- the colors get switch just fine but I'm not able to make it work with imagemagick without using the ghostscript sRGB profile.

Yes, the tests are backwards. As you say, I should first switch the color channels with imagemagick and then correct them using the ICC profile. Unfortunately, I had no luck with that approach. This might be either due to imagemagick or due to my inability to use it right.

*By jbarlow83 on 2020-07-07T07:04:20.047Z* If a color image has a color profile, img2pdf will silently discard it instead of preserving it. The example image is a PNG set to the ProPhoto RGB profile, with the profile embedded. Pillow is capable of extracting it. The PDF produced by img2pdf will have the profile dropped, which means it is assumed to be sRGB. As a result, it will look like "washed-out purple" rather than "fuschia-pink". Also included is a PDF repaired with pikepdf to include the appropriate color profile. This is for the case of a paletted (/Indexed) image. [pink-palette-icc](/uploads/1b5fec9258c202b7f8efc9d445a0e98f/pink-palette-icc.png) [pink-palette-icc-img2pdf.pdf](/uploads/fae69ff4c0debd8bbbf0cbf57e911f90/pink-palette-icc-img2pdf.pdf) [pink-palette-icc-correct.pdf](/uploads/eb627573f5d444105db06ea1b939bfc2/pink-palette-icc-correct.pdf) --- *By josch on 2020-08-06T22:15:26.567Z* --- I think this is a duplicate of https://gitlab.mister-muffin.de/josch/img2pdf/issues/65 I just rebased the patch onto recent img2pdf (especially with the new test suite) and am checking: ```python if 'icc_profile' in imgdata.info: iccp = imgdata.info.get('icc_profile') ``` I'm still struggeling with testing this though. With the help of @iwata74 I'm now able to use imagemagick to compare the pdf as rendered by ghostscript like so: ``` $ img2pdf png_with_icc.png -o out.pdf $ gs -dQUIET -dNOPAUSE -dBATCH -sDEVICE=png16m -r96 -sOutputFile="out-gs.png" out.pdf $ convert -profile reverse.icc png_with_icc.png png_reversed.png $ compare -metric AE out-gs.png png_reversed.png ``` Under Debian unstable I created `png_with_icc.png` with `/usr/share/color/icc/sRGB.icc` embedded in it and used `/usr/share/color/icc/ghostscript/srgb.icc` as `reverse.icc` in the commands above. Strangely enough, this procedure only works when rendering the pdf as created by img2pdf with ghostscript. If I use mupdf, poppler or pdfimages, I get slightly different results which are not exact anymore. I'm also puzzled by the fact that I cannot distinguish your two files pink-palette-icc-img2pdf.pdf and pink-palette-icc-correct.pdf from each other -- which pdf viewer are you using? My first try in implementing this is in commit f0b57985eee77c2572ae94821d684df7659146c3 but as you can see, compare_poppler, compare_mupdf and compare_pdfimages_png are commented out because I could not get them to compare exactly. Do you know what the problem could be? --- *By josch on 2020-08-06T22:16:50.069Z* --- Mentioned in issue #65 --- *By jbarlow83 on 2020-08-07T08:35:07.665Z* --- If you can't distinguish the files your PDF viewer or system is not color managed. The difference should be pretty stark. Here is how it looks to me: ![Screen_Shot_2020-08-07_at_01.31.42](/uploads/fe6894989d5422afc79c5f8ee0e40bcc/Screen_Shot_2020-08-07_at_01.31.42.png) I double-checked them in Acrobat and macOS Preview. Firefox cannot tell the difference (it does not have color managed PDF, but does have color managed images).If it's still an issue I believe there are test ICC profiles that exchange RGB->GBR or something, so it's immediately obvious if the color is wrong. I could see about regenerating with those. When comparing, one or more of those programs may not apply color profiles correctly or might round them differently when they are applied, because it can be a lossy conversion. --- *By josch on 2020-08-07T10:52:03.343Z* --- Thanks a lot for going through the trouble of posting a screenshot. Indeed I see a difference but to my eyes, it's really minor, so it would not surprise me, if I am just either really bad at distinguishing these colors or my laptop monitor is really bad or both. XD But of course what counts is not that ***I*** am able to distinguish these but that the testsuite is able to distinguish these. The idea of using a test profile sounds very intriguing and very useful. Then even people with bad eyes and/or monitors like me would be able to easily spot a problem. :) I found [this, for example](https://github.com/mpv-player/random-stuff/tree/master/icc_profiles) and that looks useful, but unfortunately, the used ICC profile is proprietary. Do you know of some test profiles under a free license? Or maybe it's really simple to auto-generate a test profile? --- *By josch on 2020-08-07T17:39:38.923Z* --- I was now playing around with channel-swapping ICC profiles and noticed, that my image viewer (feh) ignores ICC profiles embedded in PNG images -- no wonder I never saw a difference! Firefox seems to be okay with ICC profiles in png images, but ignores them in PDF files. My normal PDF viewer (mupdf) also ignores embedded profiles while evince takes them into account. What a mess! So no wonder nothing worked for me if my normal image viewer as well as my normal pdf viewer both are ignorant about ICC profiles. Thanks for the color-swapping tip -- that made me finally find out all this. :) --- *By josch on 2020-08-07T18:19:04.893Z* --- @jbarlow83 sorry to bother you again, but I somehow have to make sure that I'm not crazy. Take these two images: ![no-icc](/uploads/1fabc03f907f081e2c6319141938585e/no-icc.png) ![icc](/uploads/447d98a8d0536b38610397c3ecc5dc6c/icc.png) The first image has no embedded ICC profile. The second one contains a profile that switches colors. In firefox, those two PNG images, indeed look different, so it seems the profile embedded in the second gets respected. But if I open the second image (the one with the profile) with `display` from imagemagick, then it looks exactly like the first. Even worse: compare -metric AE icc.png no-icc.png null: 0 So even imagemagick claims that those two images are absolutely identical even though they should not be. How am I supposed to write proper tests for PNG images and PDF files if the tooling is absolutely incapable of working with ICC profiles? You write that you are working with Acrobat and MacOS preview but I'm afraid that's a no-go for the testsuite of img2pdf. Do you have any suggestions for the tools I should use to properly work with images and PDFs containing ICC profiles? Or maybe I'm making a mistake with imagemagick? I find it extremely weird, that the most popular image manipulation software cannot handle this? --- *By jbarlow83 on 2020-08-08T09:15:49.856Z* --- Well, if imagemagick were competent at everything, img2pdf wouldn't need to exist, would it? :) I think imagemagick ignores the color profile in some cases. But I think your tests are "backwards" too: it would be easier to have images with different numerical values, that with the correct color profile will present as the same wavelengths of light. You want to switch the colors, and apply a profile that switches them back to normal. Try these ones, which are permissively licensed and could be included in your test suite. They conveniently have text that becomes visible only if something is wrong: http://displaycal.net/icc-color-management-test/ --- *By josch on 2020-08-09T20:19:53.856Z* --- @jbarlow83 thank you for your input! The original issue got fixed with the 0.4.0 upload, right? Actually I would prefer it, if there existed other software that was able to losslessly turn a raster graphic into a pdf, because then I'd have to do less work. That's why in the README I compare img2pdf to other software. I'm not writing software just to stay busy -- I have enough better things to do. The only reason I maintain img2pdf is, because I do not see any other software that is able to do the same thing, sadly. In an ideal future, imagemagick would be able to do the same thing img2pdf is able to do and then I could finally put img2pdf into the attic. Thanks for the additional resource containing ICC profiles (and images) for testing purposes, but even though they are distributed under a free license, to me they are still binary blobs and even a binary blob under a free license, does not give me the freedom to change and improve it. So instead I investigated how to create my own color channel switching ICC profiles from scratch in commit 11907242a5b5cebb8c267e8f8d4b945d7357baf0 -- the colors get switch just fine but I'm not able to make it work with imagemagick without using the ghostscript sRGB profile. Yes, the tests are backwards. As you say, I should first switch the color channels with imagemagick and then correct them using the ICC profile. Unfortunately, I had no luck with that approach. This might be either due to imagemagick or due to my inability to use it right.