PDF timestamp handling issue #155

By default, it seems that img2pdf treats the "current" time as UTC time without taking the configured system timezone into account. A workaround is to use the --creationdate and --moddate flags and pass timespecs to those that include the timezone info.

The img2pdf version is 0.4.4, running on Arch Linux.

My test script:

#!/bin/bash

# create test image with imagemagick
convert logo: test-image.jpg

DATE="$(date --iso-8601=seconds)"
echo "expected timestamp (ISO format): $DATE"
img2pdf -o test1.pdf test-image.jpg
pdfinfo test1.pdf | grep Date

# workaround:
DATE="$(date --iso-8601=seconds)"
echo "expected timestamp (ISO format): $DATE"
img2pdf --creationdate="$DATE" --moddate="$DATE" -o test2.pdf test-image.jpg
pdfinfo test2.pdf | grep Date

And its output on my system:

expected timestamp (ISO format): 2023-02-12T11:51:41-08:00
CreationDate:    Sun Feb 12 03:51:41 2023 PST
ModDate:         Sun Feb 12 03:51:41 2023 PST
expected timestamp (ISO format): 2023-02-12T11:51:41-08:00
CreationDate:    Sun Feb 12 11:51:41 2023 PST
ModDate:         Sun Feb 12 11:51:41 2023 PST

By default, it seems that img2pdf treats the "current" time as UTC time without taking the configured system timezone into account. A workaround is to use the `--creationdate` and `--moddate` flags and pass timespecs to those that include the timezone info. The img2pdf version is 0.4.4, running on Arch Linux. My test script: ```bash #!/bin/bash # create test image with imagemagick convert logo: test-image.jpg DATE="$(date --iso-8601=seconds)" echo "expected timestamp (ISO format): $DATE" img2pdf -o test1.pdf test-image.jpg pdfinfo test1.pdf | grep Date # workaround: DATE="$(date --iso-8601=seconds)" echo "expected timestamp (ISO format): $DATE" img2pdf --creationdate="$DATE" --moddate="$DATE" -o test2.pdf test-image.jpg pdfinfo test2.pdf | grep Date ``` And its output on my system: ``` expected timestamp (ISO format): 2023-02-12T11:51:41-08:00 CreationDate: Sun Feb 12 03:51:41 2023 PST ModDate: Sun Feb 12 03:51:41 2023 PST expected timestamp (ISO format): 2023-02-12T11:51:41-08:00 CreationDate: Sun Feb 12 11:51:41 2023 PST ModDate: Sun Feb 12 11:51:41 2023 PST ```

(I was the user (phmccarty) that created this issue, but for some reason my account was deleted... Hoping that it will stay around after creating a fresh account today.)

Interestingly, re-running my test script on Arch Linux with Python 3.11.3, I see different results, so behavior must have changed between Python 3.10 and 3.11.

Current results (without my MR 168) are below. Notice that the --creationdate and --moddate flags are no longer taking effect as a workaround.

expected timestamp (ISO format): 2023-05-29T14:31:59-07:00
CreationDate:    Mon May 29 07:31:59 2023 PDT
ModDate:         Mon May 29 07:31:59 2023 PDT
expected timestamp (ISO format): 2023-05-29T14:31:59-07:00
CreationDate:    Mon May 29 07:31:59 2023 PDT
ModDate:         Mon May 29 07:31:59 2023 PDT

Regardless, the changes in MR 168 appear to help, at least on Arch Linux, the only distro I've tested. Hopefully that approach works for older Python versions and on other distros/platforms.

(I was the user (phmccarty) that created this issue, but for some reason my account was deleted... Hoping that it will stay around after creating a fresh account today.) Interestingly, re-running my test script on Arch Linux with Python 3.11.3, I see different results, so behavior must have changed between Python 3.10 and 3.11. Current results (without my MR 168) are below. Notice that the `--creationdate` and `--moddate` flags are no longer taking effect as a workaround. ``` expected timestamp (ISO format): 2023-05-29T14:31:59-07:00 CreationDate: Mon May 29 07:31:59 2023 PDT ModDate: Mon May 29 07:31:59 2023 PDT expected timestamp (ISO format): 2023-05-29T14:31:59-07:00 CreationDate: Mon May 29 07:31:59 2023 PDT ModDate: Mon May 29 07:31:59 2023 PDT ``` Regardless, the changes in MR 168 appear to help, at least on Arch Linux, the only distro I've tested. Hopefully that approach works for older Python versions and on other distros/platforms.

Sorry for that. There are hundreds of bots per month creating multiple accounts and spamming my gitea installation with several repos, issues and comments to post their spam per day. I have an automated script that uses some heuristics to find these bots and that script must've wrongly flagged your account and removed it. I'm sorry. Thank you for coming back! I put your username on the allow-list so that this cannot happen again in the future.

Sorry for that. There are hundreds of bots per month creating multiple accounts and spamming my gitea installation with several repos, issues and comments to post their spam per day. I have an automated script that uses some heuristics to find these bots and that script must've wrongly flagged your account and removed it. I'm sorry. Thank you for coming back! I put your username on the allow-list so that this cannot happen again in the future.

Thanks for checking in on this!

> Sorry for that. There are hundreds of bots per month creating multiple accounts and spamming my gitea installation with several repos, issues and comments to post their spam per day. I have an automated script that uses some heuristics to find these bots and that script must've wrongly flagged your account and removed it. I'm sorry. Thank you for coming back! I put your username on the allow-list so that this cannot happen again in the future. Thanks for checking in on this!

I'm definitely coming back to this issue before the next img2pdf release. First though I have to understand the problem better because my brain long discarded any knowledge I used to have about date/time representations in PDF. From what you write this sounds indeed like a bug in img2pdf that needs fixing.

I poked around a little bit more. I saw that you implemented some more code to represent the timezone offset properly. Would it not be easier to just store the timestamp in UTC instead of the local time? This would only inconvenience people reading the PDF in a text editor as any viewer would be able to display the timestamp in local time.

now = datetime.now().astimezone().astimezone(timezone.utc)

This would first make datetime.now() aware of the local timezone and then convert that datetime to utc.

I'm still having problems with this though with evince which which is not showing the correct time no matter what I do. This shows another problem of implementing this: you currently rely on the pdfinfo tool doing the right thing with respect to timezones. Is there another tools doing the same thing so that we can check that the implementation is indeed correct and maybe evince is buggy?

I poked around a little bit more. I saw that you implemented some more code to represent the timezone offset properly. Would it not be easier to just store the timestamp in UTC instead of the local time? This would only inconvenience people reading the PDF in a text editor as any viewer would be able to display the timestamp in local time. now = datetime.now().astimezone().astimezone(timezone.utc) This would first make `datetime.now()` aware of the local timezone and then convert that datetime to utc. I'm still having problems with this though with evince which which is not showing the correct time no matter what I do. This shows another problem of implementing this: you currently rely on the `pdfinfo` tool doing the right thing with respect to timezones. Is there another tools doing the same thing so that we can check that the implementation is indeed correct and maybe evince is buggy?

This works for me both with pdfinfo as well as with evince:

diff --git a/src/img2pdf.py b/src/img2pdf.py
index 4300999..2acc0f6 100755
--- a/src/img2pdf.py
+++ b/src/img2pdf.py
@@ -36,7 +36,7 @@ if hasattr(GifImagePlugin, "LoadingStrategy"):
 
 # TiffImagePlugin.DEBUG = True
 from PIL.ExifTags import TAGS
-from datetime import datetime
+from datetime import datetime, timezone
 from jp2 import parsejp2
 from enum import Enum
 from io import BytesIO
@@ -722,7 +722,7 @@ class pdfdoc(object):
             self.writer.docinfo = PdfDict(indirect=True)
 
         def datetime_to_pdfdate(dt):
-            return dt.strftime("%Y%m%d%H%M%SZ")
+            return dt.strftime("%Y%m%d%H%M%S+00'00'")
 
         for k in ["Title", "Author", "Creator", "Producer", "Subject"]:
             v = locals()[k.lower()]
@@ -732,7 +732,7 @@ class pdfdoc(object):
                 v = PdfString.encode(v)
             self.writer.docinfo[getattr(PdfName, k)] = v
 
-        now = datetime.now()
+        now = datetime.now(tz=timezone.utc)
         for k in ["CreationDate", "ModDate"]:
             v = locals()[k.lower()]
             if v is None and nodate:
@@ -752,7 +752,7 @@ class pdfdoc(object):
                 )
 
         def datetime_to_xmpdate(dt):
-            return dt.strftime("%Y-%m-%dT%H:%M:%SZ")
+            return dt.strftime("%Y-%m-%dT%H:%M:%S+00:00")
 
         self.xmp = b"""<?xpacket begin='\xef\xbb\xbf' id='W5M0MpCehiHzreSzNTczkc9d'?>
 <x:xmpmeta xmlns:x='adobe:ns:meta/' x:xmptk='XMP toolkit 2.9.1-13, framework 1.6'>

Can you confirm?

This works for me both with pdfinfo as well as with evince: ```diff diff --git a/src/img2pdf.py b/src/img2pdf.py index 4300999..2acc0f6 100755 --- a/src/img2pdf.py +++ b/src/img2pdf.py @@ -36,7 +36,7 @@ if hasattr(GifImagePlugin, "LoadingStrategy"): # TiffImagePlugin.DEBUG = True from PIL.ExifTags import TAGS -from datetime import datetime +from datetime import datetime, timezone from jp2 import parsejp2 from enum import Enum from io import BytesIO @@ -722,7 +722,7 @@ class pdfdoc(object): self.writer.docinfo = PdfDict(indirect=True) def datetime_to_pdfdate(dt): - return dt.strftime("%Y%m%d%H%M%SZ") + return dt.strftime("%Y%m%d%H%M%S+00'00'") for k in ["Title", "Author", "Creator", "Producer", "Subject"]: v = locals()[k.lower()] @@ -732,7 +732,7 @@ class pdfdoc(object): v = PdfString.encode(v) self.writer.docinfo[getattr(PdfName, k)] = v - now = datetime.now() + now = datetime.now(tz=timezone.utc) for k in ["CreationDate", "ModDate"]: v = locals()[k.lower()] if v is None and nodate: @@ -752,7 +752,7 @@ class pdfdoc(object): ) def datetime_to_xmpdate(dt): - return dt.strftime("%Y-%m-%dT%H:%M:%SZ") + return dt.strftime("%Y-%m-%dT%H:%M:%S+00:00") self.xmp = b"""<?xpacket begin='\xef\xbb\xbf' id='W5M0MpCehiHzreSzNTczkc9d'?> <x:xmpmeta xmlns:x='adobe:ns:meta/' x:xmptk='XMP toolkit 2.9.1-13, framework 1.6'> ``` Can you confirm?

Can you confirm?

Your patch works for the default case, but it does not work when specifying --creationdate and/or --moddate strings that have non-UTC timezones.

The output of my script with your patch:

expected timestamp (ISO format): 2023-06-10T16:35:25-07:00
CreationDate:    Sat Jun 10 16:35:25 2023 PDT
ModDate:         Sat Jun 10 16:35:25 2023 PDT
expected timestamp (ISO format): 2023-06-10T16:35:25-07:00
CreationDate:    Sat Jun 10 09:35:25 2023 PDT
ModDate:         Sat Jun 10 09:35:25 2023 PDT

Edit: In order to correct the --creationdate and --moddate offsets, you would need to convert those datetime objects to UTC as well before calling datetime_to_pdfdate and datetime_to_xmpdate.

> Can you confirm? Your patch works for the default case, but it does not work when specifying `--creationdate` and/or `--moddate` strings that have non-UTC timezones. The output of my script with your patch: ``` expected timestamp (ISO format): 2023-06-10T16:35:25-07:00 CreationDate: Sat Jun 10 16:35:25 2023 PDT ModDate: Sat Jun 10 16:35:25 2023 PDT expected timestamp (ISO format): 2023-06-10T16:35:25-07:00 CreationDate: Sat Jun 10 09:35:25 2023 PDT ModDate: Sat Jun 10 09:35:25 2023 PDT ``` Edit: In order to correct the `--creationdate` and `--moddate` offsets, you would need to convert those datetime objects to UTC as well before calling `datetime_to_pdfdate` and `datetime_to_xmpdate`.

Thanks for the hint about astimezone() method, by the way; I had somehow missed it while researching for a solution to this issue...

Would it not be easier to just store the timestamp in UTC instead of the local time? This would only inconvenience people reading the PDF in a text editor as any viewer would be able to display the timestamp in local time.

I originally wanted to add the timezone info to now(), which astimezone() is appropriate for. Not knowing about astimezone() originally, I then opted to convert now() time to UTC.

I agree that it would be easier to store the UTC timestamps by using the hardcoded offset in the string, as you did in your patch, as long as --creationdate and --moddate strings are converted before that point (see my edited previous comment). Not needing to extract the offset substring from the %z specifier would certainly be simpler to understand in the code.

I'm still having problems with this though with evince which which is not showing the correct time no matter what I do.

I'm wondering if we are interpreting evince's document properties the same... With my merge request applied, I checked evince, and I see that it printed the timestamps according to the "raw" offset (not converted to the local timezone). But I see that evince reports the correct times with respect to the timezone offset from the date string.

Note that, by default, pdfinfo will display the date timestamps according to the local timezone, but you can use its -rawdates flag to view the raw format.

Thanks for the hint about `astimezone()` method, by the way; I had somehow missed it while researching for a solution to this issue... > Would it not be easier to just store the timestamp in UTC instead of the local time? This would only inconvenience people reading the PDF in a text editor as any viewer would be able to display the timestamp in local time. I originally wanted to add the timezone info to `now()`, which `astimezone()` is appropriate for. Not knowing about `astimezone()` originally, I then opted to convert `now()` time to UTC. I agree that it would be easier to store the UTC timestamps by using the hardcoded offset in the string, as you did in your patch, as long as `--creationdate` and `--moddate` strings are converted before that point (see my edited previous comment). Not needing to extract the offset substring from the `%z` specifier would certainly be simpler to understand in the code. > I'm still having problems with this though with evince which which is not showing the correct time no matter what I do. I'm wondering if we are interpreting evince's document properties the same... With my merge request applied, I checked evince, and I see that it printed the timestamps according to the "raw" offset (not converted to the local timezone). But I see that evince reports the correct times with respect to the timezone offset from the date string. Note that, by default, `pdfinfo` will display the date timestamps according to the local timezone, but you can use its `-rawdates` flag to view the raw format.

Thank you!

I also added a bunch of test cases for this in b25429a4c1 so that this problem doesn't happen again.

Could you try running pytest with the latest git HEAD to see if the tests succeed for you as well?

Thank you! I also added a bunch of test cases for this in b25429a4c11bd62171066e8b9caf1b80c610ee58 so that this problem doesn't happen again. Could you try running `pytest` with the latest git HEAD to see if the tests succeed for you as well?

I also added a bunch of test cases for this in b25429a4c1 so that this problem doesn't happen again.

Could you try running pytest with the latest git HEAD to see if the tests succeed for you as well?

The new tests pass for me, though I had to apply a patch to override the default value for --pdfa so that img2pdf can find the expected color profile. This is on Arch Linux, and that sRGB.icc file is provided by the colord package.

diff --git a/src/img2pdf_test.py b/src/img2pdf_test.py
index f89fa74..e49bcb2 100755
--- a/src/img2pdf_test.py
+++ b/src/img2pdf_test.py
@@ -6957,7 +6957,7 @@ def test_faketime(tmp_path_factory, jpg_img, engine, testdata, timezone, pdfa):
     out_pdf = tmp_path_factory.mktemp("faketime") / "out.pdf"
     subprocess.check_call(
         ["env", f"TZ={timezone}", "faketime", "-f", testdata, img2pdfprog]
-        + (["--pdfa"] if pdfa else [])
+        + (["--pdfa", "/usr/share/color/icc/colord/sRGB.icc"] if pdfa else [])
         + [
             "--producer=",
             "--engine=" + engine,
@@ -7004,7 +7004,7 @@ def test_date(tmp_path_factory, jpg_img, engine, testdata, timezone, pdfa):
     out_pdf = tmp_path_factory.mktemp("faketime") / "out.pdf"
     subprocess.check_call(
         ["env", f"TZ={timezone}", img2pdfprog]
-        + (["--pdfa"] if pdfa else [])
+        + (["--pdfa", "/usr/share/color/icc/colord/sRGB.icc"] if pdfa else [])
         + [
             f"--moddate={testdata}",
             f"--creationdate={testdata}",

> I also added a bunch of test cases for this in b25429a4c11bd62171066e8b9caf1b80c610ee58 so that this problem doesn't happen again. > > Could you try running `pytest` with the latest git HEAD to see if the tests succeed for you as well? The new tests pass for me, though I had to apply a patch to override the default value for `--pdfa` so that img2pdf can find the expected color profile. This is on Arch Linux, and that `sRGB.icc` file is provided by the `colord` package. ```diff diff --git a/src/img2pdf_test.py b/src/img2pdf_test.py index f89fa74..e49bcb2 100755 --- a/src/img2pdf_test.py +++ b/src/img2pdf_test.py @@ -6957,7 +6957,7 @@ def test_faketime(tmp_path_factory, jpg_img, engine, testdata, timezone, pdfa): out_pdf = tmp_path_factory.mktemp("faketime") / "out.pdf" subprocess.check_call( ["env", f"TZ={timezone}", "faketime", "-f", testdata, img2pdfprog] - + (["--pdfa"] if pdfa else []) + + (["--pdfa", "/usr/share/color/icc/colord/sRGB.icc"] if pdfa else []) + [ "--producer=", "--engine=" + engine, @@ -7004,7 +7004,7 @@ def test_date(tmp_path_factory, jpg_img, engine, testdata, timezone, pdfa): out_pdf = tmp_path_factory.mktemp("faketime") / "out.pdf" subprocess.check_call( ["env", f"TZ={timezone}", img2pdfprog] - + (["--pdfa"] if pdfa else []) + + (["--pdfa", "/usr/share/color/icc/colord/sRGB.icc"] if pdfa else []) + [ f"--moddate={testdata}", f"--creationdate={testdata}", ```

Thank you for finding this! But this is the wrong fix. The file /usr/share/color/icc/colord/sRGB.icc also exists in Debian but is different from /usr/share/color/icc/sRGB.icc (they come from different source packages with different licenses). I think the correct solution is either:

let the --pdfa option try both locations and choose the first that exist by default or
maybe a package in arch linux also ships /usr/share/color/icc/sRGB.icc and you just have it not installed -- can you check?

Thank you for finding this! But this is the wrong fix. The file `/usr/share/color/icc/colord/sRGB.icc` also exists in Debian but is different from `/usr/share/color/icc/sRGB.icc` (they come from different source packages with different licenses). I think the correct solution is either: - let the `--pdfa` option try both locations and choose the first that exist by default or - maybe a package in arch linux also ships /usr/share/color/icc/sRGB.icc and you just have it not installed -- can you check?

maybe a package in arch linux also ships /usr/share/color/icc/sRGB.icc and you just have it not installed -- can you check?

I discovered that /usr/share/color/icc/sRGB.icc is provided by icc-profiles-free in Debian, and that file appears to be sourced from the upstream openicc project, if I am reading the packaging metadata correctly.

In the Arch Linux AUR, there is an openicc package, but the sRGB.icc is installed to /usr/share/color/icc/OpenICC/sRGB.icc instead.

I checked Fedora too, and it provides an icc-profiles-openicc package that installs /usr/share/color/icc/OpenICC/sRGB.icc as well.

> - maybe a package in arch linux also ships /usr/share/color/icc/sRGB.icc and you just have it not installed -- can you check? I discovered that `/usr/share/color/icc/sRGB.icc` is provided by `icc-profiles-free` in Debian, and that file appears to be sourced from the upstream openicc project, if I am reading the [packaging metadata](https://salsa.debian.org/debian/icc-profiles-free/-/tree/master/debian) correctly. In the Arch Linux AUR, there is an [openicc package](https://aur.archlinux.org/packages/openicc), but the sRGB.icc is installed to `/usr/share/color/icc/OpenICC/sRGB.icc` instead. I checked Fedora too, and it provides an [icc-profiles-openicc](https://src.fedoraproject.org/rpms/icc-profiles-openicc) package that installs `/usr/share/color/icc/OpenICC/sRGB.icc` as well.

nice, thank you for that research!

Then I propose to check for the existance of profiles in the following order and use the first one that exists as the default:

/usr/share/color/icc/sRGB.icc
/usr/share/color/icc/OpenICC/sRGB.icc
/usr/share/color/icc/colord/sRGB.icc

nice, thank you for that research! Then I propose to check for the existance of profiles in the following order and use the first one that exists as the default: 1. `/usr/share/color/icc/sRGB.icc` 2. `/usr/share/color/icc/OpenICC/sRGB.icc` 3. `/usr/share/color/icc/colord/sRGB.icc`

@phmccarty would this patch fix your problem:

diff --git a/src/img2pdf.py b/src/img2pdf.py
index 6c58ea5..54701c9 100755
--- a/src/img2pdf.py
+++ b/src/img2pdf.py
@@ -3766,6 +3766,17 @@ def gui():
     app.mainloop()
 
 
+def get_default_icc_profile():
+    for profile in [
+        "/usr/share/color/icc/sRGB.icc",
+        "/usr/share/color/icc/OpenICC/sRGB.icc",
+        "/usr/share/color/icc/colord/sRGB.icc",
+    ]:
+        if os.path.exists(profile):
+            return profile
+    return "/usr/share/color/icc/sRGB.icc"
+
+
 def get_main_parser():
     rendered_papersizes = ""
     for k, v in sorted(papersizes.items()):
@@ -4036,10 +4047,13 @@ RGB.""",
     outargs.add_argument(
         "--pdfa",
         nargs="?",
-        const="/usr/share/color/icc/sRGB.icc",
+        const=get_default_icc_profile(),
         default=None,
         help="Output a PDF/A-1b compliant document. By default, this will "
-        "embed /usr/share/color/icc/sRGB.icc as the color profile.",
+        "embed either /usr/share/color/icc/sRGB.icc, "
+        "/usr/share/color/icc/OpenICC/sRGB.icc or "
+        "/usr/share/color/icc/colord/sRGB.icc as the color profile, whichever "
+        "is found to exist first.",
     )
 
     sizeargs = parser.add_argument_group(

@phmccarty would this patch fix your problem: ```patch diff --git a/src/img2pdf.py b/src/img2pdf.py index 6c58ea5..54701c9 100755 --- a/src/img2pdf.py +++ b/src/img2pdf.py @@ -3766,6 +3766,17 @@ def gui(): app.mainloop() +def get_default_icc_profile(): + for profile in [ + "/usr/share/color/icc/sRGB.icc", + "/usr/share/color/icc/OpenICC/sRGB.icc", + "/usr/share/color/icc/colord/sRGB.icc", + ]: + if os.path.exists(profile): + return profile + return "/usr/share/color/icc/sRGB.icc" + + def get_main_parser(): rendered_papersizes = "" for k, v in sorted(papersizes.items()): @@ -4036,10 +4047,13 @@ RGB.""", outargs.add_argument( "--pdfa", nargs="?", - const="/usr/share/color/icc/sRGB.icc", + const=get_default_icc_profile(), default=None, help="Output a PDF/A-1b compliant document. By default, this will " - "embed /usr/share/color/icc/sRGB.icc as the color profile.", + "embed either /usr/share/color/icc/sRGB.icc, " + "/usr/share/color/icc/OpenICC/sRGB.icc or " + "/usr/share/color/icc/colord/sRGB.icc as the color profile, whichever " + "is found to exist first.", ) sizeargs = parser.add_argument_group( ```

@josch Yep, that patch works for me, thanks!

nice! fixed in 29921eeabd

please file issues for any other problem you find

thanks!

nice! fixed in 29921eeabde1a2f9eb31f1383e5a820d9772c333 please file issues for any other problem you find thanks!

Labels Milestones

PDF timestamp handling issue #155