CMYK.tif_*_pdfrw tests fail on Aarch64/Fedora 29 #51
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
By Georg Sauthoff on 2018-11-25T14:28:28.406Z
With the newest release, 2 tests fail on Aarch64/Fedora 29:
see also the complete build log
On x86_64 (also Fedora 29), they do succeed.
By Georg Sauthoff on 2018-12-08T20:58:52.761Z
Sure, on x86-64/Fedora 29 I get the same hash:
On aarch64 I get a different one:
By Georg Sauthoff on 2018-12-08T21:14:54.839Z
Hm, it really looks that the difference is due to different zlib compress output.
This
yields the same hash on both x86-64 and aarch64:
By josch on 2018-12-08T21:21:36.172Z
Did you try to compress something that isn't the image? Like this:
Does that result in the same output on both platforms? It would be weird if the plain CMYK data would somehow be special to the zlib compression.
Did you try to redirect the zlib compressed output on aarch64 to a file? Is it really zlib compression that comes out or something different?
In any case, this doesn't look like a bug in img2pdf anymore. :)
By Georg Sauthoff on 2018-12-08T21:36:08.512Z
It really depends on whether zlib compression is specified to yield bit-identical results with different implementations.
After skimming through RFC 1951 - I think that 2 conforming implementations are allowed to produce to different bitstreams for the same input. For example, there is a paragraph on some optional lazy matching to improve compression.
Of course, a zlib bug could also cause differences - I'll try to exclude this possibility with another test.
By Georg Sauthoff on 2018-12-08T22:03:41.495Z
Ok, so the aarch64 zlib compression is fine:
This
yields
True
on both aarch64 and x86-64.I was also successful to decompress the aarch64 compressed data under x86-64. The result is identical to the input.
Regarding why you can't reproduce this on debian aarch64 - the Fedora zlib version is perhaps more recent and/or has some different optimizations enabled - compiler switches or even specific SIMD optimized versions.
By josch on 2018-12-08T22:05:36.035Z
Maybe you should bring this up to your zlib maintainers. In my humble opinion I think it's fair that one can assume identical output of a compression library irrespective of the architecture the compressor is running on.
By Georg Sauthoff on 2018-12-08T22:19:04.724Z
Probably all compression algorithms are specified in a similar way, i.e. not to require bit-identical output for compression but
decompress(compress(x))==x
.The motivation is perhaps similar to programming languages, where a specifications usually allows for some implementation defined and undefined behavior - to allow for portability and optimization opportunities.
Thus, I would say that it's futile to argue with the libzlib author - especially given how RFC 1951 is written.
By josch on 2018-12-08T22:28:04.773Z
What's your zlib version? We have 1.2.11 in Debian.
By josch on 2018-12-08T22:38:49.441Z
That also seems to be the most recent version with what upstream is concerned.
It thus seems that you are doing something different in Fedora.
I'm not saying that you should go to the zlib upstream maintainer with this. But go to the zlib Fedora package maintainer and inform them that the behaviour of their package differs with what the Debian package does and with what the package does on other architectures. The output might still be valid zlib format but you will probably agree that mmdebstrap might not be the last software that expects bit-by-bit identical zlib output across all architectures.
By Georg Sauthoff on 2018-12-09T09:58:27.182Z
As I wrote, there are many possibilities to do things differently when building zlib:
Perhaps out-of-scope for this bug - but why does mmdebstrap expect bit-identical zlib output accross all architectures?
So far, img2pdf is the only software I know that requires bit-identical zlib compress output across architectures and zlib releases. And I don't see a strong requirement for it. When doing reproducible builds, I expect that you don't change architectures/zlib implementations during one reproducible build.
I've posted the above observations to the Fedora devel list, let's see what other developers think:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/R7GD4L5Z6HELCDAL2RDESWR2F3ZXHWVX/
By josch on 2018-12-09T11:46:09.982Z
The reason for expecting bit-identical zlib output is simple: it is much easier to check whether img2pdf indeed produced the expected output when the files are identical. I cannot think of any simple test that would indeed make sure that the output is a valid pdf document but does not expect bit-identical-output compared to a known-good pdf. Any such test would require a pdf parser that would be able to extract the payload, decompress the payload and check whether it decompresses to the same output. Any implementation of that is again prone to errors which makes it a bad fit for such kind of test.
Thanks for bringing this up to the Fedora list and please keep me updated about the result of the discussion.
As far as I see it and since Fedora so far seems to be the only distribution with this problem, if you don't want to change zlib on aarch64, then you would have to patch img2pdf to either not perform this particular check or patch it so that it indeed decodes the pdf before comparing it.
By Georg Sauthoff on 2018-12-09T17:51:35.482Z
With
qpdf
it is relatively straight-forward to recompress all streams - the idea then is to apply this normalization on both PDFs and then compare the normalized PDFs.Something like this:
Tested it and it works on aarch64/x86-64
By josch on 2018-12-10T06:23:16.190Z
Yes, there exist a number of these commands.
mutool clean
is another that is able to uncompress a pdf. But they are not the right tool for a test case. The PDF Format is pretty lax about how parsers interpret it. Try changing the/Length
to 980 or increase one of the xref table entries by one.qpdf
will still be able to make sense of the pdf and convert it into a fixed version. But this will hide implementation problems. In this case: off-by one errors which I've already had in the past. And these errors would not be caught by the other tests because the CMYK test is special in that CMYK images are the only ones that are saved as raw CYMK instead of applying the PNG Paeth filter beforehand. So it would be wrong to say: If the two PDFs are not bit-by-bit equal, just fix them beforehand using qpdf or mutool. These tools do fix problems and would make the test pointless.By Georg Sauthoff on 2018-12-12T20:22:46.300Z
Ok, qpdf does too much/isn't pedantic enough.
I have a simple patch that just recompresses the last stream in a PDF and adjusts the lengths, before comparing the PDFs. Because it copy'n'pastes as much as possible from the original PDF and the lengths adjustments are relative, a
\Length
value that was 1 byte too short should also be 1 byte too short after the recompressing - even if the size of the compressed stream changes:What do you think?
There were some replies to my Fedora mailinglist posting and the consensus basically is:
By josch on 2018-12-12T21:28:03.518Z
Cool, thanks for the patch! But I don't think you should bother with setting the
Length
attribute to a new value because if you want the output to be a valid PDF then you need more than just setting the Length. You would have to change the offsets in the xref table at the end as well, for example.As for the Fedora discussion: I also agree that one shouldn't rely on bit-identical compression between different zlib versions or implementation.
Before applying your patch, maybe lets see if Fedora reverts its Fedora-specific downstream changes to the aarch64 package.
As for now, you might want to just disable this one test-case or even apply a Fedora-specific change to the package in the manner you suggested.
Thanks!
By Georg Sauthoff on 2018-12-12T21:33:58.460Z
Well, the code also rewrites the
startxref
offset. And since the recompressed stream is in the last object we don't have to adjust any offsets in thexref
table.By Georg Sauthoff on 2018-12-15T14:32:50.380Z
Well, Fedora doesn't revert those Arm patches. Apparently, the performance benefit is just too good.
I've just submitted a package update to Fedora that includes the above patch.
By josch on 2018-12-24T17:14:06.442Z
Status changed to closed by commit
2faeb2005d
By josch on 2019-01-07T09:52:40.073Z
I just published a new release 0.3.3 on pypi which includes your patch from above -- thanks!
It also contains a much more extensive testsuite called
test.sh
which you could run in your builds if you like. It was only tested under Debian though, so please file a bug if you encounter problems on Fedora. :)By Georg Sauthoff on 2020-04-30T18:53:15.119Z
Mentioned in issue #75