test.sh fails under Fedora #56
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
By Georg Sauthoff on 2019-02-18T19:22:31.870Z
So I tried to bump the Fedora package to img2pdf 0.3.3 and integrate the additional test.sh suite.
Unfortunately, test.sh fails quite early - e.g. on Fedora 29/x86_64:
See also: https://copr-be.cloud.fedoraproject.org/results/gsauthof/fedora/fedora-29-x86_64/00859463-python-img2pdf/builder-live.log
I have to say that the way test.sh is written doesn't really lead to any actionable diagnostics. I mean even when I run the script with
set -xthe failing script doesn't tell me which files exactly produce checksum errors. And even if it would, the reference files don't seem to be part of the repository.Also, the script fails on the first error instead of running all tests and then displaying a summary of failed/succeeded test cases.
When I am in the situation writing some tests that call a bunch of command utilities I usually use Python as better shell script replacement. That means pytest as convenient test library and the subprocess package for executing the commands. With pytest its quite easy to parametrize test cases and by default output is only displayed for failing tests which keeps the signal to noise ratio low.
By josch on 2019-02-20T18:49:19.456Z
Hi Georg, sorry for the late reply. I'm currently on holidays. I'll tend to this bug after I'm back in two weeks.
Indeed, the script fails on first error but in this case, it could not possibly continue because what fails is the test setup itself. In the beginning, the script uses imagemagick to generate some fake input for img2pdf. That's why there are no "reference files" -- they are generated on the fly. The checksum errors mean, that imagemagick produced some unexpected output. The test is aborted because imagemagick produced unexpected input files. Producing these files has nothing to do with img2pdf as only imagemagick is involved. I'll have to look into it to see why imagemagick decided to not produce the same image on Fedora as it does on Debian.
I also agree, that this testing script could've been done much better and that's why I would certainly welcome pull requests that improve the situation. If somebody is willing to help, please open a separate pull requests to discuss your ideas. Thanks!
By josch on 2019-03-09T16:27:28.607Z
Okay, so today I had some time to spend on this.
Basically what test.sh does is to first generate some dummy test input images and then run the tests on them. To make sure that these dummy images are generated correctly, their md5sum is checked. Funnily it seems that on Fedora x86_64, the generated files are not the same as imagemagick produces them on Debian amd64. This is weird and should not happen, especially because the imagemagick version does not differ that much (6.9.10.23 in Debian versus 6.9.9.38 in Fedora).
So I questioned the whole idea of using imagemagick to produce the test input in the first place and ran the following code on 12 architectures:
On Debian amd64 this yields the following md5sums:
On the other architectures, this is not always so. I made a table to visualize the differences:
A green heart means the same md5sum as on amd64. Differently colored hearts signify a different md5sum. There are two interesting observations:
On hurd-i386, alpha.png is different but the checksum of all the other files matches. This is odd because alpha.png is the input from which everything else gets generated.
All other differences come from palette8.png which is also weird because the call to
convertis the same as for palette1.png, palette2.png and palette4.png, just with a different-colorsvalue. Why is the algorithm used by imagemagick doing different things depending on the architecture and only if 256 colors are chosen?My next step would be to run the whole thing on Fedora. But maybe you @gms have some free time and run above few lines on your box and then write here which files differed from the amd64 md5sums I posted above?
I might need to replace imagemagick by another software that is actually reproducible independent on the architecture the algorithm is running on.
By josch on 2019-03-09T18:05:13.385Z
Okay, nevermind, on Fedora 29 x86_64, every single png image is different. Here is the diff image created by imagemagick compare of alpha.png:
No idea whether this is due to the difference in imagemagick version or some other reasons.
I'll just replace imagemagick with something else that is guaranteed to be stable across versions and architectures.
By josch on 2019-03-12T02:11:31.896Z
The issue should now be fixed in commit
d1f101c36a. I replaced imagemagick by a python script using numpy and scipy to create the png input and it produces bit-by-bit identical output on all Debian architectures and Fedora 29 x86_64.By josch on 2019-03-12T02:11:32.816Z
Status changed to closed
By Georg Sauthoff on 2019-03-19T19:10:33.276Z
Ok, I'll prepare a new img2pdf Fedora package when you cut a new PyPi release.
By josch on 2020-06-05T23:21:41.243Z
commit
a84a1b8480now adds over 4600 lines of pytest -- i hope that is useful for somebody