CHANGELOG.md

@@ -1,8 +1,3 @@
-1.5.2 (2024-06-26)
-------------------
-
- - mmdebstrap-autopkgtest-build-qemu produces bit-by-bit reproducible output
-
 1.5.1 (2024-06-03)
 ------------------
 

README.md

@@ -158,19 +158,16 @@ Contributors
 
  - Johannes Schauer Marin Rodrigues (main author)
  - Helmut Grohne
- - Jochen Sprickerhof
  - Gioele Barabucci
  - Benjamin Drung
+ - Jochen Sprickerhof
  - Josh Triplett
  - Konstantin Demin
- - Chris Hofstaedtler
- - Colin Watson
  - David Kalnischkies
  - Emilio Pozuelo Monfort
  - Francesco Poli
  - Jakub Wilk
  - Joe Groocock
- - Max-Julian Pogner
  - Nicolas Vigier
  - Raul Tambre
  - Steve Dodd

mmdebstrap

@@ -23,7 +23,7 @@
 use strict;
 use warnings;
 
-our $VERSION = '1.5.2';
+our $VERSION = '1.5.1';
 
 use English;
 use Getopt::Long;
@@ -4293,8 +4293,8 @@ sub get_sourceslist_by_suite {
         # the security mirror changes, starting with bullseye
         # https://lists.debian.org/87r26wqr2a.fsf@43-1.org
         my $bullseye_or_later = 0;
-        if (any { $_ eq $suite }
-            ('oldstable', 'stable', 'bullseye', 'bookworm', 'trixie')) {
+        if (any { $_ eq $suite } ('oldstable', 'stable', 'bullseye', 'bookworm', 'trixie'))
+        {
             $bullseye_or_later = 1;
         }
         my $distro_info = '/usr/share/distro-info/debian.csv';
@@ -6973,29 +6973,13 @@ The above uses C<--map-auto> to map the block of user/group ids for the
 effective user/group to a block starting at user/group ID 0. We also want to
 map the current effective user/group ID into the subuid/subgid range using
 C<--map-user> and C<--map-group>, respectively. But if that uid/gid overlaps
-with the respective range, a "hole" will be removed from the mapping by the
-L<unshare(1)> utility and the remaining uid/gid values will get shifted. Thus,
-we map the current effective user/group ID to the highest possible uid/gid,
-putting them at the end. Since that means that the user/group will be "nobody"
-and not "root" inside the namespace, C<--keep-caps> propagate permitted
-capabilities into the ambient set and thus give the user C<CAP_DAC_OVERRIDE>
-and other capabilities that it would've had. The following does B<NOT> work:
-
-    $ unshare --map-root-user --map-auto ...            # or equavalient:
-    $ unshare --map-user=0 --map-group=0 --map-users=auto --map-groups=auto ...
-
-Using the format of L<lxc-usernsexec(1)>, these options will result in the
-following mapping:
-
-    0:$UID:1 + 1:$SUBUIDBASE:65535
-
-So a hole is punched in the automatically mapped range at the very beginning to
-make space for the user id of the current user as root inside the namespace and
-the remaining uids are shifted. With C<--map-user=65536> the mapping is as
-follows and places the current user at the end of the range, preventing any
-shifting:
-
-    0:$SUBUIDBASE:65536 + 65536:$UID:1
+with the respective range, a "hole" will be removed from the mapping and the
+remaining uid/gid values will get shifted. Thus, we map the current effective
+user/group ID to the highest possible uid/gid, putting them at the end. Since
+that means that the user/group will be "nobody" and not "root" inside the
+namespace, C<--keep-caps> propagate permitted capabilities into the ambient set
+and thus give the user C<CAP_DAC_OVERRIDE> and other capabilities that it
+would've had.
 
 Lastly, if you don't mind using superuser privileges and have systemd-nspawn
 available and you know your subuid/subgid offset (100000 in this example):

mmdebstrap-autopkgtest-build-qemu

@@ -29,11 +29,10 @@ B<mmdebstrap-autopkgtest-build-qemu> [I<OPTIONS>] B<--boot>=B<efi> I<RELEASE> I<
 B<mmdebstrap-autopkgtest-build-qemu> is a mostly compatible drop-in replacement
 for L<autopkgtest-build-qemu(1)> with two main differences: Firstly, it uses
 L<mmdebstrap(1)> instead of L<vmdb2(1)> and thus is able to create QEMU disk
-images without requiring superuser privileges and with bit-by-bit reproducible
-output. Secondly, it uses L<systemd-boot(7)> and thus only supports booting via
-EFI. For architectures for which L<autopkgtest-virt-qemu(1)> does not default
-to EFI booting you must pass B<--boot=efi> when invoking the autopkgtest virt
-backend.
+images without requiring superuser privileges.  Secondly, it uses
+L<systemd-boot(7)> and thus only supports booting via EFI. For architectures
+for which L<autopkgtest-virt-qemu(1)> does not default to EFI booting you must
+pass B<--boot=efi> when invoking the autopkgtest virt backend.
 
 =head1 POSITIONAL PARAMETERS
 
@@ -111,12 +110,6 @@ all path components or by creating the image in a world-readable directory like
 Make sure to add B<--boot=efi> to both the B<mmdebstrap-autopkgtest-build-qemu>
 as well as the B<autopkgtest-virt-qemu> invocation.
 
-Create bit-by-bit reproducible images from a given snapshot.d.o timestamp.
-
-    SOURCE_DATE_EPOCH=1612543740 mmdebstrap-autopkgtest-build-qemu --boot=efi \
-        --mirror=http://snapshot.debian.org/archive/debian/20210205T164900Z/ \
-        unstable /path/to/debian-unstable.img
-
 =head1 SEE ALSO
 
 L<autopkgtest-build-qemu(1)>, L<autopkgtest-virt-qemu(1)>, L<mmdebstrap(1)>, L<autopkgtest(1)>
@@ -280,7 +273,7 @@ test_installed() {
 	fi
 }
 
-for pkg in autopkgtest dosfstools e2fsprogs fdisk mount mtools passwd uidmap libarchive13; do
+for pkg in autopkgtest dosfstools e2fsprogs fdisk mount mtools passwd uidmap; do
 	test_installed "$pkg"
 done
 
@@ -327,24 +320,33 @@ FAT_SIZE_SECTORS=$((1024*254))
 #  - snapshots and overlays work just as well with raw images
 #  - users who prefer qcow2 get to choose to run it themselves with their own
 #    custom options like compression
+#
+# --map-users=auto --map-user=0     => 0:$UID:1 + 1:$SUBUIDBASE:65535
+# --map-users=auto --map-user=65536 => 0:$SUBUIDBASE:65536 + 65536:$UID:1
+#
+# Make the image writeable to the first subgid. mmdebstrap will map this gid to
+# the root group. unshare instead will map the current gid to 0 and the first
+# subgid to 1. Therefore mmdebstrap will be able to write to the image.
+rm -f "$IMAGE"
+: >"$IMAGE"
+unshare --map-user=0 --map-group=0 --map-groups=auto chown 0:1 "$IMAGE"
+chmod 0660 "$IMAGE"
+
+# Make sure that the unshared user is able to access the file.
+# Alternatively to using /sbin/mkfs.ext4 could use --format=ext2 which would
+# add an extra copy operation and come with the limitations of ext2.
+# Another solution: https://github.com/tytso/e2fsprogs/pull/118
+if ! mmdebstrap --unshare-helper touch "$IMAGE"; then
+	die "$IMAGE cannot be accessed by the unshared user -- either make all path components up to the image itself world-executable or place the image into a world-readable path like /tmp"
+fi
 
 set -- \
 	--mode=unshare \
-	--format=tar \
 	--variant=important \
 	--architecture="$ARCHITECTURE"
 
-case $MIRROR in http://snapshot.debian.org/archive/*|https://snapshot.debian.org/archive/*)
-	set -- "$@" --aptopt='Acquire::Check-Valid-Until "false"';;
-esac
-
-EXT_FEATURES=
-if test "$RELEASE" = jessie; then
-	set -- "$@" --keyring=/usr/share/keyrings/debian-archive-removed-keys.gpg
-	set -- "$@" --aptopt='Apt::Key::gpgvcommand "/usr/libexec/mmdebstrap/gpgvnoexpkeysig"'
+test "$RELEASE" = jessie &&
 	set -- "$@" --hook-dir=/usr/share/mmdebstrap/hooks/jessie-or-older
-	EXT_FEATURES="^metadata_csum,^metadata_csum_seed,^orphan_file"
-fi
 
 set -- "$@" \
 	"--include=init,$LINUXIMAGE,python3" \
@@ -362,35 +364,30 @@ if test -n "$SCRIPT"; then
 		'--customize-hook=rm -f "$1/userscript"'
 fi
 
+EXT4_OFFSET_BYTES=$(( (FAT_OFFSET_SECTORS + FAT_SIZE_SECTORS) * 512))
+EXT4_OPTIONS="offset=$EXT4_OFFSET_BYTES,assume_storage_prezeroed=1"
+
+# the --no-mtab option to mount is a workaround for https://github.com/util-linux/util-linux/issues/2981
+# revert 8c0ddc32660ca4e98c988966251f9c05d6bcccef once it is no longer needed
 set -- "$@" \
 	"--customize-hook=download vmlinuz '$WORKDIR/kernel'" \
 	"--customize-hook=download initrd.img '$WORKDIR/initrd'" \
+	'--customize-hook=mount --no-mtab --bind "$1" "$1/mnt"' \
+	'--customize-hook=mount --no-mtab --bind "$1/mnt/mnt" "$1/mnt/dev"' \
+	'--customize-hook=/sbin/mkfs.ext4 -d "$1/mnt" -L autopkgtestvm -E '"'$EXT4_OPTIONS' '$IMAGE' '$SIZE'" \
+	'--customize-hook=umount --lazy --no-mtab "$1/mnt/dev"' \
+	'--customize-hook=umount --lazy --no-mtab "$1/mnt"' \
 	"$RELEASE" \
-	-
+	/dev/null
 
 test -n "$MIRROR" && set -- "$@" "$MIRROR"
 test -n "$KEYRING" && set -- "$@" "--keyring=$KEYRING"
 
-echo "+ mmdebstrap $*" >&2
-# https://github.com/koalaman/shellcheck/issues/2555
-# shellcheck disable=SC3040
-set -o pipefail
-mmdebstrap "$@" | {
-	set -- -t ext4 -L autopkgtestvm -d -
-	if test -n "$EXT_FEATURES"; then
-		set -- "$@" -O "$EXT_FEATURES"
-	fi
-	EXTOPTS="offset=$(( (FAT_OFFSET_SECTORS + FAT_SIZE_SECTORS) * 512))"
-	if test -n "${SOURCE_DATE_EPOCH:-}"; then
-		uuid="$(uuidgen --sha1 --namespace="$(uuidgen --sha1 --namespace='@dns' --name mister-muffin.de)" --name "$SOURCE_DATE_EPOCH")"
-		set -- "$@" -U "$uuid"
-		EXTOPTS="$EXTOPTS,hash_seed=$uuid"
-	fi
-	set -- "$@" -E "$EXTOPTS" "$IMAGE" "$SIZE"
-	echo "+ mke2fs $*" >&2
-	/sbin/mke2fs "$@"
-}
+echo "mmdebstrap $*"
+mmdebstrap "$@" || die "mmdebstrap failed"
 
+unshare -U -r --map-groups=auto chown 0:0 "$IMAGE"
+chmod "$(printf %o "$(( 0666 & ~0$(umask) ))")" "$IMAGE"
 
 echo "root=LABEL=autopkgtestvm rw console=ttyS0" > "$WORKDIR/cmdline"
 
@@ -464,5 +461,4 @@ if test -n "$VMFPKG" && test "$(dpkg-query -f '${db:Status-Status}' -W "$VMFPKG"
 	echo "I: you might need to install $VMFPKG to use this image with autopkgtest-virt-qemu" >&2
 fi
 
-echo "I: SUCCESS! Your new image can be found here: $IMAGE" >&2
-echo "I: Don't forget to pass --boot=efi when running autopkgtest-virt-qemu with this image" >&2
+echo "I: don't forget to pass --boot=efi when running autopkgtest-virt-qemu with this image" >&2

tests/check-against-debootstrap-dist

@@ -157,44 +157,24 @@ if [ "{{ VARIANT }}" = "-" ] && [ "{{ DIST}}" = oldstable ]; then
 fi
 
 for log in faillog lastlog; do
-	f1="/tmp/debian-{{ DIST }}-debootstrap/var/log/$log"
-	f2="/tmp/debian-{{ DIST }}-mm/var/log/$log"
-	# skip cmp if file is absent in both chroots
-	if [ ! -e "$f1" ] && [ ! -e "$f2" ]; then
-		continue
-	fi
-	if ! cmp "$f1" "$f2" >&2;then
+	if ! cmp /tmp/debian-{{ DIST }}-debootstrap/var/log/$log /tmp/debian-{{ DIST }}-mm/var/log/$log >&2;then
 		# if the files differ, make sure they are all zeroes
-		cmp -n "$(stat -c %s "$f1")" "$f1" /dev/zero >&2
-		cmp -n "$(stat -c %s "$f2")" "$f2" /dev/zero >&2
+		cmp -n "$(stat -c %s "/tmp/debian-{{ DIST }}-debootstrap/var/log/$log")" "/tmp/debian-{{ DIST }}-debootstrap/var/log/$log" /dev/zero >&2
+		cmp -n "$(stat -c %s "/tmp/debian-{{ DIST }}-mm/var/log/$log")" "/tmp/debian-{{ DIST }}-mm/var/log/$log" /dev/zero >&2
 		# then delete them
-		rm "$f1" "$f2"
+		rm /tmp/debian-{{ DIST }}-debootstrap/var/log/$log /tmp/debian-{{ DIST }}-mm/var/log/$log
 	fi
 done
 
+# the order in which systemd and cron get installed differ and thus the order
+# of lines in /etc/group and /etc/gshadow differs
 if [ "{{ VARIANT }}" = "-" ]; then
-	# the order in which systemd and cron get installed differ and thus the order
-	# of lines in /etc/group and /etc/gshadow differs
 	for f in group group- gshadow gshadow-; do
 		for d in mm debootstrap; do
 			sort /tmp/debian-{{ DIST }}-$d/etc/$f > /tmp/debian-{{ DIST }}-$d/etc/$f.bak
 			mv /tmp/debian-{{ DIST }}-$d/etc/$f.bak /tmp/debian-{{ DIST }}-$d/etc/$f
 		done
 	done
-	# the order in which systemd and passwd get installed differ and thus
-	# the order of lines in /etc/shadow and /etc/shadow- differs
-	for f in shadow shadow-; do
-		for d in mm debootstrap; do
-			sort /tmp/debian-{{ DIST }}-$d/etc/$f > /tmp/debian-{{ DIST }}-$d/etc/$f.bak
-			mv /tmp/debian-{{ DIST }}-$d/etc/$f.bak /tmp/debian-{{ DIST }}-$d/etc/$f
-		done
-	done
-	# and since the order was different, ignore the *- files
-	for f in shadow- passwd-; do
-		for d in mm debootstrap; do
-			rm /tmp/debian-{{ DIST }}-$d/etc/$f
-		done
-	done
 fi
 
 # since debootstrap 1.0.133 there is no tzdata in the buildd variant and thus