initial commit

This commit is contained in:
Johannes 'josch' Schauer 2018-09-18 11:20:24 +02:00
commit 9ed4c65e35
Signed by untrusted user: josch
GPG key ID: F2CBA5C78FBD83E1
4 changed files with 1855 additions and 0 deletions

103
README.md Normal file
View file

@ -0,0 +1,103 @@
mmdebstrap
==========
An alternative to debootstrap which uses apt internally and is thus able to use
more than one mirror and resolve more complex dependencies.
Usage
-----
Use like debootstrap:
sudo mmdebstrap unstable ./unstable-chroot
Without superuser privileges:
mmdebstrap unstable unstable-chroot.tar
With complex apt options:
cat /etc/apt/sources.list | mmdebstrap > unstable-chroot.tar
The sales pitch in comparison to debootstrap
--------------------------------------------
Summary:
- more than one mirror possible
- security and updates mirror included for Debian stable chroots
- 2-3 times faster
- chroot with apt in 11 seconds
- gzipped tarball with apt is 27M small
- bit-by-bit reproducible output
- unprivileged operation using Linux user namespaces, fakechroot or proot
- can operate on filesystems mounted with nodev
- foreign architecture chroots with qemu-user
The author believes that a chroot of a Debian stable release should include the
latest packages including security fixes by default. This has been a wontfix
with debootstrap since 2009 (See #543819 and #762222). Since mmdebstrap uses
apt internally, support for multiple mirrors comes for free and stable or
oldstable **chroots will include security and updates mirrors**.
A side-effect of using apt is being **2-3 times faster** than debootstrap. The
timings were carried out on a laptop with an Intel Core i5-5200U.
| variant | mmdebstrap | debootstrap |
| ------- | ---------- | ------------ |
| minbase | 25.25 s | 51.47 s |
| buildd | 30.99 s | 59.38 s |
| - | 29.85 s | 127.18 s |
Apt considers itself an `Essential: yes` package. This feature allows one to
create a chroot containing just the `Essential: yes` packages and apt (and
their hard dependencies) in **just 11 seconds**.
If desired, a most minimal chroot with just the `Essential: yes` packages and
their hard dependencies can be created with a gzipped tarball size of just 34M.
By using dpkg's `--path-exclude` option to exclude documentation, even smaller
gzipped tarballs of 21M in size are possible. If apt is included, the result is
a **gzipped tarball of only 27M**.
These small sizes are also achieved because apt caches and other cruft is
stripped from the chroot. This also makes the result **bit-by-bit
reproducible** if the `$SOURCE_DATE_EPOCH` environment variable is set.
The author believes, that it should not be necessary to have superuser
privileges to create a file (the chroot tarball) in one's home directory. If
mmdebstrap is run by an unprivileged user, either Linux user namespaces,
fakechroot or proot are used to create a chroot tarball. Debootstrap supports
fakechroot but will not create a tarball with the right permissions by itself.
Support for Linux user namespaces and proot is missing (see bugs #829134 and
#698347, respectively).
When creating a chroot tarball with debootstrap, the temporary chroot directory
cannot be on a filesystem that has been mounted with nodev. In unprivileged
mode, mknod is never used, which means that /tmp can be used as a temporary
directory location even if if it's mounted with nodev as a security measure.
If the chroot architecture cannot be executed by the current machine, qemu-user
is used to allow one to create a foreign architecture chroot.
Limitations in comparison to debootstrap
----------------------------------------
Debootstrap supports creating a Debian chroot on non-Debian systems but
mmdebstrap requires apt.
There is no `SCRIPT` argument.
There is no `--second-stage` option.
Tests
=====
The script `test.sh` compares the output of mmdebstrap with debootstrap in
several scenarios. Since debootstrap needs superuser privileges, `test.sh`
needs `sudo` to run.
Bugs
====
mmdebstrap has bugs. Report them here:
https://gitlab.mister-muffin.de/josch/mmdebstrap/issues

99
make_mirror.sh Executable file
View file

@ -0,0 +1,99 @@
#!/bin/sh
set -eu
mirrordir="./mirror"
mirror="http://deb.debian.org/debian"
nativearch=$(dpkg --print-architecture)
components=main
if [ -e "$mirrordir/dists/unstable/Release" ]; then
http_code=$(curl --output /dev/null --silent --location --head --time-cond "$mirrordir/dists/unstable/Release" --write-out '%{http_code}' "$mirror/dists/unstable/Release")
case "$http_code" in
200) ;; # need update
304) echo up-to-date; exit 0;;
*) echo unexpected status: $http_code; exit 1;;
esac
fi
for dist in stable testing unstable; do
rootdir=$(mktemp --directory)
for p in /etc/apt/apt.conf.d /etc/apt/sources.list.d /etc/apt/preferences.d /var/cache/apt /var/lib/apt/lists/partial /var/lib/dpkg; do
mkdir -p "$rootdir/$p"
done
cat << END > "$rootdir/etc/apt/apt.conf"
Apt::Architecture "$nativearch";
Dir::Etc "$rootdir/etc/apt";
Dir::State "$rootdir/var/lib/apt";
Dir::Cache "$rootdir/var/cache/apt";
Apt::Install-Recommends false;
Apt::Get::Download-Only true;
Dir::Etc::Trusted "/etc/apt/trusted.gpg";
Dir::Etc::TrustedParts "/etc/apt/trusted.gpg.d";
END
> "$rootdir/var/lib/dpkg/status"
cat << END > "$rootdir/etc/apt/sources.list"
deb [arch=$nativearch] $mirror $dist $components
END
APT_CONFIG="$rootdir/etc/apt/apt.conf" apt-get update
pkgs=$(APT_CONFIG="$rootdir/etc/apt/apt.conf" apt-get indextargets \
--format '$(FILENAME)' 'Created-By: Packages' "Architecture: $nativearch" \
| xargs --delimiter='\n' /usr/lib/apt/apt-helper cat-file \
| grep-dctrl --no-field-names --show-field=Package --exact-match \
\( --field=Essential yes --or --field=Priority required \
--or --field=Priority important --or --field=Priority standard \))
pkgs="$(echo $pkgs) build-essential"
APT_CONFIG="$rootdir/etc/apt/apt.conf" apt-get --yes install $pkgs
# to be able to also test gpg verification, we need to create a mirror
mkdir -p "$mirrordir/dists/$dist/" "$mirrordir/dists/$dist/main/binary-amd64/"
curl --location "$mirror/dists/$dist/Release" > "$mirrordir/dists/$dist/Release"
curl --location "$mirror/dists/$dist/Release.gpg" > "$mirrordir/dists/$dist/Release.gpg"
curl --location "$mirror/dists/$dist/main/binary-amd64/Packages.gz" > "$mirrordir/dists/$dist/main/binary-amd64/Packages.gz"
# the deb files downloaded by apt must be moved to their right locations in the
# pool directory
#
# Instead of parsing the Packages file, we could also attempt to move the deb
# files ourselves to the appropriate pool directories. But that approach
# requires re-creating the heuristic by which the directory is chosen, requires
# stripping the epoch from the filename and will break once mirrors change.
# This way, it doesn't matter where the mirror ends up storing the package.
gzip -dc "$mirrordir/dists/$dist/main/binary-amd64/Packages.gz" \
| grep-dctrl --no-field-names --show-field=Package,Version,Architecture,Filename,MD5sum '' \
| paste -sd " \n" \
| while read name ver arch fname md5; do
dir="${fname%/*}"
basename="${fname##*/}"
# apt stores deb files with the colon encoded as %3a while
# mirrors do not contain the epoch at all #645895
case "$ver" in *:*) ver="${ver%%:*}%3a${ver#*:}";; esac
aptname="$rootdir/var/cache/apt/archives/${name}_${ver}_${arch}.deb"
if [ -e "$aptname" ]; then
# make sure that we found the right file by checking its hash
echo "$md5 $aptname" | md5sum --check
mkdir -p "$mirrordir/$dir"
mv "$aptname" "$mirrordir/$fname"
fi
done
rm "$rootdir/var/cache/apt/archives/lock"
rmdir "$rootdir/var/cache/apt/archives/partial"
# now the apt cache should be empty
if [ ! -z "$(ls -1qA "$rootdir/var/cache/apt/archives/")" ]; then
echo "/var/cache/apt/archives not empty"
exit 1
fi
rm -r "$rootdir"
done

1478
mmdebstrap Executable file

File diff suppressed because it is too large Load diff

175
test.sh Executable file
View file

@ -0,0 +1,175 @@
#!/bin/sh
set -eu
mirrordir="./mirror"
mirror="http://deb.debian.org/debian"
rootdir=$(mktemp --directory)
nativearch=$(dpkg --print-architecture)
components=main
abort=no
for dist in stable testing unstable; do
if [ -e debian-$dist-mm ]; then
echo "debian-$dist-mm exists"
abort=yes
fi
if [ -e debian-$dist-debootstrap ]; then
echo "debian-$dist-debootstrap exists"
abort=yes
fi
done
if [ $abort = yes ]; then
exit 1
fi
./make_mirror.sh
trap 'kill $pid' INT QUIT TERM EXIT
cd mirror
python3 -m http.server 8000 & pid=$!
cd -
# wait for the server to start
sleep 1
echo "running http server with pid $pid"
export SOURCE_DATE_EPOCH=$(date +%s)
for dist in stable testing unstable; do
> timings
> sizes
for variant in minbase buildd -; do
# skip because of different userids for apt/systemd
if [ "$dist" = 'stable' -a "$variant" = '-' ]; then
continue
fi
echo =========================================================
echo $dist $variant
echo =========================================================
/usr/bin/time --output=timings --append --format=%e ./mmdebstrap --variant=$variant --mode=unshare $dist debian-$dist-mm.tar "http://localhost:8000"
stat --format=%s debian-$dist-mm.tar >> sizes
mkdir ./debian-$dist-mm
cd ./debian-$dist-mm
sudo tar -xf ../debian-$dist-mm.tar
cd -
/usr/bin/time --output=timings --append --format=%e sudo debootstrap --merged-usr --variant=$variant $dist ./debian-$dist-debootstrap "http://localhost:8000/"
sudo tar --sort=name --mtime=@$SOURCE_DATE_EPOCH --clamp-mtime --numeric-owner --one-file-system -C ./debian-$dist-debootstrap -cf debian-$dist-debootstrap.tar .
sudo rm -r ./debian-$dist-debootstrap
stat --format=%s debian-$dist-debootstrap.tar >> sizes
mkdir ./debian-$dist-debootstrap
cd ./debian-$dist-debootstrap
sudo tar -xf ../debian-$dist-debootstrap.tar
cd -
# diff cannot compare device nodes, so we use tar to do that for us and then
# delete the directory
tar -C ./debian-$dist-debootstrap -cf dev1.tar ./dev
tar -C ./debian-$dist-mm -cf dev2.tar ./dev
cmp dev1.tar dev2.tar
rm dev1.tar dev2.tar
sudo rm -r ./debian-$dist-debootstrap/dev ./debian-$dist-mm/dev
# remove downloaded deb packages
sudo rm debian-$dist-debootstrap/var/cache/apt/archives/*.deb
# remove aux-cache
sudo rm debian-$dist-debootstrap/var/cache/ldconfig/aux-cache
# remove logs
sudo rm debian-$dist-debootstrap/var/log/dpkg.log \
debian-$dist-debootstrap/var/log/bootstrap.log \
debian-$dist-mm/var/log/apt/eipp.log.xz \
debian-$dist-debootstrap/var/log/alternatives.log
# remove *-old files
sudo rm debian-$dist-debootstrap/var/cache/debconf/config.dat-old \
debian-$dist-mm/var/cache/debconf/config.dat-old
sudo rm debian-$dist-debootstrap/var/cache/debconf/templates.dat-old \
debian-$dist-mm/var/cache/debconf/templates.dat-old
sudo rm debian-$dist-debootstrap/var/lib/dpkg/status-old \
debian-$dist-mm/var/lib/dpkg/status-old
# remove dpkg files
sudo rm debian-$dist-debootstrap/var/lib/dpkg/available \
debian-$dist-debootstrap/var/lib/dpkg/cmethopt
# since we installed packages directly from the .deb files, Priorities differ
# this we first check for equality and then remove the files
sudo chroot debian-$dist-debootstrap dpkg --list > dpkg1
sudo chroot debian-$dist-mm dpkg --list > dpkg2
diff -u dpkg1 dpkg2
rm dpkg1 dpkg2
grep -v '^Priority: ' debian-$dist-debootstrap/var/lib/dpkg/status > status1
grep -v '^Priority: ' debian-$dist-mm/var/lib/dpkg/status > status2
diff -u status1 status2
rm status1 status2
sudo rm debian-$dist-debootstrap/var/lib/dpkg/status debian-$dist-mm/var/lib/dpkg/status
# since we installed using apt, we have to remove some leftovers
sudo rm debian-$dist-mm/var/cache/apt/archives/lock \
debian-$dist-mm/var/lib/apt/extended_states \
debian-$dist-mm/var/lib/apt/lists/lock
sudo rmdir debian-$dist-mm/var/lib/apt/lists/auxfiles
# debootstrap exposes the hosts's kernel version
sudo rm debian-$dist-debootstrap/etc/apt/apt.conf.d/01autoremove-kernels \
debian-$dist-mm/etc/apt/apt.conf.d/01autoremove-kernels
# who creates /run/mount?
sudo rm -f debian-$dist-debootstrap/run/mount/utab
sudo rmdir debian-$dist-debootstrap/run/mount
# debootstrap doesn't clean apt
sudo rm debian-$dist-debootstrap/var/lib/apt/lists/localhost:8000_dists_${dist}_main_binary-amd64_Packages \
debian-$dist-debootstrap/var/lib/apt/lists/localhost:8000_dists_${dist}_Release \
debian-$dist-debootstrap/var/lib/apt/lists/localhost:8000_dists_${dist}_Release.gpg
if [ "$variant" = "-" ]; then
sudo rm debian-$dist-debootstrap/etc/machine-id
sudo rm debian-$dist-mm/etc/machine-id
sudo rm debian-$dist-debootstrap/var/lib/systemd/catalog/database
sudo rm debian-$dist-mm/var/lib/systemd/catalog/database
fi
# check if the file content differs
sudo diff --no-dereference --brief --recursive debian-$dist-debootstrap debian-$dist-mm
sudo rm -rf ./debian-$dist-debootstrap ./debian-$dist-mm \
./debian-$dist-debootstrap.tar ./debian-$dist-mm.tar
done
eval $(awk '{print "var"NR"="$1}' timings)
echo
echo "timings"
echo "======="
echo
echo "variant | mmdebstrap | debootstrap"
echo "--------+------------+------------"
echo "minbase | $var1 | $var2"
echo "buildd | $var3 | $var4"
if [ "$dist" != 'stable' ]; then
echo "- | $var5 | $var6"
fi
eval $(awk '{print "var"NR"="$1}' sizes)
echo
echo "sizes"
echo "======="
echo
echo "variant | mmdebstrap | debootstrap"
echo "--------+------------+------------"
echo "minbase | $var1 | $var2"
echo "buildd | $var3 | $var4"
if [ "$dist" != 'stable' ]; then
echo "- | $var5 | $var6"
fi
rm timings sizes
done
kill $pid
wait $pid || true
trap - INT QUIT TERM EXIT