initial commit
This commit is contained in:
commit
9ed4c65e35
GPG key ID:
F2CBA5C78FBD83E1
4 changed files with 1855 additions and 0 deletions
103
README.md
Normal file
103
README.md
Normal file
|
|
@ -0,0 +1,103 @@
|
||||||
|
mmdebstrap
|
||||||
|
==========
|
||||||
|
|
||||||
|
An alternative to debootstrap which uses apt internally and is thus able to use
|
||||||
|
more than one mirror and resolve more complex dependencies.
|
||||||
|
|
||||||
|
Usage
|
||||||
|
-----
|
||||||
|
|
||||||
|
Use like debootstrap:
|
||||||
|
|
||||||
|
sudo mmdebstrap unstable ./unstable-chroot
|
||||||
|
|
||||||
|
Without superuser privileges:
|
||||||
|
|
||||||
|
mmdebstrap unstable unstable-chroot.tar
|
||||||
|
|
||||||
|
With complex apt options:
|
||||||
|
|
||||||
|
cat /etc/apt/sources.list | mmdebstrap > unstable-chroot.tar
|
||||||
|
|
||||||
|
The sales pitch in comparison to debootstrap
|
||||||
|
--------------------------------------------
|
||||||
|
|
||||||
|
Summary:
|
||||||
|
|
||||||
|
- more than one mirror possible
|
||||||
|
- security and updates mirror included for Debian stable chroots
|
||||||
|
- 2-3 times faster
|
||||||
|
- chroot with apt in 11 seconds
|
||||||
|
- gzipped tarball with apt is 27M small
|
||||||
|
- bit-by-bit reproducible output
|
||||||
|
- unprivileged operation using Linux user namespaces, fakechroot or proot
|
||||||
|
- can operate on filesystems mounted with nodev
|
||||||
|
- foreign architecture chroots with qemu-user
|
||||||
|
|
||||||
|
The author believes that a chroot of a Debian stable release should include the
|
||||||
|
latest packages including security fixes by default. This has been a wontfix
|
||||||
|
with debootstrap since 2009 (See #543819 and #762222). Since mmdebstrap uses
|
||||||
|
apt internally, support for multiple mirrors comes for free and stable or
|
||||||
|
oldstable **chroots will include security and updates mirrors**.
|
||||||
|
|
||||||
|
A side-effect of using apt is being **2-3 times faster** than debootstrap. The
|
||||||
|
timings were carried out on a laptop with an Intel Core i5-5200U.
|
||||||
|
|
||||||
|
| variant | mmdebstrap | debootstrap |
|
||||||
|
| ------- | ---------- | ------------ |
|
||||||
|
| minbase | 25.25 s | 51.47 s |
|
||||||
|
| buildd | 30.99 s | 59.38 s |
|
||||||
|
| - | 29.85 s | 127.18 s |
|
||||||
|
|
||||||
|
Apt considers itself an `Essential: yes` package. This feature allows one to
|
||||||
|
create a chroot containing just the `Essential: yes` packages and apt (and
|
||||||
|
their hard dependencies) in **just 11 seconds**.
|
||||||
|
|
||||||
|
If desired, a most minimal chroot with just the `Essential: yes` packages and
|
||||||
|
their hard dependencies can be created with a gzipped tarball size of just 34M.
|
||||||
|
By using dpkg's `--path-exclude` option to exclude documentation, even smaller
|
||||||
|
gzipped tarballs of 21M in size are possible. If apt is included, the result is
|
||||||
|
a **gzipped tarball of only 27M**.
|
||||||
|
|
||||||
|
These small sizes are also achieved because apt caches and other cruft is
|
||||||
|
stripped from the chroot. This also makes the result **bit-by-bit
|
||||||
|
reproducible** if the `$SOURCE_DATE_EPOCH` environment variable is set.
|
||||||
|
|
||||||
|
The author believes, that it should not be necessary to have superuser
|
||||||
|
privileges to create a file (the chroot tarball) in one's home directory. If
|
||||||
|
mmdebstrap is run by an unprivileged user, either Linux user namespaces,
|
||||||
|
fakechroot or proot are used to create a chroot tarball. Debootstrap supports
|
||||||
|
fakechroot but will not create a tarball with the right permissions by itself.
|
||||||
|
Support for Linux user namespaces and proot is missing (see bugs #829134 and
|
||||||
|
#698347, respectively).
|
||||||
|
|
||||||
|
When creating a chroot tarball with debootstrap, the temporary chroot directory
|
||||||
|
cannot be on a filesystem that has been mounted with nodev. In unprivileged
|
||||||
|
mode, mknod is never used, which means that /tmp can be used as a temporary
|
||||||
|
directory location even if if it's mounted with nodev as a security measure.
|
||||||
|
|
||||||
|
If the chroot architecture cannot be executed by the current machine, qemu-user
|
||||||
|
is used to allow one to create a foreign architecture chroot.
|
||||||
|
|
||||||
|
Limitations in comparison to debootstrap
|
||||||
|
----------------------------------------
|
||||||
|
|
||||||
|
Debootstrap supports creating a Debian chroot on non-Debian systems but
|
||||||
|
mmdebstrap requires apt.
|
||||||
|
|
||||||
|
There is no `SCRIPT` argument.
|
||||||
|
|
||||||
|
There is no `--second-stage` option.
|
||||||
|
|
||||||
|
Tests
|
||||||
|
=====
|
||||||
|
|
||||||
|
The script `test.sh` compares the output of mmdebstrap with debootstrap in
|
||||||
|
several scenarios. Since debootstrap needs superuser privileges, `test.sh`
|
||||||
|
needs `sudo` to run.
|
||||||
|
|
||||||
|
Bugs
|
||||||
|
====
|
||||||
|
|
||||||
|
mmdebstrap has bugs. Report them here:
|
||||||
|
https://gitlab.mister-muffin.de/josch/mmdebstrap/issues
|
||||||
99
make_mirror.sh
Executable file
99
make_mirror.sh
Executable file
|
|
@ -0,0 +1,99 @@
|
||||||
|
#!/bin/sh
|
||||||
|
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
mirrordir="./mirror"
|
||||||
|
|
||||||
|
mirror="http://deb.debian.org/debian"
|
||||||
|
nativearch=$(dpkg --print-architecture)
|
||||||
|
components=main
|
||||||
|
|
||||||
|
if [ -e "$mirrordir/dists/unstable/Release" ]; then
|
||||||
|
http_code=$(curl --output /dev/null --silent --location --head --time-cond "$mirrordir/dists/unstable/Release" --write-out '%{http_code}' "$mirror/dists/unstable/Release")
|
||||||
|
case "$http_code" in
|
||||||
|
200) ;; # need update
|
||||||
|
304) echo up-to-date; exit 0;;
|
||||||
|
*) echo unexpected status: $http_code; exit 1;;
|
||||||
|
esac
|
||||||
|
fi
|
||||||
|
|
||||||
|
for dist in stable testing unstable; do
|
||||||
|
rootdir=$(mktemp --directory)
|
||||||
|
|
||||||
|
for p in /etc/apt/apt.conf.d /etc/apt/sources.list.d /etc/apt/preferences.d /var/cache/apt /var/lib/apt/lists/partial /var/lib/dpkg; do
|
||||||
|
mkdir -p "$rootdir/$p"
|
||||||
|
done
|
||||||
|
|
||||||
|
cat << END > "$rootdir/etc/apt/apt.conf"
|
||||||
|
Apt::Architecture "$nativearch";
|
||||||
|
Dir::Etc "$rootdir/etc/apt";
|
||||||
|
Dir::State "$rootdir/var/lib/apt";
|
||||||
|
Dir::Cache "$rootdir/var/cache/apt";
|
||||||
|
Apt::Install-Recommends false;
|
||||||
|
Apt::Get::Download-Only true;
|
||||||
|
Dir::Etc::Trusted "/etc/apt/trusted.gpg";
|
||||||
|
Dir::Etc::TrustedParts "/etc/apt/trusted.gpg.d";
|
||||||
|
END
|
||||||
|
|
||||||
|
> "$rootdir/var/lib/dpkg/status"
|
||||||
|
|
||||||
|
cat << END > "$rootdir/etc/apt/sources.list"
|
||||||
|
deb [arch=$nativearch] $mirror $dist $components
|
||||||
|
END
|
||||||
|
|
||||||
|
|
||||||
|
APT_CONFIG="$rootdir/etc/apt/apt.conf" apt-get update
|
||||||
|
|
||||||
|
pkgs=$(APT_CONFIG="$rootdir/etc/apt/apt.conf" apt-get indextargets \
|
||||||
|
--format '$(FILENAME)' 'Created-By: Packages' "Architecture: $nativearch" \
|
||||||
|
| xargs --delimiter='\n' /usr/lib/apt/apt-helper cat-file \
|
||||||
|
| grep-dctrl --no-field-names --show-field=Package --exact-match \
|
||||||
|
\( --field=Essential yes --or --field=Priority required \
|
||||||
|
--or --field=Priority important --or --field=Priority standard \))
|
||||||
|
|
||||||
|
pkgs="$(echo $pkgs) build-essential"
|
||||||
|
|
||||||
|
APT_CONFIG="$rootdir/etc/apt/apt.conf" apt-get --yes install $pkgs
|
||||||
|
|
||||||
|
# to be able to also test gpg verification, we need to create a mirror
|
||||||
|
mkdir -p "$mirrordir/dists/$dist/" "$mirrordir/dists/$dist/main/binary-amd64/"
|
||||||
|
curl --location "$mirror/dists/$dist/Release" > "$mirrordir/dists/$dist/Release"
|
||||||
|
curl --location "$mirror/dists/$dist/Release.gpg" > "$mirrordir/dists/$dist/Release.gpg"
|
||||||
|
curl --location "$mirror/dists/$dist/main/binary-amd64/Packages.gz" > "$mirrordir/dists/$dist/main/binary-amd64/Packages.gz"
|
||||||
|
|
||||||
|
# the deb files downloaded by apt must be moved to their right locations in the
|
||||||
|
# pool directory
|
||||||
|
#
|
||||||
|
# Instead of parsing the Packages file, we could also attempt to move the deb
|
||||||
|
# files ourselves to the appropriate pool directories. But that approach
|
||||||
|
# requires re-creating the heuristic by which the directory is chosen, requires
|
||||||
|
# stripping the epoch from the filename and will break once mirrors change.
|
||||||
|
# This way, it doesn't matter where the mirror ends up storing the package.
|
||||||
|
gzip -dc "$mirrordir/dists/$dist/main/binary-amd64/Packages.gz" \
|
||||||
|
| grep-dctrl --no-field-names --show-field=Package,Version,Architecture,Filename,MD5sum '' \
|
||||||
|
| paste -sd " \n" \
|
||||||
|
| while read name ver arch fname md5; do
|
||||||
|
dir="${fname%/*}"
|
||||||
|
basename="${fname##*/}"
|
||||||
|
# apt stores deb files with the colon encoded as %3a while
|
||||||
|
# mirrors do not contain the epoch at all #645895
|
||||||
|
case "$ver" in *:*) ver="${ver%%:*}%3a${ver#*:}";; esac
|
||||||
|
aptname="$rootdir/var/cache/apt/archives/${name}_${ver}_${arch}.deb"
|
||||||
|
if [ -e "$aptname" ]; then
|
||||||
|
# make sure that we found the right file by checking its hash
|
||||||
|
echo "$md5 $aptname" | md5sum --check
|
||||||
|
mkdir -p "$mirrordir/$dir"
|
||||||
|
mv "$aptname" "$mirrordir/$fname"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
rm "$rootdir/var/cache/apt/archives/lock"
|
||||||
|
rmdir "$rootdir/var/cache/apt/archives/partial"
|
||||||
|
# now the apt cache should be empty
|
||||||
|
if [ ! -z "$(ls -1qA "$rootdir/var/cache/apt/archives/")" ]; then
|
||||||
|
echo "/var/cache/apt/archives not empty"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
rm -r "$rootdir"
|
||||||
|
done
|
||||||
1478
mmdebstrap
Executable file
1478
mmdebstrap
Executable file
File diff suppressed because it is too large
Load diff
175
test.sh
Executable file
175
test.sh
Executable file
|
|
@ -0,0 +1,175 @@
|
||||||
|
#!/bin/sh
|
||||||
|
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
mirrordir="./mirror"
|
||||||
|
|
||||||
|
mirror="http://deb.debian.org/debian"
|
||||||
|
rootdir=$(mktemp --directory)
|
||||||
|
nativearch=$(dpkg --print-architecture)
|
||||||
|
components=main
|
||||||
|
|
||||||
|
abort=no
|
||||||
|
for dist in stable testing unstable; do
|
||||||
|
if [ -e debian-$dist-mm ]; then
|
||||||
|
echo "debian-$dist-mm exists"
|
||||||
|
abort=yes
|
||||||
|
fi
|
||||||
|
if [ -e debian-$dist-debootstrap ]; then
|
||||||
|
echo "debian-$dist-debootstrap exists"
|
||||||
|
abort=yes
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
if [ $abort = yes ]; then
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
./make_mirror.sh
|
||||||
|
|
||||||
|
trap 'kill $pid' INT QUIT TERM EXIT
|
||||||
|
|
||||||
|
cd mirror
|
||||||
|
python3 -m http.server 8000 & pid=$!
|
||||||
|
cd -
|
||||||
|
|
||||||
|
# wait for the server to start
|
||||||
|
sleep 1
|
||||||
|
echo "running http server with pid $pid"
|
||||||
|
|
||||||
|
export SOURCE_DATE_EPOCH=$(date +%s)
|
||||||
|
|
||||||
|
for dist in stable testing unstable; do
|
||||||
|
> timings
|
||||||
|
> sizes
|
||||||
|
for variant in minbase buildd -; do
|
||||||
|
# skip because of different userids for apt/systemd
|
||||||
|
if [ "$dist" = 'stable' -a "$variant" = '-' ]; then
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
echo =========================================================
|
||||||
|
echo $dist $variant
|
||||||
|
echo =========================================================
|
||||||
|
|
||||||
|
/usr/bin/time --output=timings --append --format=%e ./mmdebstrap --variant=$variant --mode=unshare $dist debian-$dist-mm.tar "http://localhost:8000"
|
||||||
|
|
||||||
|
stat --format=%s debian-$dist-mm.tar >> sizes
|
||||||
|
mkdir ./debian-$dist-mm
|
||||||
|
cd ./debian-$dist-mm
|
||||||
|
sudo tar -xf ../debian-$dist-mm.tar
|
||||||
|
cd -
|
||||||
|
|
||||||
|
/usr/bin/time --output=timings --append --format=%e sudo debootstrap --merged-usr --variant=$variant $dist ./debian-$dist-debootstrap "http://localhost:8000/"
|
||||||
|
sudo tar --sort=name --mtime=@$SOURCE_DATE_EPOCH --clamp-mtime --numeric-owner --one-file-system -C ./debian-$dist-debootstrap -cf debian-$dist-debootstrap.tar .
|
||||||
|
sudo rm -r ./debian-$dist-debootstrap
|
||||||
|
|
||||||
|
stat --format=%s debian-$dist-debootstrap.tar >> sizes
|
||||||
|
mkdir ./debian-$dist-debootstrap
|
||||||
|
cd ./debian-$dist-debootstrap
|
||||||
|
sudo tar -xf ../debian-$dist-debootstrap.tar
|
||||||
|
cd -
|
||||||
|
|
||||||
|
# diff cannot compare device nodes, so we use tar to do that for us and then
|
||||||
|
# delete the directory
|
||||||
|
tar -C ./debian-$dist-debootstrap -cf dev1.tar ./dev
|
||||||
|
tar -C ./debian-$dist-mm -cf dev2.tar ./dev
|
||||||
|
cmp dev1.tar dev2.tar
|
||||||
|
rm dev1.tar dev2.tar
|
||||||
|
sudo rm -r ./debian-$dist-debootstrap/dev ./debian-$dist-mm/dev
|
||||||
|
|
||||||
|
# remove downloaded deb packages
|
||||||
|
sudo rm debian-$dist-debootstrap/var/cache/apt/archives/*.deb
|
||||||
|
# remove aux-cache
|
||||||
|
sudo rm debian-$dist-debootstrap/var/cache/ldconfig/aux-cache
|
||||||
|
# remove logs
|
||||||
|
sudo rm debian-$dist-debootstrap/var/log/dpkg.log \
|
||||||
|
debian-$dist-debootstrap/var/log/bootstrap.log \
|
||||||
|
debian-$dist-mm/var/log/apt/eipp.log.xz \
|
||||||
|
debian-$dist-debootstrap/var/log/alternatives.log
|
||||||
|
# remove *-old files
|
||||||
|
sudo rm debian-$dist-debootstrap/var/cache/debconf/config.dat-old \
|
||||||
|
debian-$dist-mm/var/cache/debconf/config.dat-old
|
||||||
|
sudo rm debian-$dist-debootstrap/var/cache/debconf/templates.dat-old \
|
||||||
|
debian-$dist-mm/var/cache/debconf/templates.dat-old
|
||||||
|
sudo rm debian-$dist-debootstrap/var/lib/dpkg/status-old \
|
||||||
|
debian-$dist-mm/var/lib/dpkg/status-old
|
||||||
|
# remove dpkg files
|
||||||
|
sudo rm debian-$dist-debootstrap/var/lib/dpkg/available \
|
||||||
|
debian-$dist-debootstrap/var/lib/dpkg/cmethopt
|
||||||
|
# since we installed packages directly from the .deb files, Priorities differ
|
||||||
|
# this we first check for equality and then remove the files
|
||||||
|
sudo chroot debian-$dist-debootstrap dpkg --list > dpkg1
|
||||||
|
sudo chroot debian-$dist-mm dpkg --list > dpkg2
|
||||||
|
diff -u dpkg1 dpkg2
|
||||||
|
rm dpkg1 dpkg2
|
||||||
|
grep -v '^Priority: ' debian-$dist-debootstrap/var/lib/dpkg/status > status1
|
||||||
|
grep -v '^Priority: ' debian-$dist-mm/var/lib/dpkg/status > status2
|
||||||
|
diff -u status1 status2
|
||||||
|
rm status1 status2
|
||||||
|
sudo rm debian-$dist-debootstrap/var/lib/dpkg/status debian-$dist-mm/var/lib/dpkg/status
|
||||||
|
# since we installed using apt, we have to remove some leftovers
|
||||||
|
sudo rm debian-$dist-mm/var/cache/apt/archives/lock \
|
||||||
|
debian-$dist-mm/var/lib/apt/extended_states \
|
||||||
|
debian-$dist-mm/var/lib/apt/lists/lock
|
||||||
|
sudo rmdir debian-$dist-mm/var/lib/apt/lists/auxfiles
|
||||||
|
# debootstrap exposes the hosts's kernel version
|
||||||
|
sudo rm debian-$dist-debootstrap/etc/apt/apt.conf.d/01autoremove-kernels \
|
||||||
|
debian-$dist-mm/etc/apt/apt.conf.d/01autoremove-kernels
|
||||||
|
# who creates /run/mount?
|
||||||
|
sudo rm -f debian-$dist-debootstrap/run/mount/utab
|
||||||
|
sudo rmdir debian-$dist-debootstrap/run/mount
|
||||||
|
# debootstrap doesn't clean apt
|
||||||
|
sudo rm debian-$dist-debootstrap/var/lib/apt/lists/localhost:8000_dists_${dist}_main_binary-amd64_Packages \
|
||||||
|
debian-$dist-debootstrap/var/lib/apt/lists/localhost:8000_dists_${dist}_Release \
|
||||||
|
debian-$dist-debootstrap/var/lib/apt/lists/localhost:8000_dists_${dist}_Release.gpg
|
||||||
|
|
||||||
|
if [ "$variant" = "-" ]; then
|
||||||
|
sudo rm debian-$dist-debootstrap/etc/machine-id
|
||||||
|
sudo rm debian-$dist-mm/etc/machine-id
|
||||||
|
sudo rm debian-$dist-debootstrap/var/lib/systemd/catalog/database
|
||||||
|
sudo rm debian-$dist-mm/var/lib/systemd/catalog/database
|
||||||
|
fi
|
||||||
|
|
||||||
|
# check if the file content differs
|
||||||
|
sudo diff --no-dereference --brief --recursive debian-$dist-debootstrap debian-$dist-mm
|
||||||
|
|
||||||
|
sudo rm -rf ./debian-$dist-debootstrap ./debian-$dist-mm \
|
||||||
|
./debian-$dist-debootstrap.tar ./debian-$dist-mm.tar
|
||||||
|
done
|
||||||
|
|
||||||
|
eval $(awk '{print "var"NR"="$1}' timings)
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "timings"
|
||||||
|
echo "======="
|
||||||
|
echo
|
||||||
|
echo "variant | mmdebstrap | debootstrap"
|
||||||
|
echo "--------+------------+------------"
|
||||||
|
echo "minbase | $var1 | $var2"
|
||||||
|
echo "buildd | $var3 | $var4"
|
||||||
|
if [ "$dist" != 'stable' ]; then
|
||||||
|
echo "- | $var5 | $var6"
|
||||||
|
fi
|
||||||
|
|
||||||
|
eval $(awk '{print "var"NR"="$1}' sizes)
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "sizes"
|
||||||
|
echo "======="
|
||||||
|
echo
|
||||||
|
echo "variant | mmdebstrap | debootstrap"
|
||||||
|
echo "--------+------------+------------"
|
||||||
|
echo "minbase | $var1 | $var2"
|
||||||
|
echo "buildd | $var3 | $var4"
|
||||||
|
if [ "$dist" != 'stable' ]; then
|
||||||
|
echo "- | $var5 | $var6"
|
||||||
|
fi
|
||||||
|
|
||||||
|
rm timings sizes
|
||||||
|
done
|
||||||
|
|
||||||
|
kill $pid
|
||||||
|
|
||||||
|
wait $pid || true
|
||||||
|
|
||||||
|
trap - INT QUIT TERM EXIT
|
||||||
|
|
||||||
Loading…
Reference in a new issue