initial commit
This commit is contained in:
commit
9ed4c65e35
GPG key ID:
F2CBA5C78FBD83E1
4 changed files with 1855 additions and 0 deletions
103
README.md
Normal file
103
README.md
Normal file
|
|
@ -0,0 +1,103 @@
|
|||
mmdebstrap
|
||||
==========
|
||||
|
||||
An alternative to debootstrap which uses apt internally and is thus able to use
|
||||
more than one mirror and resolve more complex dependencies.
|
||||
|
||||
Usage
|
||||
-----
|
||||
|
||||
Use like debootstrap:
|
||||
|
||||
sudo mmdebstrap unstable ./unstable-chroot
|
||||
|
||||
Without superuser privileges:
|
||||
|
||||
mmdebstrap unstable unstable-chroot.tar
|
||||
|
||||
With complex apt options:
|
||||
|
||||
cat /etc/apt/sources.list | mmdebstrap > unstable-chroot.tar
|
||||
|
||||
The sales pitch in comparison to debootstrap
|
||||
--------------------------------------------
|
||||
|
||||
Summary:
|
||||
|
||||
- more than one mirror possible
|
||||
- security and updates mirror included for Debian stable chroots
|
||||
- 2-3 times faster
|
||||
- chroot with apt in 11 seconds
|
||||
- gzipped tarball with apt is 27M small
|
||||
- bit-by-bit reproducible output
|
||||
- unprivileged operation using Linux user namespaces, fakechroot or proot
|
||||
- can operate on filesystems mounted with nodev
|
||||
- foreign architecture chroots with qemu-user
|
||||
|
||||
The author believes that a chroot of a Debian stable release should include the
|
||||
latest packages including security fixes by default. This has been a wontfix
|
||||
with debootstrap since 2009 (See #543819 and #762222). Since mmdebstrap uses
|
||||
apt internally, support for multiple mirrors comes for free and stable or
|
||||
oldstable **chroots will include security and updates mirrors**.
|
||||
|
||||
A side-effect of using apt is being **2-3 times faster** than debootstrap. The
|
||||
timings were carried out on a laptop with an Intel Core i5-5200U.
|
||||
|
||||
| variant | mmdebstrap | debootstrap |
|
||||
| ------- | ---------- | ------------ |
|
||||
| minbase | 25.25 s | 51.47 s |
|
||||
| buildd | 30.99 s | 59.38 s |
|
||||
| - | 29.85 s | 127.18 s |
|
||||
|
||||
Apt considers itself an `Essential: yes` package. This feature allows one to
|
||||
create a chroot containing just the `Essential: yes` packages and apt (and
|
||||
their hard dependencies) in **just 11 seconds**.
|
||||
|
||||
If desired, a most minimal chroot with just the `Essential: yes` packages and
|
||||
their hard dependencies can be created with a gzipped tarball size of just 34M.
|
||||
By using dpkg's `--path-exclude` option to exclude documentation, even smaller
|
||||
gzipped tarballs of 21M in size are possible. If apt is included, the result is
|
||||
a **gzipped tarball of only 27M**.
|
||||
|
||||
These small sizes are also achieved because apt caches and other cruft is
|
||||
stripped from the chroot. This also makes the result **bit-by-bit
|
||||
reproducible** if the `$SOURCE_DATE_EPOCH` environment variable is set.
|
||||
|
||||
The author believes, that it should not be necessary to have superuser
|
||||
privileges to create a file (the chroot tarball) in one's home directory. If
|
||||
mmdebstrap is run by an unprivileged user, either Linux user namespaces,
|
||||
fakechroot or proot are used to create a chroot tarball. Debootstrap supports
|
||||
fakechroot but will not create a tarball with the right permissions by itself.
|
||||
Support for Linux user namespaces and proot is missing (see bugs #829134 and
|
||||
#698347, respectively).
|
||||
|
||||
When creating a chroot tarball with debootstrap, the temporary chroot directory
|
||||
cannot be on a filesystem that has been mounted with nodev. In unprivileged
|
||||
mode, mknod is never used, which means that /tmp can be used as a temporary
|
||||
directory location even if if it's mounted with nodev as a security measure.
|
||||
|
||||
If the chroot architecture cannot be executed by the current machine, qemu-user
|
||||
is used to allow one to create a foreign architecture chroot.
|
||||
|
||||
Limitations in comparison to debootstrap
|
||||
----------------------------------------
|
||||
|
||||
Debootstrap supports creating a Debian chroot on non-Debian systems but
|
||||
mmdebstrap requires apt.
|
||||
|
||||
There is no `SCRIPT` argument.
|
||||
|
||||
There is no `--second-stage` option.
|
||||
|
||||
Tests
|
||||
=====
|
||||
|
||||
The script `test.sh` compares the output of mmdebstrap with debootstrap in
|
||||
several scenarios. Since debootstrap needs superuser privileges, `test.sh`
|
||||
needs `sudo` to run.
|
||||
|
||||
Bugs
|
||||
====
|
||||
|
||||
mmdebstrap has bugs. Report them here:
|
||||
https://gitlab.mister-muffin.de/josch/mmdebstrap/issues
|
||||
99
make_mirror.sh
Executable file
99
make_mirror.sh
Executable file
|
|
@ -0,0 +1,99 @@
|
|||
#!/bin/sh
|
||||
|
||||
set -eu
|
||||
|
||||
mirrordir="./mirror"
|
||||
|
||||
mirror="http://deb.debian.org/debian"
|
||||
nativearch=$(dpkg --print-architecture)
|
||||
components=main
|
||||
|
||||
if [ -e "$mirrordir/dists/unstable/Release" ]; then
|
||||
http_code=$(curl --output /dev/null --silent --location --head --time-cond "$mirrordir/dists/unstable/Release" --write-out '%{http_code}' "$mirror/dists/unstable/Release")
|
||||
case "$http_code" in
|
||||
200) ;; # need update
|
||||
304) echo up-to-date; exit 0;;
|
||||
*) echo unexpected status: $http_code; exit 1;;
|
||||
esac
|
||||
fi
|
||||
|
||||
for dist in stable testing unstable; do
|
||||
rootdir=$(mktemp --directory)
|
||||
|
||||
for p in /etc/apt/apt.conf.d /etc/apt/sources.list.d /etc/apt/preferences.d /var/cache/apt /var/lib/apt/lists/partial /var/lib/dpkg; do
|
||||
mkdir -p "$rootdir/$p"
|
||||
done
|
||||
|
||||
cat << END > "$rootdir/etc/apt/apt.conf"
|
||||
Apt::Architecture "$nativearch";
|
||||
Dir::Etc "$rootdir/etc/apt";
|
||||
Dir::State "$rootdir/var/lib/apt";
|
||||
Dir::Cache "$rootdir/var/cache/apt";
|
||||
Apt::Install-Recommends false;
|
||||
Apt::Get::Download-Only true;
|
||||
Dir::Etc::Trusted "/etc/apt/trusted.gpg";
|
||||
Dir::Etc::TrustedParts "/etc/apt/trusted.gpg.d";
|
||||
END
|
||||
|
||||
> "$rootdir/var/lib/dpkg/status"
|
||||
|
||||
cat << END > "$rootdir/etc/apt/sources.list"
|
||||
deb [arch=$nativearch] $mirror $dist $components
|
||||
END
|
||||
|
||||
|
||||
APT_CONFIG="$rootdir/etc/apt/apt.conf" apt-get update
|
||||
|
||||
pkgs=$(APT_CONFIG="$rootdir/etc/apt/apt.conf" apt-get indextargets \
|
||||
--format '$(FILENAME)' 'Created-By: Packages' "Architecture: $nativearch" \
|
||||
| xargs --delimiter='\n' /usr/lib/apt/apt-helper cat-file \
|
||||
| grep-dctrl --no-field-names --show-field=Package --exact-match \
|
||||
\( --field=Essential yes --or --field=Priority required \
|
||||
--or --field=Priority important --or --field=Priority standard \))
|
||||
|
||||
pkgs="$(echo $pkgs) build-essential"
|
||||
|
||||
APT_CONFIG="$rootdir/etc/apt/apt.conf" apt-get --yes install $pkgs
|
||||
|
||||
# to be able to also test gpg verification, we need to create a mirror
|
||||
mkdir -p "$mirrordir/dists/$dist/" "$mirrordir/dists/$dist/main/binary-amd64/"
|
||||
curl --location "$mirror/dists/$dist/Release" > "$mirrordir/dists/$dist/Release"
|
||||
curl --location "$mirror/dists/$dist/Release.gpg" > "$mirrordir/dists/$dist/Release.gpg"
|
||||
curl --location "$mirror/dists/$dist/main/binary-amd64/Packages.gz" > "$mirrordir/dists/$dist/main/binary-amd64/Packages.gz"
|
||||
|
||||
# the deb files downloaded by apt must be moved to their right locations in the
|
||||
# pool directory
|
||||
#
|
||||
# Instead of parsing the Packages file, we could also attempt to move the deb
|
||||
# files ourselves to the appropriate pool directories. But that approach
|
||||
# requires re-creating the heuristic by which the directory is chosen, requires
|
||||
# stripping the epoch from the filename and will break once mirrors change.
|
||||
# This way, it doesn't matter where the mirror ends up storing the package.
|
||||
gzip -dc "$mirrordir/dists/$dist/main/binary-amd64/Packages.gz" \
|
||||
| grep-dctrl --no-field-names --show-field=Package,Version,Architecture,Filename,MD5sum '' \
|
||||
| paste -sd " \n" \
|
||||
| while read name ver arch fname md5; do
|
||||
dir="${fname%/*}"
|
||||
basename="${fname##*/}"
|
||||
# apt stores deb files with the colon encoded as %3a while
|
||||
# mirrors do not contain the epoch at all #645895
|
||||
case "$ver" in *:*) ver="${ver%%:*}%3a${ver#*:}";; esac
|
||||
aptname="$rootdir/var/cache/apt/archives/${name}_${ver}_${arch}.deb"
|
||||
if [ -e "$aptname" ]; then
|
||||
# make sure that we found the right file by checking its hash
|
||||
echo "$md5 $aptname" | md5sum --check
|
||||
mkdir -p "$mirrordir/$dir"
|
||||
mv "$aptname" "$mirrordir/$fname"
|
||||
fi
|
||||
done
|
||||
|
||||
rm "$rootdir/var/cache/apt/archives/lock"
|
||||
rmdir "$rootdir/var/cache/apt/archives/partial"
|
||||
# now the apt cache should be empty
|
||||
if [ ! -z "$(ls -1qA "$rootdir/var/cache/apt/archives/")" ]; then
|
||||
echo "/var/cache/apt/archives not empty"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
rm -r "$rootdir"
|
||||
done
|
||||
1478
mmdebstrap
Executable file
1478
mmdebstrap
Executable file
File diff suppressed because it is too large
Load diff
175
test.sh
Executable file
175
test.sh
Executable file
|
|
@ -0,0 +1,175 @@
|
|||
#!/bin/sh
|
||||
|
||||
set -eu
|
||||
|
||||
mirrordir="./mirror"
|
||||
|
||||
mirror="http://deb.debian.org/debian"
|
||||
rootdir=$(mktemp --directory)
|
||||
nativearch=$(dpkg --print-architecture)
|
||||
components=main
|
||||
|
||||
abort=no
|
||||
for dist in stable testing unstable; do
|
||||
if [ -e debian-$dist-mm ]; then
|
||||
echo "debian-$dist-mm exists"
|
||||
abort=yes
|
||||
fi
|
||||
if [ -e debian-$dist-debootstrap ]; then
|
||||
echo "debian-$dist-debootstrap exists"
|
||||
abort=yes
|
||||
fi
|
||||
done
|
||||
if [ $abort = yes ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
./make_mirror.sh
|
||||
|
||||
trap 'kill $pid' INT QUIT TERM EXIT
|
||||
|
||||
cd mirror
|
||||
python3 -m http.server 8000 & pid=$!
|
||||
cd -
|
||||
|
||||
# wait for the server to start
|
||||
sleep 1
|
||||
echo "running http server with pid $pid"
|
||||
|
||||
export SOURCE_DATE_EPOCH=$(date +%s)
|
||||
|
||||
for dist in stable testing unstable; do
|
||||
> timings
|
||||
> sizes
|
||||
for variant in minbase buildd -; do
|
||||
# skip because of different userids for apt/systemd
|
||||
if [ "$dist" = 'stable' -a "$variant" = '-' ]; then
|
||||
continue
|
||||
fi
|
||||
echo =========================================================
|
||||
echo $dist $variant
|
||||
echo =========================================================
|
||||
|
||||
/usr/bin/time --output=timings --append --format=%e ./mmdebstrap --variant=$variant --mode=unshare $dist debian-$dist-mm.tar "http://localhost:8000"
|
||||
|
||||
stat --format=%s debian-$dist-mm.tar >> sizes
|
||||
mkdir ./debian-$dist-mm
|
||||
cd ./debian-$dist-mm
|
||||
sudo tar -xf ../debian-$dist-mm.tar
|
||||
cd -
|
||||
|
||||
/usr/bin/time --output=timings --append --format=%e sudo debootstrap --merged-usr --variant=$variant $dist ./debian-$dist-debootstrap "http://localhost:8000/"
|
||||
sudo tar --sort=name --mtime=@$SOURCE_DATE_EPOCH --clamp-mtime --numeric-owner --one-file-system -C ./debian-$dist-debootstrap -cf debian-$dist-debootstrap.tar .
|
||||
sudo rm -r ./debian-$dist-debootstrap
|
||||
|
||||
stat --format=%s debian-$dist-debootstrap.tar >> sizes
|
||||
mkdir ./debian-$dist-debootstrap
|
||||
cd ./debian-$dist-debootstrap
|
||||
sudo tar -xf ../debian-$dist-debootstrap.tar
|
||||
cd -
|
||||
|
||||
# diff cannot compare device nodes, so we use tar to do that for us and then
|
||||
# delete the directory
|
||||
tar -C ./debian-$dist-debootstrap -cf dev1.tar ./dev
|
||||
tar -C ./debian-$dist-mm -cf dev2.tar ./dev
|
||||
cmp dev1.tar dev2.tar
|
||||
rm dev1.tar dev2.tar
|
||||
sudo rm -r ./debian-$dist-debootstrap/dev ./debian-$dist-mm/dev
|
||||
|
||||
# remove downloaded deb packages
|
||||
sudo rm debian-$dist-debootstrap/var/cache/apt/archives/*.deb
|
||||
# remove aux-cache
|
||||
sudo rm debian-$dist-debootstrap/var/cache/ldconfig/aux-cache
|
||||
# remove logs
|
||||
sudo rm debian-$dist-debootstrap/var/log/dpkg.log \
|
||||
debian-$dist-debootstrap/var/log/bootstrap.log \
|
||||
debian-$dist-mm/var/log/apt/eipp.log.xz \
|
||||
debian-$dist-debootstrap/var/log/alternatives.log
|
||||
# remove *-old files
|
||||
sudo rm debian-$dist-debootstrap/var/cache/debconf/config.dat-old \
|
||||
debian-$dist-mm/var/cache/debconf/config.dat-old
|
||||
sudo rm debian-$dist-debootstrap/var/cache/debconf/templates.dat-old \
|
||||
debian-$dist-mm/var/cache/debconf/templates.dat-old
|
||||
sudo rm debian-$dist-debootstrap/var/lib/dpkg/status-old \
|
||||
debian-$dist-mm/var/lib/dpkg/status-old
|
||||
# remove dpkg files
|
||||
sudo rm debian-$dist-debootstrap/var/lib/dpkg/available \
|
||||
debian-$dist-debootstrap/var/lib/dpkg/cmethopt
|
||||
# since we installed packages directly from the .deb files, Priorities differ
|
||||
# this we first check for equality and then remove the files
|
||||
sudo chroot debian-$dist-debootstrap dpkg --list > dpkg1
|
||||
sudo chroot debian-$dist-mm dpkg --list > dpkg2
|
||||
diff -u dpkg1 dpkg2
|
||||
rm dpkg1 dpkg2
|
||||
grep -v '^Priority: ' debian-$dist-debootstrap/var/lib/dpkg/status > status1
|
||||
grep -v '^Priority: ' debian-$dist-mm/var/lib/dpkg/status > status2
|
||||
diff -u status1 status2
|
||||
rm status1 status2
|
||||
sudo rm debian-$dist-debootstrap/var/lib/dpkg/status debian-$dist-mm/var/lib/dpkg/status
|
||||
# since we installed using apt, we have to remove some leftovers
|
||||
sudo rm debian-$dist-mm/var/cache/apt/archives/lock \
|
||||
debian-$dist-mm/var/lib/apt/extended_states \
|
||||
debian-$dist-mm/var/lib/apt/lists/lock
|
||||
sudo rmdir debian-$dist-mm/var/lib/apt/lists/auxfiles
|
||||
# debootstrap exposes the hosts's kernel version
|
||||
sudo rm debian-$dist-debootstrap/etc/apt/apt.conf.d/01autoremove-kernels \
|
||||
debian-$dist-mm/etc/apt/apt.conf.d/01autoremove-kernels
|
||||
# who creates /run/mount?
|
||||
sudo rm -f debian-$dist-debootstrap/run/mount/utab
|
||||
sudo rmdir debian-$dist-debootstrap/run/mount
|
||||
# debootstrap doesn't clean apt
|
||||
sudo rm debian-$dist-debootstrap/var/lib/apt/lists/localhost:8000_dists_${dist}_main_binary-amd64_Packages \
|
||||
debian-$dist-debootstrap/var/lib/apt/lists/localhost:8000_dists_${dist}_Release \
|
||||
debian-$dist-debootstrap/var/lib/apt/lists/localhost:8000_dists_${dist}_Release.gpg
|
||||
|
||||
if [ "$variant" = "-" ]; then
|
||||
sudo rm debian-$dist-debootstrap/etc/machine-id
|
||||
sudo rm debian-$dist-mm/etc/machine-id
|
||||
sudo rm debian-$dist-debootstrap/var/lib/systemd/catalog/database
|
||||
sudo rm debian-$dist-mm/var/lib/systemd/catalog/database
|
||||
fi
|
||||
|
||||
# check if the file content differs
|
||||
sudo diff --no-dereference --brief --recursive debian-$dist-debootstrap debian-$dist-mm
|
||||
|
||||
sudo rm -rf ./debian-$dist-debootstrap ./debian-$dist-mm \
|
||||
./debian-$dist-debootstrap.tar ./debian-$dist-mm.tar
|
||||
done
|
||||
|
||||
eval $(awk '{print "var"NR"="$1}' timings)
|
||||
|
||||
echo
|
||||
echo "timings"
|
||||
echo "======="
|
||||
echo
|
||||
echo "variant | mmdebstrap | debootstrap"
|
||||
echo "--------+------------+------------"
|
||||
echo "minbase | $var1 | $var2"
|
||||
echo "buildd | $var3 | $var4"
|
||||
if [ "$dist" != 'stable' ]; then
|
||||
echo "- | $var5 | $var6"
|
||||
fi
|
||||
|
||||
eval $(awk '{print "var"NR"="$1}' sizes)
|
||||
|
||||
echo
|
||||
echo "sizes"
|
||||
echo "======="
|
||||
echo
|
||||
echo "variant | mmdebstrap | debootstrap"
|
||||
echo "--------+------------+------------"
|
||||
echo "minbase | $var1 | $var2"
|
||||
echo "buildd | $var3 | $var4"
|
||||
if [ "$dist" != 'stable' ]; then
|
||||
echo "- | $var5 | $var6"
|
||||
fi
|
||||
|
||||
rm timings sizes
|
||||
done
|
||||
|
||||
kill $pid
|
||||
|
||||
wait $pid || true
|
||||
|
||||
trap - INT QUIT TERM EXIT
|
||||
|
||||
Loading…
Reference in a new issue