josch/mmdebstrap
2
3
Fork 6
Code Issues 8 Pull requests 1 Projects Releases Wiki Activity

Compare commits

base: josch:main
Branches Tags
josch:main
boklm:fix_apt-transport-mirror
boklm:main
josch:1.5.4
josch:1.5.3
josch:1.5.2
josch:1.5.1
josch:1.5.0
josch:1.4.3
josch:1.4.2
josch:1.4.1
josch:1.4.0
josch:1.3.8
josch:1.3.7
josch:1.3.6
josch:1.3.5
josch:1.3.4
josch:1.3.3
josch:1.3.2
josch:1.3.1
josch:1.3.0
josch:1.2.5
josch:1.2.4
josch:1.2.3
josch:1.2.2
josch:1.2.1
josch:1.2.0
josch:1.1.0
josch:1.0.1
josch:1.0.0
josch:0.8.6
josch:0.8.5
josch:0.8.4
josch:0.8.3
josch:0.8.2
josch:0.8.1
josch:0.8.0
josch:0.7.5
josch:0.7.4
josch:0.7.3
josch:0.7.2
josch:0.7.1
josch:0.7.0
josch:0.6.1
josch:0.6.0
josch:0.5.1
josch:0.5.0
josch:0.4.1
josch:0.4.0
josch:0.3.0
josch:0.2.0
josch:0.1.0
..
compare: boklm:main
Branches Tags
boklm:fix_apt-transport-mirror
boklm:main
josch:main
josch:1.5.4
josch:1.5.3
josch:1.5.2
josch:1.5.1
josch:1.5.0
josch:1.4.3
josch:1.4.2
josch:1.4.1
josch:1.4.0
josch:1.3.8
josch:1.3.7
josch:1.3.6
josch:1.3.5
josch:1.3.4
josch:1.3.3
josch:1.3.2
josch:1.3.1
josch:1.3.0
josch:1.2.5
josch:1.2.4
josch:1.2.3
josch:1.2.2
josch:1.2.1
josch:1.2.0
josch:1.1.0
josch:1.0.1
josch:1.0.0
josch:0.8.6
josch:0.8.5
josch:0.8.4
josch:0.8.3
josch:0.8.2
josch:0.8.1
josch:0.8.0
josch:0.7.5
josch:0.7.4
josch:0.7.3
josch:0.7.2
josch:0.7.1
josch:0.7.0
josch:0.6.1
josch:0.6.0
josch:0.5.1
josch:0.5.0
josch:0.4.1
josch:0.4.0
josch:0.3.0
josch:0.2.0
josch:0.1.0

No commits in common. "main" and "main" have entirely different histories.
main ... main

159 changed files with 6797 additions and 11296 deletions
Show stats Expand all files Collapse all files

6
.mailmap
View file

@ -1,6 +0,0 @@
Johannes Schauer Marin Rodrigues <josch@mister-muffin.de>
Johannes Schauer Marin Rodrigues <josch@mister-muffin.de> <j.schauer@email.de>
Johannes Schauer Marin Rodrigues <josch@mister-muffin.de> <josch@debian.org>
Johannes Schauer Marin Rodrigues <josch@mister-muffin.de> <Johannes Schauer Marin Rodrigues josch@debian.org>
Helmut Grohne <helmut@subdivi.de> <helmut.grohne@intenta.de>
Benjamin Drung <benjamin.drung@ionos.com> <benjamin.drung@cloud.ionos.com>

254
CHANGELOG.md
View file

@ -1,257 +1,3 @@
1.5.4 (2024-10-28)
------------------
- do not generate apt sources.list entry if SUITE is empty
1.5.3 (2024-09-13)
------------------
- tidy up any zombie processes
- chrootless hurd-i386
- add --skip=cleanup/reproducible/machine-id
- m-a-b-q: replace test_installed by dpkg-checkbuilddeps
1.5.2 (2024-06-26)
------------------
- mmdebstrap-autopkgtest-build-qemu produces bit-by-bit reproducible output
1.5.1 (2024-06-03)
------------------
- in root and unshare mode, run 'mount --make-rprivate /' before bind-mounting
- switch apt variant from using 'apt-get dist-upgrade' to apt patterns
1.5.0 (2024-05-14)
------------------
- add --format=ext4
1.4.3 (2024-02-01)
------------------
- take hard links into account when computing disk usage
1.4.2 (2024-01-29)
------------------
- allow for start-stop-daemon to be in either /sbin or /usr/sbin
- mmdebstrap-autopkgtest-build-qemu: fix octal mode computation and hostname
1.4.1 (2024-01-09)
------------------
- set DPkg::Chroot-Directory in APT_CONFIG to simplify calling apt in hooks
- disallow running chrootless as root without fakeroot unless
--skip=check/chrootless is used
- only print short --help output if wrong args are passed
- read files passed as --aptopt and --dpkgopt outside the unshared namespace
1.4.0 (2023-10-24)
------------------
- add mmdebstrap-autopkgtest-build-qemu
- export container=mmdebstrap-unshare env variable in unshare-mode hooks
- add new skip options: output/dev, output/mknod, tar-in/mknod,
copy-in/mknod, sync-in/mknod
- stop copying qemu-$arch-static binary into the chroot
- tarfilter: add --type-exclude option
- set MMDEBSTRAP_FORMAT in hooks
- do not install priority:required in buildd variant following debootstrap
1.3.8 (2023-08-20)
------------------
- hooks/merged-usr: implement post-merging as debootstrap does
- exclude ./lost+found from tarball
1.3.7 (2023-06-21)
------------------
- add hooks/copy-host-apt-sources-and-preferences
1.3.6 (2023-06-16)
------------------
- bugfix release
1.3.5 (2023-03-20)
------------------
- bugfix release
1.3.4 (2023-03-16)
------------------
- more safeguards before automatically choosing unshare mode
1.3.3 (2023-02-19)
------------------
- testsuite improvements
1.3.2 (2023-02-16)
------------------
- unshare mode works in privileged docker containers
1.3.1 (2023-01-20)
------------------
- bugfix release
1.3.0 (2023-01-16)
------------------
- add hooks/maybe-jessie-or-older and hooks/maybe-merged-usr
- add --skip=check/signed-by
- hooks/jessie-or-older: split into two individual hook files
- skip running apt-get update if we are very sure that it was already run
- be more verbose when 'apt-get update' failed
- warn if a hook is named like one but not executable and if a hook is
executable but not named like one
- to find signed-by value, run gpg on the individual keys to print better
error messages in case it fails (gpg doesn't give an indication which file
it was unable to read) and print progress bar
- allow empty sources.list entries
1.2.5 (2023-01-04)
------------------
- bugfix release
1.2.4 (2022-12-23)
------------------
- bugfix release
- add jessie-or-older extract hook
1.2.3 (2022-11-16)
------------------
- use Text::ParseWords::shellwords instead of spawning a new shell
- mount and unmount once, instead for each run_chroot() call
1.2.2 (2022-10-27)
------------------
- allow /etc/apt/trusted.gpg.d/ not to exist
- always create /var/lib/dpkg/arch to make foreign architecture chrootless
tarballs bit-by-bit identical
- write an empty /etc/machine-id instead of writing 'uninitialized'
- only print progress bars on interactive terminals that are wide enough
1.2.1 (2022-09-08)
------------------
- bugfix release
1.2.0 (2022-09-05)
------------------
- remove proot mode
- error out if stdout is an interactive terminal
- replace taridshift by tarfilter --idshift
- tarfilter: add --transform option
- multiple --skip options can be separated by comma or whitespace
- also cleanup the contents of /run
- support apt patterns and paths with commas and whitespace in --include
- hooks: store the values of the --include option in MMDEBSTRAP_INCLUDE
- add new --skip options: chroot/start-stop-daemon, chroot/policy-rc.d
chroot/mount, chroot/mount/dev, chroot/mount/proc, chroot/mount/sys,
cleanup/run
1.1.0 (2022-07-26)
----------------
- mount a new /dev/pts instance into the chroot to make posix_openpt work
- adjust merged-/usr hook to work the same way as debootstrap
- add no-merged-usr hook
1.0.1 (2022-05-29)
------------------
- bugfix release
1.0.0 (2022-05-28)
------------------
- all documented interfaces are now considered stable
- allow file:// mirrors
- /var/cache/apt/archives/ is now allowed to contain *.deb packages
- add file-mirror-automount hook-dir
- set $MMDEBSTRAP_VERBOSITY in hooks
- rewrite coverage with multiple individual and skippable shell scripts
0.8.6 (2022-03-25)
------------------
- allow running root mode inside unshare mode
0.8.5 (2022-03-07)
------------------
- improve documentation
0.8.4 (2022-02-11)
------------------
- tarfilter: add --strip-components option
- don't install essential packages in run_install()
- remove /var/lib/dbus/machine-id
0.8.3 (2022-01-08)
------------------
- allow codenames with apt patterns (requires apt >= 2.3.14)
- don't overwrite existing files in setup code
- don't copy in qemu-user-static binary if it's not needed
0.8.2 (2021-12-14)
------------------
- use apt patterns to select priority variants (requires apt >= 2.3.10)
0.8.1 (2021-10-07)
------------------
- enforce dpkg >= 1.20.0 and apt >= 2.3.7
- allow working directory be not world readable
- do not run xz and zstd with --threads=0 since this is a bad default for
machines with more than 100 cores
- bit-by-bit identical chrootless mode
0.8.0 (2021-09-21)
------------------
- allow running inside chroot in root mode
- allow running without /dev, /sys or /proc
- new --format=null which gets automatically selected if the output is
/dev/null and doesn't produce a tarball or other permanent output
- allow ASCII-armored keyrings (requires gnupg >= 2.2.8)
- run zstd with --threads=0
- tarfilter: add --pax-exclude and --pax-include to strip extended attributes
- add --skip=setup, --skip=update and --skip=cleanup
- add --skip=cleanup/apt/lists and --skip=cleanup/apt/cache
- pass extended attributes (excluding system) to tar2sqfs
- use apt-get update -error-on=any (requires apt >= 2.1.16)
- support Debian 11 Buster
- use apt from outside using DPkg::Chroot-Directory (requires apt >= 2.3.7)
* build chroots without apt (for example from buildinfo files)
* no need to install additional packages like apt-transport-* or
ca-certificates inside the chroot
* no need for additional key material inside the chroot
* possible use of file:// and copy://
- use apt pattern to select essential set
- write 'uninitialized' to /etc/machine-id
- allow running in root mode without mount working, either because of missing
CAP_SYS_ADMIN or missing /usr/bin/mount
- make /etc/ld.so.cache under fakechroot mode bit-by-bit identical to root
and unshare mode
- move hooks/setup00-merged-usr.sh to hooks/merged-usr/setup00.sh
- add gpgvnoexpkeysig script for very old snapshot.d.o timestamps with expired
signature
0.7.5 (2021-02-06)
------------------

49
README.md
View file

@ -23,11 +23,6 @@ For the full documentation use:
pod2man ./mmdebstrap | man -l -
Or read a HTML version of the man page in either of these locations:
- https://gitlab.mister-muffin.de/josch/mmdebstrap/wiki
- https://manpages.debian.org/unstable/mmdebstrap/mmdebstrap.1.en.html
The sales pitch in comparison to debootstrap
--------------------------------------------
@ -39,12 +34,11 @@ Summary:
- chroot with apt in 11 seconds
- gzipped tarball with apt is 27M small
- bit-by-bit reproducible output
- unprivileged operation using Linux user namespaces or fakechroot
- unprivileged operation using Linux user namespaces, fakechroot or proot
- can operate on filesystems mounted with nodev
- foreign architecture chroots with qemu-user
- variant installing only Essential:yes packages and dependencies
- temporary chroots by redirecting to /dev/null
- chroots without apt inside (for chroot from buildinfo file with debootsnap)
The author believes that a chroot of a Debian stable release should include the
latest packages including security fixes by default. This has been a wontfix
@ -81,11 +75,11 @@ reproducible** if the `$SOURCE_DATE_EPOCH` environment variable is set.
The author believes, that it should not be necessary to have superuser
privileges to create a file (the chroot tarball) in one's home directory.
Thus, mmdebstrap provides multiple options to create a chroot tarball with the
right permissions **without superuser privileges**. This avoids a whole class
of bugs like #921815. Depending on what is available, it uses either Linux user
namespaces or fakechroot. Debootstrap supports fakechroot but will not
create a tarball with the right permissions by itself. Support for Linux user
namespaces is missing (see #829134).
right permissions **without superuser privileges**. Depending on what is
available, it uses either Linux user namespaces, fakechroot or proot.
Debootstrap supports fakechroot but will not create a tarball with the right
permissions by itself. Support for Linux user namespaces and proot is missing
(see bugs #829134 and #698347, respectively).
When creating a chroot tarball with debootstrap, the temporary chroot directory
cannot be on a filesystem that has been mounted with nodev. In unprivileged
@ -99,19 +93,13 @@ Limitations in comparison to debootstrap
----------------------------------------
Debootstrap supports creating a Debian chroot on non-Debian systems but
mmdebstrap requires apt and is thus limited to Debian and derivatives. This
means that mmdebstrap can never fully replace debootstrap and debootstrap will
continue to be relevant in situations where you want to create a Debian chroot
from a platform without apt and dpkg.
mmdebstrap requires apt and is thus limited to Debian and derivatives.
There is no `SCRIPT` argument.
The following options, don't exist: `--second-stage`, `--exclude`,
`--resolve-deps`, `--force-check-gpg`, `--merged-usr` and `--no-merged-usr`.
The quirks from debootstrap are needed to create chroots of Debian unstable
from snapshot.d.o before timestamp 20141107T220431Z or Debian 8 (Jessie) or
later.
Tests
=====
@ -142,11 +130,6 @@ By default, `coverage.sh` will skip running a single test which tries creating
a Ubuntu Focal chroot. To not skip that test, run `coverage.sh` with the
environment variable `ONLINE=yes`.
If a test fails you can run individual tests by executing `coverage.py` with
the test name and optionally limit it to a specific distribution like so:
CMD=./mmdebstrap ./coverage.py --dist unstable check-against-debootstrap-dist
Bugs
====
@ -156,23 +139,7 @@ https://gitlab.mister-muffin.de/josch/mmdebstrap/issues
Contributors
============
- Johannes Schauer Marin Rodrigues (main author)
- Johannes Schauer (main author)
- Helmut Grohne
- Jochen Sprickerhof
- Gioele Barabucci
- Benjamin Drung
- Josh Triplett
- Konstantin Demin
- Chris Hofstaedtler
- Colin Watson
- David Kalnischkies
- Emilio Pozuelo Monfort
- Francesco Poli
- Jakub Wilk
- Joe Groocock
- Max-Julian Pogner
- Nicolas Vigier
- Raul Tambre
- Steve Dodd
- Trent W. Buck
- Vagrant Cascadian

118
caching_proxy.py
View file

@ -1,118 +0,0 @@
#!/usr/bin/env python3
import sys
import os
import time
import http.client
import http.server
from io import StringIO
import pathlib
import urllib.parse
oldcachedir = None
newcachedir = None
readonly = False
class ProxyRequestHandler(http.server.BaseHTTPRequestHandler):
def do_GET(self):
assert int(self.headers.get("Content-Length", 0)) == 0
assert self.headers["Host"]
pathprefix = "http://" + self.headers["Host"] + "/"
assert self.path.startswith(pathprefix)
sanitizedpath = urllib.parse.unquote(self.path.removeprefix(pathprefix))
oldpath = oldcachedir / sanitizedpath
newpath = newcachedir / sanitizedpath
if not readonly:
newpath.parent.mkdir(parents=True, exist_ok=True)
# just send back to client
if newpath.exists():
print(f"proxy cached: {self.path}", file=sys.stderr)
self.wfile.write(b"HTTP/1.1 200 OK\r\n")
self.send_header("Content-Length", newpath.stat().st_size)
self.end_headers()
with newpath.open(mode="rb") as new:
while True:
buf = new.read(64 * 1024) # same as shutil uses
if not buf:
break
self.wfile.write(buf)
self.wfile.flush()
return
if readonly:
newpath = pathlib.Path("/dev/null")
# copy from oldpath to newpath and send back to client
# Only take files from the old cache if they are .deb files or Packages
# files in the by-hash directory as only those are unique by their path
# name. Other files like InRelease files have to be downloaded afresh.
if oldpath.exists() and (
oldpath.suffix == ".deb" or "by-hash" in oldpath.parts
):
print(f"proxy cached: {self.path}", file=sys.stderr)
self.wfile.write(b"HTTP/1.1 200 OK\r\n")
self.send_header("Content-Length", oldpath.stat().st_size)
self.end_headers()
with oldpath.open(mode="rb") as old, newpath.open(mode="wb") as new:
# we are not using shutil.copyfileobj() because we want to
# write to two file objects simultaneously
while True:
buf = old.read(64 * 1024) # same as shutil uses
if not buf:
break
self.wfile.write(buf)
new.write(buf)
self.wfile.flush()
return
# download fresh copy
try:
print(f"\rproxy download: {self.path}", file=sys.stderr)
conn = http.client.HTTPConnection(self.headers["Host"], timeout=5)
conn.request("GET", self.path, None, dict(self.headers))
res = conn.getresponse()
assert (res.status, res.reason) == (200, "OK"), (res.status, res.reason)
self.wfile.write(b"HTTP/1.1 200 OK\r\n")
for k, v in res.getheaders():
# do not allow a persistent connection
if k == "connection":
continue
self.send_header(k, v)
self.end_headers()
with newpath.open(mode="wb") as f:
# we are not using shutil.copyfileobj() because we want to
# write to two file objects simultaneously and throttle the
# writing speed to 1024 kB/s
while True:
buf = res.read(64 * 1024) # same as shutil uses
if not buf:
break
self.wfile.write(buf)
f.write(buf)
time.sleep(64 / 1024) # 1024 kB/s
self.wfile.flush()
except Exception as e:
self.send_error(502)
def main():
global oldcachedir, newcachedir, readonly
if sys.argv[1] == "--readonly":
readonly = True
oldcachedir = pathlib.Path(sys.argv[2])
newcachedir = pathlib.Path(sys.argv[3])
else:
oldcachedir = pathlib.Path(sys.argv[1])
newcachedir = pathlib.Path(sys.argv[2])
print(f"starting caching proxy for {newcachedir}", file=sys.stderr)
httpd = http.server.ThreadingHTTPServer(
server_address=("", 8080), RequestHandlerClass=ProxyRequestHandler
)
httpd.serve_forever()
if __name__ == "__main__":
main()

474
coverage.py
View file

@ -1,474 +0,0 @@
#!/usr/bin/env python3
from debian.deb822 import Deb822, Release
import email.utils
import os
import sys
import shutil
import subprocess
import argparse
import time
from datetime import timedelta
from collections import defaultdict
from itertools import product
have_qemu = os.getenv("HAVE_QEMU", "yes") == "yes"
have_binfmt = os.getenv("HAVE_BINFMT", "yes") == "yes"
run_ma_same_tests = os.getenv("RUN_MA_SAME_TESTS", "yes") == "yes"
use_host_apt_config = os.getenv("USE_HOST_APT_CONFIG", "no") == "yes"
cmd = os.getenv("CMD", "./mmdebstrap")
default_dist = os.getenv("DEFAULT_DIST", "unstable")
all_dists = ["oldstable", "stable", "testing", "unstable"]
default_mode = "auto"
all_modes = ["auto", "root", "unshare", "fakechroot", "chrootless"]
default_variant = "apt"
all_variants = [
"extract",
"custom",
"essential",
"apt",
"minbase",
"buildd",
"-",
"standard",
]
default_format = "auto"
all_formats = ["auto", "directory", "tar", "squashfs", "ext2", "ext4", "null"]
mirror = os.getenv("mirror", "http://127.0.0.1/debian")
hostarch = subprocess.check_output(["dpkg", "--print-architecture"]).decode().strip()
release_path = f"./shared/cache/debian/dists/{default_dist}/InRelease"
if not os.path.exists(release_path):
print("path doesn't exist:", release_path, file=sys.stderr)
print("run ./make_mirror.sh first", file=sys.stderr)
exit(1)
if os.getenv("SOURCE_DATE_EPOCH") is not None:
s_d_e = os.getenv("SOURCE_DATE_EPOCH")
else:
with open(release_path) as f:
rel = Release(f)
s_d_e = str(email.utils.mktime_tz(email.utils.parsedate_tz(rel["Date"])))
separator = (
"------------------------------------------------------------------------------"
)
def skip(condition, dist, mode, variant, fmt):
if not condition:
return ""
for line in condition.splitlines():
if not line:
continue
if eval(line):
return line.strip()
return ""
def parse_config(confname):
config_dict = defaultdict(dict)
config_order = list()
all_vals = {
"Dists": all_dists,
"Modes": all_modes,
"Variants": all_variants,
"Formats": all_formats,
}
with open(confname) as f:
for test in Deb822.iter_paragraphs(f):
if "Test" not in test.keys():
print("Test without name", file=sys.stderr)
exit(1)
name = test["Test"]
config_order.append(name)
for k in test.keys():
v = test[k]
if k not in [
"Test",
"Dists",
"Modes",
"Variants",
"Formats",
"Skip-If",
"Needs-QEMU",
"Needs-Root",
"Needs-APT-Config",
]:
print(f"Unknown field name {k} in test {name}")
exit(1)
if k in all_vals.keys():
if v == "default":
print(
f"Setting {k} to default in Test {name} is redundant",
file=sys.stderr,
)
exit(1)
if v == "any":
v = all_vals[k]
else:
# else, split the value by whitespace
v = v.split()
for i in v:
if i not in all_vals[k]:
print(
f"{i} is not a valid value for {k}", file=sys.stderr
)
exit(1)
config_dict[name][k] = v
return config_order, config_dict
def format_test(num, total, name, dist, mode, variant, fmt, config_dict):
ret = f"({num}/{total}) {name}"
if len(config_dict[name].get("Dists", [])) > 1:
ret += f" --dist={dist}"
if len(config_dict[name].get("Modes", [])) > 1:
ret += f" --mode={mode}"
if len(config_dict[name].get("Variants", [])) > 1:
ret += f" --variant={variant}"
if len(config_dict[name].get("Formats", [])) > 1:
ret += f" --format={fmt}"
return ret
def print_time_per_test(time_per_test, name="test"):
print(
f"average time per {name}:",
sum(time_per_test.values(), start=timedelta()) / len(time_per_test),
file=sys.stderr,
)
print(
f"median time per {name}:",
sorted(time_per_test.values())[len(time_per_test) // 2],
file=sys.stderr,
)
head_tail_num = 10
print(f"{head_tail_num} fastests {name}s:", file=sys.stderr)
for k, v in sorted(time_per_test.items(), key=lambda i: i[1])[
: min(head_tail_num, len(time_per_test))
]:
print(f" {k}: {v}", file=sys.stderr)
print(f"{head_tail_num} slowest {name}s:", file=sys.stderr)
for k, v in sorted(time_per_test.items(), key=lambda i: i[1], reverse=True)[
: min(head_tail_num, len(time_per_test))
]:
print(f" {k}: {v}", file=sys.stderr)
def main():
parser = argparse.ArgumentParser()
parser.add_argument("test", nargs="*", help="only run these tests")
parser.add_argument(
"-x",
"--exitfirst",
action="store_const",
dest="maxfail",
const=1,
help="exit instantly on first error or failed test.",
)
parser.add_argument(
"--maxfail",
metavar="num",
action="store",
type=int,
dest="maxfail",
default=0,
help="exit after first num failures or errors.",
)
parser.add_argument(
"--mode",
metavar="mode",
help=f"only run tests with this mode (Default = {default_mode})",
)
parser.add_argument(
"--dist",
metavar="dist",
help=f"only run tests with this dist (Default = {default_dist})",
)
parser.add_argument(
"--variant",
metavar="variant",
help=f"only run tests with this variant (Default = {default_variant})",
)
parser.add_argument(
"--format",
metavar="format",
help=f"only run tests with this format (Default = {default_format})",
)
parser.add_argument(
"--skip", metavar="test", action="append", help="skip this test"
)
args = parser.parse_args()
# copy over files from git or as distributed
for git, dist, target in [
("./mmdebstrap", "/usr/bin/mmdebstrap", "mmdebstrap"),
("./tarfilter", "/usr/bin/mmtarfilter", "tarfilter"),
(
"./proxysolver",
"/usr/lib/apt/solvers/mmdebstrap-dump-solution",
"proxysolver",
),
(
"./ldconfig.fakechroot",
"/usr/libexec/mmdebstrap/ldconfig.fakechroot",
"ldconfig.fakechroot",
),
]:
if os.path.exists(git):
shutil.copy(git, f"shared/{target}")
else:
shutil.copy(dist, f"shared/{target}")
# copy over hooks from git or as distributed
if os.path.exists("hooks"):
shutil.copytree("hooks", "shared/hooks", dirs_exist_ok=True)
else:
shutil.copytree(
"/usr/share/mmdebstrap/hooks", "shared/hooks", dirs_exist_ok=True
)
# parse coverage.txt
config_order, config_dict = parse_config("coverage.txt")
indirbutnotcovered = set(
[d for d in os.listdir("tests") if not d.startswith(".")]
) - set(config_order)
if indirbutnotcovered:
print(
"test(s) missing from coverage.txt: %s"
% (", ".join(sorted(indirbutnotcovered))),
file=sys.stderr,
)
exit(1)
coveredbutnotindir = set(config_order) - set(
[d for d in os.listdir("tests") if not d.startswith(".")]
)
if coveredbutnotindir:
print(
"test(s) missing from ./tests: %s"
% (", ".join(sorted(coveredbutnotindir))),
file=sys.stderr,
)
exit(1)
# produce the list of tests using the cartesian product of all allowed
# dists, modes, variants and formats of a given test
tests = []
for name in config_order:
test = config_dict[name]
for dist, mode, variant, fmt in product(
test.get("Dists", [default_dist]),
test.get("Modes", [default_mode]),
test.get("Variants", [default_variant]),
test.get("Formats", [default_format]),
):
skipreason = skip(test.get("Skip-If"), dist, mode, variant, fmt)
if skipreason:
tt = ("skip", skipreason)
elif (
test.get("Needs-APT-Config", "false") == "true" and use_host_apt_config
):
tt = ("skip", "test cannot use host apt config")
elif have_qemu:
tt = "qemu"
elif test.get("Needs-QEMU", "false") == "true":
tt = ("skip", "test needs QEMU")
elif test.get("Needs-Root", "false") == "true":
tt = "sudo"
elif mode == "root":
tt = "sudo"
else:
tt = "null"
tests.append((tt, name, dist, mode, variant, fmt))
torun = []
num_tests = len(tests)
if args.test:
# check if all given tests are either a valid name or a valid number
for test in args.test:
if test in [name for (_, name, _, _, _, _) in tests]:
continue
if not test.isdigit():
print(f"cannot find test named {test}", file=sys.stderr)
exit(1)
if int(test) >= len(tests) or int(test) <= 0 or str(int(test)) != test:
print(f"test number {test} doesn't exist", file=sys.stderr)
exit(1)
for i, (_, name, _, _, _, _) in enumerate(tests):
# if either the number or test name matches, then we use this test,
# otherwise we skip it
if name in args.test:
torun.append(i)
if str(i + 1) in args.test:
torun.append(i)
num_tests = len(torun)
starttime = time.time()
skipped = defaultdict(list)
failed = []
num_success = 0
num_finished = 0
time_per_test = {}
acc_time_per_test = defaultdict(list)
for i, (test, name, dist, mode, variant, fmt) in enumerate(tests):
if torun and i not in torun:
continue
print(separator, file=sys.stderr)
print("(%d/%d) %s" % (i + 1, len(tests), name), file=sys.stderr)
print("dist: %s" % dist, file=sys.stderr)
print("mode: %s" % mode, file=sys.stderr)
print("variant: %s" % variant, file=sys.stderr)
print("format: %s" % fmt, file=sys.stderr)
if num_finished > 0:
currenttime = time.time()
timeleft = timedelta(
seconds=int(
(num_tests - num_finished)
* (currenttime - starttime)
/ num_finished
)
)
print("time left: %s" % timeleft, file=sys.stderr)
if failed:
print("failed: %d" % len(failed), file=sys.stderr)
num_finished += 1
with open("tests/" + name) as fin, open("shared/test.sh", "w") as fout:
for line in fin:
line = line.replace("{{ CMD }}", cmd)
line = line.replace("{{ SOURCE_DATE_EPOCH }}", s_d_e)
line = line.replace("{{ DIST }}", dist)
line = line.replace("{{ MIRROR }}", mirror)
line = line.replace("{{ MODE }}", mode)
line = line.replace("{{ VARIANT }}", variant)
line = line.replace("{{ FORMAT }}", fmt)
line = line.replace("{{ HOSTARCH }}", hostarch)
fout.write(line)
# ignore:
# SC2016 Expressions don't expand in single quotes, use double quotes for that.
# SC2050 This expression is constant. Did you forget the $ on a variable?
# SC2194 This word is constant. Did you forget the $ on a variable?
shellcheck = subprocess.run(
[
"shellcheck",
"--exclude=SC2050,SC2194,SC2016",
"-f",
"gcc",
"shared/test.sh",
],
check=False,
stdout=subprocess.PIPE,
).stdout.decode()
shfmt = subprocess.run(
[
"shfmt",
"--posix",
"--binary-next-line",
"--case-indent",
"--indent",
"2",
"--simplify",
"-d",
"shared/test.sh",
],
check=False,
stdout=subprocess.PIPE,
).stdout.decode()
argv = None
match test:
case "qemu":
argv = ["./run_qemu.sh"]
case "sudo":
argv = ["./run_null.sh", "SUDO"]
case "null":
argv = ["./run_null.sh"]
case ("skip", reason):
skipped[reason].append(
format_test(
i + 1, len(tests), name, dist, mode, variant, fmt, config_dict
)
)
print(f"skipped because of {reason}", file=sys.stderr)
continue
print(separator, file=sys.stderr)
if args.skip and name in args.skip:
print(f"skipping because of --skip={name}", file=sys.stderr)
continue
if args.dist and args.dist != dist:
print(f"skipping because of --dist={args.dist}", file=sys.stderr)
continue
if args.mode and args.mode != mode:
print(f"skipping because of --mode={args.mode}", file=sys.stderr)
continue
if args.variant and args.variant != variant:
print(f"skipping because of --variant={args.variant}", file=sys.stderr)
continue
if args.format and args.format != fmt:
print(f"skipping because of --format={args.format}", file=sys.stderr)
continue
before = time.time()
proc = subprocess.Popen(argv)
try:
proc.wait()
except KeyboardInterrupt:
proc.terminate()
proc.wait()
break
after = time.time()
walltime = timedelta(seconds=int(after - before))
formated_test_name = format_test(
i + 1, len(tests), name, dist, mode, variant, fmt, config_dict
)
time_per_test[formated_test_name] = walltime
acc_time_per_test[name].append(walltime)
print(separator, file=sys.stderr)
print(f"duration: {walltime}", file=sys.stderr)
if proc.returncode != 0 or shellcheck != "" or shfmt != "":
if shellcheck != "":
print(shellcheck)
if shfmt != "":
print(shfmt)
failed.append(formated_test_name)
print("result: FAILURE", file=sys.stderr)
else:
print("result: SUCCESS", file=sys.stderr)
num_success += 1
if args.maxfail and len(failed) >= args.maxfail:
break
print(separator, file=sys.stderr)
print(
"successfully ran %d tests" % num_success,
file=sys.stderr,
)
if skipped:
print("skipped %d:" % sum([len(v) for v in skipped.values()]), file=sys.stderr)
for reason, l in skipped.items():
print(f"skipped because of {reason}:", file=sys.stderr)
for t in l:
print(f" {t}", file=sys.stderr)
if len(time_per_test) > 1:
print_time_per_test(time_per_test)
if len(acc_time_per_test) > 1:
print_time_per_test(
{
f"{len(v)}x {k}": sum(v, start=timedelta())
for k, v in acc_time_per_test.items()
},
"accumulated test",
)
if failed:
print("failed %d:" % len(failed), file=sys.stderr)
for f in failed:
print(f, file=sys.stderr)
currenttime = time.time()
walltime = timedelta(seconds=int(currenttime - starttime))
print(f"total runtime: {walltime}", file=sys.stderr)
if failed:
exit(1)
if __name__ == "__main__":
main()

3809
coverage.sh
View file

File diff suppressed because it is too large Load diff

437
coverage.txt
View file

@ -1,437 +0,0 @@
Test: debootstrap
Dists: any
Variants: minbase buildd -
Needs-Root: true
Needs-APT-Config: true
Test: check-against-debootstrap-dist
Dists: any
Variants: minbase buildd -
Needs-Root: true
Needs-APT-Config: true
Test: as-debootstrap-unshare-wrapper
Modes: unshare
Needs-Root: true
Variants: minbase -
Needs-APT-Config: true
Test: help
Test: man
Test: version
Test: create-directory
Needs-Root: true
Test: unshare-as-root-user
Needs-Root: true
Test: dist-using-codename
Dists: any
Needs-APT-Config: true
Test: fail-without-etc-subuid
Needs-QEMU: true
Test: fail-without-username-in-etc-subuid
Needs-QEMU: true
Test: unshare-as-root-user-inside-chroot
Needs-Root: true
Needs-APT-Config: true
Test: root-mode-inside-chroot
Needs-Root: true
Needs-APT-Config: true
Test: root-mode-inside-unshare-chroot
Modes: unshare
Needs-APT-Config: true
Test: root-without-cap-sys-admin
Needs-Root: true
Test: mount-is-missing
Needs-QEMU: true
Test: mmdebstrap
Needs-Root: true
Modes: root
Formats: tar squashfs ext2 ext4
Variants: essential apt minbase buildd - standard
Skip-If:
variant == "standard" and dist == "oldstable" # #864082, #1004557, #1004558
mode == "fakechroot" and variant in ["-", "standard"] # no extended attributes
variant == "standard" and dist in ["oldstable", "stable"] and hostarch in ["armel", "armhf", "mipsel"] # #1031276
Test: check-for-bit-by-bit-identical-format-output
Modes: unshare fakechroot
Formats: tar squashfs ext2 ext4
Variants: essential apt minbase buildd - standard
Skip-If:
variant == "standard" and dist == "oldstable" # #864082, #1004557, #1004558
mode == "fakechroot" and variant in ["-", "standard"] # no extended attributes
variant == "standard" and dist in ["oldstable", "stable"] and hostarch in ["armel", "armhf", "mipsel"] # #1031276
Test: tarfilter-idshift
Needs-QEMU: true
Test: progress-bars-on-fake-tty
Test: debug-output-on-fake-tty
Test: existing-empty-directory
Needs-Root: true
Test: existing-directory-with-lost-found
Needs-Root: true
Test: fail-installing-to-non-empty-lost-found
Test: fail-installing-to-non-empty-target-directory
Test: missing-device-nodes-outside-the-chroot
Needs-QEMU: true
Test: missing-dev-sys-proc-inside-the-chroot
Modes: unshare
Variants: custom
Test: chroot-directory-not-accessible-by-apt-user
Needs-Root: true
Test: cwd-directory-not-accessible-by-unshared-user
Needs-Root: true
Modes: unshare
Test: create-gzip-compressed-tarball
Test: custom-tmpdir
Needs-Root: true
Modes: unshare
Test: xz-compressed-tarball
Test: directory-ending-in-tar
Modes: root
Needs-Root: true
Test: auto-mode-without-unshare-capabilities
Needs-QEMU: true
Test: fail-with-missing-lz4
Test: fail-with-path-with-quotes
Test: create-tarball-with-tmp-mounted-nodev
Needs-QEMU: true
Test: read-from-stdin-write-to-stdout
Test: supply-components-manually
Modes: root
Needs-Root: true
Needs-APT-Config: true
Test: stable-default-mirror
Needs-QEMU: true
Test: pass-distribution-but-implicitly-write-to-stdout
Needs-QEMU: true
Test: aspcud-apt-solver
Test: mirror-is-stdin
Test: copy-mirror
Needs-QEMU: true
Test: file-mirror
Needs-QEMU: true
Test: file-mirror-automount-hook
Modes: root unshare fakechroot
Needs-QEMU: true
Test: mirror-is-deb
Test: mirror-is-real-file
Needs-APT-Config: true
Test: deb822-1-2
Modes: root
Needs-Root: true
Needs-APT-Config: true
Test: deb822-2-2
Modes: root
Needs-Root: true
Needs-APT-Config: true
Test: automatic-mirror-from-suite
Needs-QEMU: true
Test: invalid-mirror
Needs-APT-Config: true
Test: fail-installing-to-root
Modes: root
Needs-Root: true
Test: fail-installing-to-existing-file
Modes: root
Needs-Root: true
Test: arm64-without-qemu-support
Needs-QEMU: true
Skip-If: hostarch != "amd64"
Test: i386-which-can-be-executed-without-qemu
Needs-QEMU: true
Skip-If:
hostarch != "amd64"
not run_ma_same_tests
Test: include-foreign-libmagic-mgc
Needs-Root: true
Needs-APT-Config: true
Skip-If:
hostarch not in ["amd64", "arm64"]
not run_ma_same_tests
Test: include-foreign-libmagic-mgc-with-multiple-arch-options
Needs-Root: true
Needs-APT-Config: true
Skip-If:
hostarch not in ["amd64", "arm64"]
not run_ma_same_tests
Test: aptopt
Needs-Root: true
Test: keyring
Needs-QEMU: true
Test: keyring-overwrites
Needs-Root: true
Needs-APT-Config: true
Test: signed-by-without-host-keys
Needs-QEMU: true
Test: ascii-armored-keys
Needs-QEMU: true
Test: signed-by-with-host-keys
Needs-Root: true
Needs-APT-Config: true
Test: dpkgopt
Needs-Root: true
Test: include
Needs-Root: true
Test: multiple-include
Needs-Root: true
Test: include-with-multiple-apt-sources
Needs-Root: true
Test: essential-hook
Needs-Root: true
Test: customize-hook
Needs-Root: true
Test: failing-customize-hook
Needs-Root: true
Test: sigint-during-customize-hook
Needs-Root: true
Test: hook-directory
Needs-Root: true
Test: eatmydata-via-hook-dir
Needs-Root: true
Test: special-hooks-using-helpers
Needs-Root: true
Needs-APT-Config: true
Test: special-hooks-using-helpers-and-env-vars
Needs-Root: true
Needs-APT-Config: true
Test: special-hooks-with-mode-mode
Modes: root unshare fakechroot
Test: debootstrap-no-op-options
Needs-Root: true
Test: verbose
Variants: standard
Skip-If:
variant == "standard" and dist == "oldstable" # #864082, #1004557, #1004558
Test: debug
Variants: standard
Skip-If:
variant == "standard" and dist == "oldstable" # #864082, #1004557, #1004558
Test: quiet
Needs-Root: true
Test: logfile
Needs-Root: true
Needs-APT-Config: true
Test: without-etc-resolv-conf-and-etc-hostname
Needs-QEMU: true
Test: preserve-mode-of-etc-resolv-conf-and-etc-hostname
Modes: root
Needs-QEMU: true
Test: not-having-to-install-apt-in-include-because-a-hook-did-it-before
Test: remove-start-stop-daemon-and-policy-rc-d-in-hook
Test: skip-start-stop-daemon-policy-rc
Test: skip-mount
Modes: unshare
Test: compare-output-with-pre-seeded-var-cache-apt-archives
Needs-QEMU: true
Variants: any
Skip-If:
variant == "standard" and dist == "oldstable" # #864082, #1004557, #1004558
Test: create-directory-dry-run
Modes: root
Test: create-tarball-dry-run
Variants: any
Modes: any
Test: unpack-doc-debian
Modes: root fakechroot
Variants: extract
Needs-APT-Config: true
Test: install-doc-debian
Modes: chrootless
Variants: custom
Needs-APT-Config: true
Test: chrootless
Variants: essential
Modes: chrootless
Needs-Root: true
Skip-If:
dist == "oldstable"
Test: chrootless-fakeroot
Variants: essential
Modes: chrootless
Skip-If:
dist == "oldstable"
hostarch in ["i386", "armel", "armhf", "mipsel"] # #1023286
Test: chrootless-foreign
Variants: essential
Modes: chrootless
Skip-If:
dist == "oldstable"
hostarch not in ["amd64", "arm64"]
not run_ma_same_tests
Needs-QEMU: true
Test: install-doc-debian-and-output-tarball
Variants: custom
Modes: chrootless
Needs-APT-Config: true
Test: install-doc-debian-and-test-hooks
Variants: custom
Modes: chrootless
Needs-APT-Config: true
Test: install-libmagic-mgc-on-foreign
Variants: custom
Modes: chrootless
Skip-If:
hostarch not in ["amd64", "arm64"]
not have_binfmt
Test: install-busybox-based-sub-essential-system
Needs-Root: true
Test: create-foreign-tarball
Modes: root unshare fakechroot
Skip-If:
hostarch not in ["amd64", "arm64"]
mode == "fakechroot" and not run_ma_same_tests
mode == "fakechroot" and hostarch == "arm64" # usrmerge postinst under fakechroot wants to copy /lib/ld-linux-x86-64.so.2 (which does not exist) instead of /lib64/ld-linux-x86-64.so.2
not have_binfmt
Test: no-sbin-in-path
Modes: fakechroot
Test: dev-ptmx
Modes: root unshare
Test: error-if-stdout-is-tty
Test: variant-custom-timeout
Test: include-deb-file
Modes: root unshare fakechroot
Needs-APT-Config: true
Test: unshare-include-deb
Modes: unshare
Test: pivot_root
Modes: root unshare
Needs-APT-Config: true
Test: jessie-or-older
Needs-Root: true
Modes: root unshare fakechroot
Variants: essential apt minbase
Skip-If: mode == "fakechroot" and hostarch in ["i386", "armel", "armhf", "mipsel"] # #1023286
Test: apt-patterns
Test: apt-patterns-custom
Test: empty-sources.list
Test: merged-fakechroot-inside-unmerged-chroot
Needs-Root: true
Needs-APT-Config: true
Skip-If:
hostarch in ["i386", "armel", "armhf", "mipsel"] # #1023286
dist in ["testing", "unstable"] # #1053671
Test: auto-mode-as-normal-user
Modes: auto
Test: skip-output-dev
Modes: root unshare
Test: skip-output-mknod
Modes: root unshare
Test: skip-tar-in-mknod
Modes: unshare
Test: zombie-reaping
Modes: unshare
Test: empty-suite

128
examples/twb/debian-11-minimal.py
View file

@ -21,80 +21,68 @@ NOTE: this is the simplest config possible.
"""
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"output_file", nargs="?", default=pathlib.Path("filesystem.img"), type=pathlib.Path
)
parser.add_argument('output_file', nargs='?', default=pathlib.Path('filesystem.img'), type=pathlib.Path)
args = parser.parse_args()
filesystem_img_size = "256M" # big enough to include filesystem.squashfs + about 64M of bootloader, kernel, and ramdisk.
esp_offset = 1024 * 1024 # 1MiB
esp_label = "UEFI-ESP" # max 8 bytes for FAT32
live_media_path = "debian-live"
filesystem_img_size = '256M' # big enough to include filesystem.squashfs + about 64M of bootloader, kernel, and ramdisk.
esp_offset = 1024 * 1024 # 1MiB
esp_label = 'UEFI-ESP' # max 8 bytes for FAT32
live_media_path = 'debian-live'
with tempfile.TemporaryDirectory(prefix="debian-live-bullseye-amd64-minimal.") as td:
with tempfile.TemporaryDirectory(prefix='debian-live-bullseye-amd64-minimal.') as td:
td = pathlib.Path(td)
subprocess.check_call(
[
"mmdebstrap",
"--mode=unshare",
"--variant=apt",
'--aptopt=Acquire::http::Proxy "http://apt-cacher-ng.cyber.com.au:3142"',
'--aptopt=Acquire::https::Proxy "DIRECT"',
"--dpkgopt=force-unsafe-io",
"--include=linux-image-amd64 init initramfs-tools live-boot netbase",
"--include=dbus", # https://bugs.debian.org/814758
"--include=live-config iproute2 keyboard-configuration locales sudo user-setup",
"--include=ifupdown isc-dhcp-client", # live-config doesn't support systemd-networkd yet.
# Do the **BARE MINIMUM** to make a USB key that can boot on X86_64 UEFI.
# We use mtools so we do not ever need root privileges.
# We can't use mkfs.vfat, as that needs kpartx or losetup (i.e. root).
# We can't use mkfs.udf, as that needs mount (i.e. root).
# We can't use "refind-install --usedefault" as that runs mount(8) (i.e. root).
# We don't use genisoimage because
# 1) ISO9660 must die;
# 2) incomplete UDF 1.5+ support;
# 3) resulting filesystem can't be tweaked after flashing (e.g. debian-live/site.dir/etc/systemd/network/up.network).
#
# We use refind because 1) I hate grub; and 2) I like refind.
# If you want aarch64 or ia32 you need to install their BOOTxxx.EFI files.
# If you want kernel+initrd on something other than FAT, you need refind/drivers_xxx/xxx_xxx.EFI.
#
# FIXME: with qemu in UEFI mode (OVMF), I get dumped into startup.nsh (UEFI REPL).
# From there, I can manually type in "FS0:\EFI\BOOT\BOOTX64.EFI" to start refind, tho.
# So WTF is its problem? Does it not support fallback bootloader?
"--include=refind parted mtools",
"--essential-hook=echo refind refind/install_to_esp boolean false | chroot $1 debconf-set-selections",
"--customize-hook=echo refind refind/install_to_esp boolean true | chroot $1 debconf-set-selections",
"--customize-hook=chroot $1 mkdir -p /boot/USB /boot/EFI/BOOT",
"--customize-hook=chroot $1 cp /usr/share/refind/refind/refind_x64.efi /boot/EFI/BOOT/BOOTX64.EFI",
f"--customize-hook=chroot $1 truncate --size={filesystem_img_size} /boot/USB/filesystem.img",
f"--customize-hook=chroot $1 parted --script --align=optimal /boot/USB/filesystem.img mklabel gpt mkpart {esp_label} {esp_offset}b 100% set 1 esp on",
f"--customize-hook=chroot $1 mformat -i /boot/USB/filesystem.img@@{esp_offset} -F -v {esp_label}",
f"--customize-hook=chroot $1 mmd -i /boot/USB/filesystem.img@@{esp_offset} ::{live_media_path}",
f"""--customize-hook=echo '"Boot with default options" "boot=live live-media-path={live_media_path}"' >$1/boot/refind_linux.conf""",
# NOTE: find sidesteps the "glob expands before chroot applies" problem.
f"""--customize-hook=chroot $1 find -O3 /boot/ -xdev -mindepth 1 -maxdepth 1 -regextype posix-egrep -iregex '.*/(EFI|refind_linux.conf|vmlinuz.*|initrd.img.*)' -exec mcopy -vsbpm -i /boot/USB/filesystem.img@@{esp_offset} {{}} :: ';'""",
# FIXME: copy-out doesn't handle sparseness, so is REALLY slow (about 50 seconds).
# Therefore instead leave it in the squashfs, and extract it later.
# f'--customize-hook=copy-out /boot/USB/filesystem.img /tmp/',
# f'--customize-hook=chroot $1 rm /boot/USB/filesystem.img',
"bullseye",
td / "filesystem.squashfs",
]
)
['mmdebstrap',
'--mode=unshare',
'--variant=apt',
'--aptopt=Acquire::http::Proxy "http://apt-cacher-ng.cyber.com.au:3142"',
'--aptopt=Acquire::https::Proxy "DIRECT"',
'--dpkgopt=force-unsafe-io',
'--include=linux-image-amd64 init initramfs-tools live-boot netbase',
'--include=dbus', # https://bugs.debian.org/814758
'--include=live-config iproute2 keyboard-configuration locales sudo user-setup',
'--include=ifupdown isc-dhcp-client', # live-config doesn't support systemd-networkd yet.
with args.output_file.open("wb") as f:
subprocess.check_call(
["rdsquashfs", "--cat=boot/USB/filesystem.img", td / "filesystem.squashfs"],
stdout=f,
)
subprocess.check_call(
[
"mcopy",
"-i",
f"{args.output_file}@@{esp_offset}",
td / "filesystem.squashfs",
f"::{live_media_path}/filesystem.squashfs",
]
)
# Do the **BARE MINIMUM** to make a USB key that can boot on X86_64 UEFI.
# We use mtools so we do not ever need root privileges.
# We can't use mkfs.vfat, as that needs kpartx or losetup (i.e. root).
# We can't use mkfs.udf, as that needs mount (i.e. root).
# We can't use "refind-install --usedefault" as that runs mount(8) (i.e. root).
# We don't use genisoimage because
# 1) ISO9660 must die;
# 2) incomplete UDF 1.5+ support;
# 3) resulting filesystem can't be tweaked after flashing (e.g. debian-live/site.dir/etc/systemd/network/up.network).
#
# We use refind because 1) I hate grub; and 2) I like refind.
# If you want aarch64 or ia32 you need to install their BOOTxxx.EFI files.
# If you want kernel+initrd on something other than FAT, you need refind/drivers_xxx/xxx_xxx.EFI.
#
# FIXME: with qemu in UEFI mode (OVMF), I get dumped into startup.nsh (UEFI REPL).
# From there, I can manually type in "FS0:\EFI\BOOT\BOOTX64.EFI" to start refind, tho.
# So WTF is its problem? Does it not support fallback bootloader?
'--include=refind parted mtools',
'--essential-hook=echo refind refind/install_to_esp boolean false | chroot $1 debconf-set-selections',
'--customize-hook=echo refind refind/install_to_esp boolean true | chroot $1 debconf-set-selections',
'--customize-hook=chroot $1 mkdir -p /boot/USB /boot/EFI/BOOT',
'--customize-hook=chroot $1 cp /usr/share/refind/refind/refind_x64.efi /boot/EFI/BOOT/BOOTX64.EFI',
f'--customize-hook=chroot $1 truncate --size={filesystem_img_size} /boot/USB/filesystem.img',
f'--customize-hook=chroot $1 parted --script --align=optimal /boot/USB/filesystem.img mklabel gpt mkpart {esp_label} {esp_offset}b 100% set 1 esp on',
f'--customize-hook=chroot $1 mformat -i /boot/USB/filesystem.img@@{esp_offset} -F -v {esp_label}',
f'--customize-hook=chroot $1 mmd -i /boot/USB/filesystem.img@@{esp_offset} ::{live_media_path}',
f"""--customize-hook=echo '"Boot with default options" "boot=live live-media-path={live_media_path}"' >$1/boot/refind_linux.conf""",
# NOTE: find sidesteps the "glob expands before chroot applies" problem.
f"""--customize-hook=chroot $1 find -O3 /boot/ -xdev -mindepth 1 -maxdepth 1 -regextype posix-egrep -iregex '.*/(EFI|refind_linux.conf|vmlinuz.*|initrd.img.*)' -exec mcopy -vsbpm -i /boot/USB/filesystem.img@@{esp_offset} {{}} :: ';'""",
# FIXME: copy-out doesn't handle sparseness, so is REALLY slow (about 50 seconds).
# Therefore instead leave it in the squashfs, and extract it later.
# f'--customize-hook=copy-out /boot/USB/filesystem.img /tmp/',
# f'--customize-hook=chroot $1 rm /boot/USB/filesystem.img',
'bullseye',
td / 'filesystem.squashfs'
])
with args.output_file.open('wb') as f:
subprocess.check_call(['rdsquashfs', '--cat=boot/USB/filesystem.img', td / 'filesystem.squashfs'], stdout=f)
subprocess.check_call([
'mcopy', '-i', f'{args.output_file}@@{esp_offset}', td / 'filesystem.squashfs', f'::{live_media_path}/filesystem.squashfs'])

371
examples/twb/debian-sid-zfs.py
View file

@ -19,194 +19,197 @@ which in turn includes a bootloader (refind), kernel, ramdisk, and filesystem.sq
"""
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"output_file", nargs="?", default=pathlib.Path("filesystem.img"), type=pathlib.Path
)
parser.add_argument(
"--timezone",
default="Australia/Melbourne",
type=lambda s: s.split("/"),
help='NOTE: MUST be "Area/Zone" not e.g. "UTC", for now',
)
parser.add_argument(
"--locale", default="en_AU.UTF-8", help='NOTE: MUST end in ".UTF-8", for now'
)
parser.add_argument('output_file', nargs='?', default=pathlib.Path('filesystem.img'), type=pathlib.Path)
parser.add_argument('--timezone', default='Australia/Melbourne', type=lambda s: s.split('/'), help='NOTE: MUST be "Area/Zone" not e.g. "UTC", for now')
parser.add_argument('--locale', default='en_AU.UTF-8', help='NOTE: MUST end in ".UTF-8", for now')
args = parser.parse_args()
filesystem_img_size = "512M" # big enough to include filesystem.squashfs + about 64M of bootloader, kernel, and ramdisk.
esp_offset = 1024 * 1024 # 1MiB
esp_label = "UEFI-ESP" # max 8 bytes for FAT32
live_media_path = "debian-live"
filesystem_img_size = '512M' # big enough to include filesystem.squashfs + about 64M of bootloader, kernel, and ramdisk.
esp_offset = 1024 * 1024 # 1MiB
esp_label = 'UEFI-ESP' # max 8 bytes for FAT32
live_media_path = 'debian-live'
with tempfile.TemporaryDirectory(prefix="debian-sid-zfs.") as td:
with tempfile.TemporaryDirectory(prefix='debian-sid-zfs.') as td:
td = pathlib.Path(td)
subprocess.check_call(
[
"mmdebstrap",
"--mode=unshare",
"--variant=apt",
'--aptopt=Acquire::http::Proxy "http://apt-cacher-ng.cyber.com.au:3142"',
'--aptopt=Acquire::https::Proxy "DIRECT"',
"--dpkgopt=force-unsafe-io",
"--components=main contrib non-free", # needed for CPU security patches
"--include=init initramfs-tools xz-utils live-boot netbase",
"--include=dbus", # https://bugs.debian.org/814758
"--include=linux-image-amd64 firmware-linux",
# Have ZFS 2.0 support.
"--include=zfs-dkms zfsutils-linux zfs-zed build-essential linux-headers-amd64", # ZFS 2 support
# Make the initrd a little smaller (41MB -> 20MB), at the expensive of significantly slower image build time.
"--include=zstd",
"--essential-hook=mkdir -p $1/etc/initramfs-tools/conf.d",
"--essential-hook=>$1/etc/initramfs-tools/conf.d/zstd echo COMPRESS=zstd",
# Be the equivalent of Debian Live GNOME
# '--include=live-task-gnome',
#'--include=live-task-xfce',
# FIXME: enable this? It makes live-task-xfce go from 1G to 16G... so no.
#'--aptopt=Apt::Install-Recommends "true"',
# ...cherry-pick instead
# UPDATE: debian-installer-launcher DOES NOT WORK because we don't load crap SPECIFICALLY into /live/installer, in the ESP.
# UPDATE: network-manager-gnome DOES NOT WORK, nor is systemd-networkd auto-started... WTF?
# end result is no networking.
#'--include=live-config user-setup sudo firmware-linux haveged',
#'--include=calamares-settings-debian udisks2', # 300MB weirdo Qt GUI debian installer
#'--include=xfce4-terminal',
# x86_64 CPUs are undocumented proprietary RISC chips that EMULATE a documented x86_64 CISC ISA.
# The emulator is called "microcode", and is full of security vulnerabilities.
# Make sure security patches for microcode for *ALL* CPUs are included.
# By default, it tries to auto-detect the running CPU, so only patches the CPU of the build server.
"--include=intel-microcode amd64-microcode iucode-tool",
"--essential-hook=>$1/etc/default/intel-microcode echo IUCODE_TOOL_INITRAMFS=yes IUCODE_TOOL_SCANCPUS=no",
"--essential-hook=>$1/etc/default/amd64-microcode echo AMD64UCODE_INITRAMFS=yes",
"--dpkgopt=force-confold", # Work around https://bugs.debian.org/981004
# DHCP/DNS/SNTP clients...
# FIXME: use live-config ?
"--include=libnss-resolve libnss-myhostname systemd-timesyncd",
"--customize-hook=chroot $1 cp -alf /lib/systemd/resolv.conf /etc/resolv.conf", # This probably needs to happen LAST
# FIXME: fix resolv.conf to point to resolved, not "copy from the build-time OS"
# FIXME: fix hostname & hosts to not exist, not "copy from the build-time OS"
"--customize-hook=systemctl --root=$1 enable systemd-networkd systemd-timesyncd", # is this needed?
# Run a DHCP client on *ALL* ifaces.
# Consider network "up" (start sshd and local login prompt) when *ANY* (not ALL) ifaces are up.
"--customize-hook=>$1/etc/systemd/network/up.network printf '%s\n' '[Match]' Name='en*' '[Network]' DHCP=yes", # try DHCP on all ethernet ifaces
"--customize-hook=mkdir $1/etc/systemd/system/systemd-networkd-wait-online.service.d",
"--customize-hook=>$1/etc/systemd/system/systemd-networkd-wait-online.service.d/any-not-all.conf printf '%s\n' '[Service]' 'ExecStart=' 'ExecStart=/lib/systemd/systemd-networkd-wait-online --any'",
# Hope there's a central smarthost SMTP server called "mail" in the local search domain.
# FIXME: can live-config do this?
"--include=msmtp-mta",
"--customize-hook=>$1/etc/msmtprc printf '%s\n' 'account default' 'syslog LOG_MAIL' 'host mail' 'auto_from on'",
# Hope there's a central RELP logserver called "logserv" in the local domain.
# FIXME: can live-config do this?
"--include=rsyslog-relp",
"""--customize-hook=>$1/etc/rsyslog.conf printf '%s\n' 'module(load="imuxsock")' 'module(load="imklog")' 'module(load="omrelp")' 'action(type="omrelp" target="logserv" port="2514" template="RSYSLOG_SyslogProtocol23Format")'""",
# Run self-tests on all discoverable hard disks, and (try to) email if something goes wrong.
"--include=smartmontools bsd-mailx",
"--customize-hook=>$1/etc/smartd.conf echo 'DEVICESCAN -n standby,15 -a -o on -S on -s (S/../../7/00|L/../01/./01) -t -H -m root -M once'",
# For rarely-updated, rarely-rebooted SOEs, apply what security updates we can into transient tmpfs COW.
# This CANNOT apply kernel security updates (though it will download them).
# This CANNOT make the upgrades persistent across reboots (they re-download each boot).
# FIXME: Would it be cleaner to set Environment=NEEDRESTART_MODE=a in
# apt-daily-upgrade.service and/or
# unattended-upgrades.service, so
# needrestart is noninteractive only when apt is noninteractive?
"--include=unattended-upgrades needrestart",
"--customize-hook=echo 'unattended-upgrades unattended-upgrades/enable_auto_updates boolean true' | chroot $1 debconf-set-selections",
"""--customize-hook=>$1/etc/needrestart/conf.d/unattended-needrestart.conf echo '$nrconf{restart} = "a";'""", # https://bugs.debian.org/894444
# Do an apt update & apt upgrade at boot time (as well as @daily).
# The lack of /etc/machine-id causes these to be implicitly enabled.
# FIXME: use dropin in /etc.
"--customize-hook=>>$1/lib/systemd/system/apt-daily.service printf '%s\n' '[Install]' 'WantedBy=multi-user.target'",
"--customize-hook=>>$1/lib/systemd/system/apt-daily-upgrade.service printf '%s\n' '[Install]' 'WantedBy=multi-user.target'",
# FIXME: add support for this stuff (for the non-live final install this happens via ansible):
#
# unattended-upgrades
# smartd
# networkd (boot off ANY NIC, not EVERY NIC -- https://github.com/systemd/systemd/issues/9714)
# refind (bootloader config)
# misc safety nets
# double-check that mmdebstrap's machine-id support works properly
# Bare minimum to let me SSH in.
# FIXME: make this configurable.
# FIXME: trust a CA certificate instead -- see Zero Trust SSH, Jeremy Stott, LCA 2020 <https://youtu.be/lYzklWPTbsQ>
# WARNING: tinysshd does not support RSA, nor MaxStartups, nor sftp (unless you also install openssh-client, which is huge).
# FIXME: double-check no host keys are baked into the image (openssh-server and dropbear do this).
"--include=tinysshd rsync",
"--essential-hook=install -dm700 $1/root/.ssh",
'--essential-hook=echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIapAZ0E0353DaY6xBnasvu/DOvdWdKQ6RQURwq4l6Wu twb@cyber.com.au (Trent W. Buck)" >$1/root/.ssh/authorized_keys',
# Bare minimum to let me log in locally.
# DO NOT use this on production builds!
"--essential-hook=chroot $1 passwd --delete root",
# Configure language (not needed to boot).
# Racism saves a **LOT** of space -- something like 2GB for Debian Live images.
# FIXME: use live-config instead?
"--include=locales localepurge",
f"--essential-hook=echo locales locales/default_environment_locale select {args.locale} | chroot $1 debconf-set-selections",
f"--essential-hook=echo locales locales/locales_to_be_generated multiselect {args.locale} UTF-8 | chroot $1 debconf-set-selections",
# FIXME: https://bugs.debian.org/603700
"--customize-hook=chroot $1 sed -i /etc/locale.nopurge -e 's/^USE_DPKG/#ARGH#&/'",
"--customize-hook=chroot $1 localepurge",
"--customize-hook=chroot $1 sed -i /etc/locale.nopurge -e 's/^#ARGH#//'",
# Removing documentation also saves a LOT of space.
"--dpkgopt=path-exclude=/usr/share/doc/*",
"--dpkgopt=path-exclude=/usr/share/info/*",
"--dpkgopt=path-exclude=/usr/share/man/*",
"--dpkgopt=path-exclude=/usr/share/omf/*",
"--dpkgopt=path-exclude=/usr/share/help/*",
"--dpkgopt=path-exclude=/usr/share/gnome/help/*",
# Configure timezone (not needed to boot)`
# FIXME: use live-config instead?
"--include=tzdata",
f"--essential-hook=echo tzdata tzdata/Areas select {args.timezone[0]} | chroot $1 debconf-set-selections",
f"--essential-hook=echo tzdata tzdata/Zones/{args.timezone[0]} select {args.timezone[1]} | chroot $1 debconf-set-selections",
# Do the **BARE MINIMUM** to make a USB key that can boot on X86_64 UEFI.
# We use mtools so we do not ever need root privileges.
# We can't use mkfs.vfat, as that needs kpartx or losetup (i.e. root).
# We can't use mkfs.udf, as that needs mount (i.e. root).
# We can't use "refind-install --usedefault" as that runs mount(8) (i.e. root).
# We don't use genisoimage because
# 1) ISO9660 must die;
# 2) incomplete UDF 1.5+ support;
# 3) resulting filesystem can't be tweaked after flashing (e.g. debian-live/site.dir/etc/systemd/network/up.network).
#
# We use refind because 1) I hate grub; and 2) I like refind.
# If you want aarch64 or ia32 you need to install their BOOTxxx.EFI files.
# If you want kernel+initrd on something other than FAT, you need refind/drivers_xxx/xxx_xxx.EFI.
#
# FIXME: with qemu in UEFI mode (OVMF), I get dumped into startup.nsh (UEFI REPL).
# From there, I can manually type in "FS0:\EFI\BOOT\BOOTX64.EFI" to start refind, tho.
# So WTF is its problem? Does it not support fallback bootloader?
"--include=refind parted mtools",
"--essential-hook=echo refind refind/install_to_esp boolean false | chroot $1 debconf-set-selections",
"--customize-hook=echo refind refind/install_to_esp boolean true | chroot $1 debconf-set-selections",
"--customize-hook=chroot $1 mkdir -p /boot/USB /boot/EFI/BOOT",
"--customize-hook=chroot $1 cp /usr/share/refind/refind/refind_x64.efi /boot/EFI/BOOT/BOOTX64.EFI",
"--customize-hook=chroot $1 cp /usr/share/refind/refind/refind.conf-sample /boot/EFI/BOOT/refind.conf",
f"--customize-hook=chroot $1 truncate --size={filesystem_img_size} /boot/USB/filesystem.img",
f"--customize-hook=chroot $1 parted --script --align=optimal /boot/USB/filesystem.img mklabel gpt mkpart {esp_label} {esp_offset}b 100% set 1 esp on",
f"--customize-hook=chroot $1 mformat -i /boot/USB/filesystem.img@@{esp_offset} -F -v {esp_label}",
f"--customize-hook=chroot $1 mmd -i /boot/USB/filesystem.img@@{esp_offset} ::{live_media_path}",
f"""--customize-hook=echo '"Boot with default options" "boot=live live-media-path={live_media_path}"' >$1/boot/refind_linux.conf""",
f"""--customize-hook=chroot $1 find /boot/ -xdev -mindepth 1 -maxdepth 1 -not -name filesystem.img -not -name USB -exec mcopy -vsbpm -i /boot/USB/filesystem.img@@{esp_offset} {{}} :: ';'""",
# FIXME: copy-out doesn't handle sparseness, so is REALLY slow (about 50 seconds).
# Therefore instead leave it in the squashfs, and extract it later.
# f'--customize-hook=copy-out /boot/USB/filesystem.img /tmp/',
# f'--customize-hook=chroot $1 rm /boot/USB/filesystem.img',
"sid",
td / "filesystem.squashfs",
]
)
['mmdebstrap',
'--mode=unshare',
'--variant=apt',
'--aptopt=Acquire::http::Proxy "http://apt-cacher-ng.cyber.com.au:3142"',
'--aptopt=Acquire::https::Proxy "DIRECT"',
'--dpkgopt=force-unsafe-io',
'--components=main contrib non-free', # needed for CPU security patches
'--include=init initramfs-tools xz-utils live-boot netbase',
'--include=dbus', # https://bugs.debian.org/814758
'--include=linux-image-amd64 firmware-linux',
# Have ZFS 2.0 support.
'--include=zfs-dkms zfsutils-linux zfs-zed build-essential linux-headers-amd64', # ZFS 2 support
# Make the initrd a little smaller (41MB -> 20MB), at the expensive of significantly slower image build time.
'--include=zstd',
'--essential-hook=mkdir -p $1/etc/initramfs-tools/conf.d',
'--essential-hook=>$1/etc/initramfs-tools/conf.d/zstd echo COMPRESS=zstd',
# Be the equivalent of Debian Live GNOME
# '--include=live-task-gnome',
#'--include=live-task-xfce',
# FIXME: enable this? It makes live-task-xfce go from 1G to 16G... so no.
#'--aptopt=Apt::Install-Recommends "true"',
# ...cherry-pick instead
# UPDATE: debian-installer-launcher DOES NOT WORK because we don't load crap SPECIFICALLY into /live/installer, in the ESP.
# UPDATE: network-manager-gnome DOES NOT WORK, nor is systemd-networkd auto-started... WTF?
# end result is no networking.
#'--include=live-config user-setup sudo firmware-linux haveged',
#'--include=calamares-settings-debian udisks2', # 300MB weirdo Qt GUI debian installer
#'--include=xfce4-terminal',
# x86_64 CPUs are undocumented proprietary RISC chips that EMULATE a documented x86_64 CISC ISA.
# The emulator is called "microcode", and is full of security vulnerabilities.
# Make sure security patches for microcode for *ALL* CPUs are included.
# By default, it tries to auto-detect the running CPU, so only patches the CPU of the build server.
'--include=intel-microcode amd64-microcode iucode-tool',
'--essential-hook=>$1/etc/default/intel-microcode echo IUCODE_TOOL_INITRAMFS=yes IUCODE_TOOL_SCANCPUS=no',
'--essential-hook=>$1/etc/default/amd64-microcode echo AMD64UCODE_INITRAMFS=yes',
'--dpkgopt=force-confold', # Work around https://bugs.debian.org/981004
# DHCP/DNS/SNTP clients...
# FIXME: use live-config ?
'--include=libnss-resolve libnss-myhostname systemd-timesyncd',
'--customize-hook=chroot $1 cp -alf /lib/systemd/resolv.conf /etc/resolv.conf', # This probably needs to happen LAST
# FIXME: fix resolv.conf to point to resolved, not "copy from the build-time OS"
# FIXME: fix hostname & hosts to not exist, not "copy from the build-time OS"
'--customize-hook=systemctl --root=$1 enable systemd-networkd systemd-timesyncd', # is this needed?
# Run a DHCP client on *ALL* ifaces.
# Consider network "up" (start sshd and local login prompt) when *ANY* (not ALL) ifaces are up.
"--customize-hook=>$1/etc/systemd/network/up.network printf '%s\n' '[Match]' Name='en*' '[Network]' DHCP=yes", # try DHCP on all ethernet ifaces
'--customize-hook=mkdir $1/etc/systemd/system/systemd-networkd-wait-online.service.d',
"--customize-hook=>$1/etc/systemd/system/systemd-networkd-wait-online.service.d/any-not-all.conf printf '%s\n' '[Service]' 'ExecStart=' 'ExecStart=/lib/systemd/systemd-networkd-wait-online --any'",
# Hope there's a central smarthost SMTP server called "mail" in the local search domain.
# FIXME: can live-config do this?
'--include=msmtp-mta',
"--customize-hook=>$1/etc/msmtprc printf '%s\n' 'account default' 'syslog LOG_MAIL' 'host mail' 'auto_from on'",
# Hope there's a central RELP logserver called "logserv" in the local domain.
# FIXME: can live-config do this?
'--include=rsyslog-relp',
"""--customize-hook=>$1/etc/rsyslog.conf printf '%s\n' 'module(load="imuxsock")' 'module(load="imklog")' 'module(load="omrelp")' 'action(type="omrelp" target="logserv" port="2514" template="RSYSLOG_SyslogProtocol23Format")'""",
# Run self-tests on all discoverable hard disks, and (try to) email if something goes wrong.
'--include=smartmontools bsd-mailx',
"--customize-hook=>$1/etc/smartd.conf echo 'DEVICESCAN -n standby,15 -a -o on -S on -s (S/../../7/00|L/../01/./01) -t -H -m root -M once'",
# For rarely-updated, rarely-rebooted SOEs, apply what security updates we can into transient tmpfs COW.
# This CANNOT apply kernel security updates (though it will download them).
# This CANNOT make the upgrades persistent across reboots (they re-download each boot).
# FIXME: Would it be cleaner to set Environment=NEEDRESTART_MODE=a in
# apt-daily-upgrade.service and/or
# unattended-upgrades.service, so
# needrestart is noninteractive only when apt is noninteractive?
'--include=unattended-upgrades needrestart',
"--customize-hook=echo 'unattended-upgrades unattended-upgrades/enable_auto_updates boolean true' | chroot $1 debconf-set-selections",
"""--customize-hook=>$1/etc/needrestart/conf.d/unattended-needrestart.conf echo '$nrconf{restart} = "a";'""", # https://bugs.debian.org/894444
# Do an apt update & apt upgrade at boot time (as well as @daily).
# The lack of /etc/machine-id causes these to be implicitly enabled.
# FIXME: use dropin in /etc.
"--customize-hook=>>$1/lib/systemd/system/apt-daily.service printf '%s\n' '[Install]' 'WantedBy=multi-user.target'",
"--customize-hook=>>$1/lib/systemd/system/apt-daily-upgrade.service printf '%s\n' '[Install]' 'WantedBy=multi-user.target'",
# FIXME: add support for this stuff (for the non-live final install this happens via ansible):
#
# unattended-upgrades
# smartd
# networkd (boot off ANY NIC, not EVERY NIC -- https://github.com/systemd/systemd/issues/9714)
# refind (bootloader config)
# misc safety nets
# double-check that mmdebstrap's machine-id support works properly
# Bare minimum to let me SSH in.
# FIXME: make this configurable.
# FIXME: trust a CA certificate instead -- see Zero Trust SSH, Jeremy Stott, LCA 2020 <https://youtu.be/lYzklWPTbsQ>
# WARNING: tinysshd does not support RSA, nor MaxStartups, nor sftp (unless you also install openssh-client, which is huge).
# FIXME: double-check no host keys are baked into the image (openssh-server and dropbear do this).
'--include=tinysshd rsync',
'--essential-hook=install -dm700 $1/root/.ssh',
'--essential-hook=echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIapAZ0E0353DaY6xBnasvu/DOvdWdKQ6RQURwq4l6Wu twb@cyber.com.au (Trent W. Buck)" >$1/root/.ssh/authorized_keys',
# Bare minimum to let me log in locally.
# DO NOT use this on production builds!
'--essential-hook=chroot $1 passwd --delete root',
# Configure language (not needed to boot).
# Racism saves a **LOT** of space -- something like 2GB for Debian Live images.
# FIXME: use live-config instead?
'--include=locales localepurge',
f'--essential-hook=echo locales locales/default_environment_locale select {args.locale} | chroot $1 debconf-set-selections',
f'--essential-hook=echo locales locales/locales_to_be_generated multiselect {args.locale} UTF-8 | chroot $1 debconf-set-selections',
# FIXME: https://bugs.debian.org/603700
"--customize-hook=chroot $1 sed -i /etc/locale.nopurge -e 's/^USE_DPKG/#ARGH#&/'",
"--customize-hook=chroot $1 localepurge",
"--customize-hook=chroot $1 sed -i /etc/locale.nopurge -e 's/^#ARGH#//'",
# Removing documentation also saves a LOT of space.
'--dpkgopt=path-exclude=/usr/share/doc/*',
'--dpkgopt=path-exclude=/usr/share/info/*',
'--dpkgopt=path-exclude=/usr/share/man/*',
'--dpkgopt=path-exclude=/usr/share/omf/*',
'--dpkgopt=path-exclude=/usr/share/help/*',
'--dpkgopt=path-exclude=/usr/share/gnome/help/*',
# Configure timezone (not needed to boot)`
# FIXME: use live-config instead?
'--include=tzdata',
f'--essential-hook=echo tzdata tzdata/Areas select {args.timezone[0]} | chroot $1 debconf-set-selections',
f'--essential-hook=echo tzdata tzdata/Zones/{args.timezone[0]} select {args.timezone[1]} | chroot $1 debconf-set-selections',
# Do the **BARE MINIMUM** to make a USB key that can boot on X86_64 UEFI.
# We use mtools so we do not ever need root privileges.
# We can't use mkfs.vfat, as that needs kpartx or losetup (i.e. root).
# We can't use mkfs.udf, as that needs mount (i.e. root).
# We can't use "refind-install --usedefault" as that runs mount(8) (i.e. root).
# We don't use genisoimage because
# 1) ISO9660 must die;
# 2) incomplete UDF 1.5+ support;
# 3) resulting filesystem can't be tweaked after flashing (e.g. debian-live/site.dir/etc/systemd/network/up.network).
#
# We use refind because 1) I hate grub; and 2) I like refind.
# If you want aarch64 or ia32 you need to install their BOOTxxx.EFI files.
# If you want kernel+initrd on something other than FAT, you need refind/drivers_xxx/xxx_xxx.EFI.
#
# FIXME: with qemu in UEFI mode (OVMF), I get dumped into startup.nsh (UEFI REPL).
# From there, I can manually type in "FS0:\EFI\BOOT\BOOTX64.EFI" to start refind, tho.
# So WTF is its problem? Does it not support fallback bootloader?
'--include=refind parted mtools',
'--essential-hook=echo refind refind/install_to_esp boolean false | chroot $1 debconf-set-selections',
'--customize-hook=echo refind refind/install_to_esp boolean true | chroot $1 debconf-set-selections',
'--customize-hook=chroot $1 mkdir -p /boot/USB /boot/EFI/BOOT',
'--customize-hook=chroot $1 cp /usr/share/refind/refind/refind_x64.efi /boot/EFI/BOOT/BOOTX64.EFI',
'--customize-hook=chroot $1 cp /usr/share/refind/refind/refind.conf-sample /boot/EFI/BOOT/refind.conf',
f'--customize-hook=chroot $1 truncate --size={filesystem_img_size} /boot/USB/filesystem.img',
f'--customize-hook=chroot $1 parted --script --align=optimal /boot/USB/filesystem.img mklabel gpt mkpart {esp_label} {esp_offset}b 100% set 1 esp on',
f'--customize-hook=chroot $1 mformat -i /boot/USB/filesystem.img@@{esp_offset} -F -v {esp_label}',
f'--customize-hook=chroot $1 mmd -i /boot/USB/filesystem.img@@{esp_offset} ::{live_media_path}',
f"""--customize-hook=echo '"Boot with default options" "boot=live live-media-path={live_media_path}"' >$1/boot/refind_linux.conf""",
f"""--customize-hook=chroot $1 find /boot/ -xdev -mindepth 1 -maxdepth 1 -not -name filesystem.img -not -name USB -exec mcopy -vsbpm -i /boot/USB/filesystem.img@@{esp_offset} {{}} :: ';'""",
# FIXME: copy-out doesn't handle sparseness, so is REALLY slow (about 50 seconds).
# Therefore instead leave it in the squashfs, and extract it later.
# f'--customize-hook=copy-out /boot/USB/filesystem.img /tmp/',
# f'--customize-hook=chroot $1 rm /boot/USB/filesystem.img',
'sid',
td / 'filesystem.squashfs'
])
with args.output_file.open('wb') as f:
subprocess.check_call(['rdsquashfs', '--cat=boot/USB/filesystem.img', td / 'filesystem.squashfs'], stdout=f)
subprocess.check_call([
'mcopy', '-i', f'{args.output_file}@@{esp_offset}', td / 'filesystem.squashfs', f'::{live_media_path}/filesystem.squashfs'])
with args.output_file.open("wb") as f:
subprocess.check_call(
["rdsquashfs", "--cat=boot/USB/filesystem.img", td / "filesystem.squashfs"],
stdout=f,
)
subprocess.check_call(
[
"mcopy",
"-i",
f"{args.output_file}@@{esp_offset}",
td / "filesystem.squashfs",
f"::{live_media_path}/filesystem.squashfs",
]
)

51
gpgvnoexpkeysig
View file

@ -1,51 +0,0 @@
#!/bin/sh
#
# No copyright is claimed. This code is in the public domain; do with
# it what you wish.
#
# Author: Johannes Schauer Marin Rodrigues <josch@mister-muffin.de>
#
# This is a wrapper around gpgv as invoked by apt. It turns EXPKEYSIG results
# from gpgv into GOODSIG results. This is necessary for apt to access very old
# timestamps from snapshot.debian.org for which the GPG key is already expired:
#
# Get:1 http://snapshot.debian.org/archive/debian/20150106T000000Z unstable InRelease [242 kB]
# Err:1 http://snapshot.debian.org/archive/debian/20150106T000000Z unstable InRelease
# The following signatures were invalid: EXPKEYSIG 8B48AD6246925553 Debian Archive Automatic Signing Key (7.0/wheezy) <ftpmaster@debian.org>
# Reading package lists...
# W: GPG error: http://snapshot.debian.org/archive/debian/20150106T000000Z unstable InRelease: The following signatures were invalid: EXPKEYSIG 8B48AD6246925553 Debian Archive Automatic Signing Key (7.0/wheezy) <ftpmaster@debian.org>
# E: The repository 'http://snapshot.debian.org/archive/debian/20150106T000000Z unstable InRelease' is not signed.
#
# To use this script, call apt with
#
# -o Apt::Key::gpgvcommand=/usr/libexec/mmdebstrap/gpgvnoexpkeysig
#
# Scripts doing similar things can be found here:
#
# * debuerreotype as /usr/share/debuerreotype/scripts/.gpgv-ignore-expiration.sh
# * derivative census: salsa.d.o/deriv-team/census/-/blob/master/bin/fakegpgv
set -eu
find_gpgv_status_fd() {
while [ "$#" -gt 0 ]; do
if [ "$1" = '--status-fd' ]; then
echo "$2"
return 0
fi
shift
done
# default fd is stdout
echo 1
}
GPGSTATUSFD="$(find_gpgv_status_fd "$@")"
case $GPGSTATUSFD in
'' | *[!0-9]*)
echo "invalid --status-fd argument" >&2
exit 1
;;
esac
# we need eval because we cannot redirect a variable fd
eval 'exec gpgv "$@" '"$GPGSTATUSFD"'>&1 | sed "s/^\[GNUPG:\] EXPKEYSIG /[GNUPG:] GOODSIG /" >&'"$GPGSTATUSFD"

11
hooks/busybox/extract00.sh
View file

@ -1,14 +1,7 @@
#!/bin/sh
set -eu
if [ "${MMDEBSTRAP_VERBOSITY:-1}" -ge 3 ]; then
set -x
fi
set -exu
rootdir="$1"
# Run busybox using an absolute path so that this script also works in case
# /proc is not mounted. Busybox uses /proc/self/exe to figure out the path
# to its executable.
chroot "$rootdir" /bin/busybox --install -s
chroot "$rootdir" busybox --install -s

6
hooks/busybox/setup00.sh
View file

@ -1,10 +1,6 @@
#!/bin/sh
set -eu
if [ "${MMDEBSTRAP_VERBOSITY:-1}" -ge 3 ]; then
set -x
fi
set -exu
rootdir="$1"

44
hooks/copy-host-apt-sources-and-preferences/customize00.pl
View file

@ -1,44 +0,0 @@
#!/usr/bin/perl
#
# This script makes sure that all packages that are installed both locally as
# well as inside the chroot have the same version.
#
# It is implemented in Perl because there are no associative arrays in POSIX
# shell.
use strict;
use warnings;
sub get_pkgs {
my $root = shift;
my %pkgs = ();
open(my $fh, '-|', 'dpkg-query', "--root=$root", '--showformat',
'${binary:Package}=${Version}\n', '--show')
// die "cannot exec dpkg-query";
while (my $line = <$fh>) {
my ($pkg, $ver) = split(/=/, $line, 2);
$pkgs{$pkg} = $ver;
}
close $fh;
if ($? != 0) { die "failed to run dpkg-query" }
return %pkgs;
}
my %pkgs_local = get_pkgs('/');
my %pkgs_chroot = get_pkgs($ARGV[0]);
my @diff = ();
foreach my $pkg (keys %pkgs_chroot) {
next unless exists $pkgs_local{$pkg};
if ($pkgs_local{$pkg} ne $pkgs_chroot{$pkg}) {
push @diff, $pkg;
}
}
if (scalar @diff > 0) {
print STDERR "E: packages from the host and the chroot differ:\n";
foreach my $pkg (@diff) {
print STDERR "E: $pkg $pkgs_local{$pkg} $pkgs_chroot{$pkg}\n";
}
exit 1;
}

55
hooks/copy-host-apt-sources-and-preferences/setup00.sh
View file

@ -1,55 +0,0 @@
#!/bin/sh
#
# This script makes sure that the apt sources.list and preferences from outside
# the chroot also exist inside the chroot by *appending* them to any existing
# files. If you do not want to keep the original content, add another setup
# hook before this one which cleans up the files you don't want to keep.
#
# If instead of copying sources.list verbatim you want to mangle its contents,
# consider using python-apt for that. An example can be found in the Debian
# packaging of mmdebstrap in ./debian/tests/sourcesfilter
set -eu
if [ "${MMDEBSTRAP_VERBOSITY:-1}" -ge 3 ]; then
set -x
fi
if [ -n "${MMDEBSTRAP_SUITE:-}" ]; then
if [ "${MMDEBSTRAP_VERBOSITY:-1}" -ge 1 ]; then
echo "W: using a non-empty suite name $MMDEBSTRAP_SUITE does not make sense with this hook and might select the wrong Essential:yes package set" >&2
fi
fi
rootdir="$1"
SOURCELIST="/etc/apt/sources.list"
eval "$(apt-config shell SOURCELIST Dir::Etc::SourceList/f)"
SOURCEPARTS="/etc/apt/sources.d/"
eval "$(apt-config shell SOURCEPARTS Dir::Etc::SourceParts/d)"
PREFERENCES="/etc/apt/preferences"
eval "$(apt-config shell PREFERENCES Dir::Etc::Preferences/f)"
PREFERENCESPARTS="/etc/apt/preferences.d/"
eval "$(apt-config shell PREFERENCESPARTS Dir::Etc::PreferencesParts/d)"
for f in "$SOURCELIST" \
"$SOURCEPARTS"/*.list \
"$SOURCEPARTS"/*.sources \
"$PREFERENCES" \
"$PREFERENCESPARTS"/*; do
[ -e "$f" ] || continue
mkdir --parents "$(dirname "$rootdir/$f")"
if [ -e "$rootdir/$f" ]; then
if [ "${MMDEBSTRAP_VERBOSITY:-1}" -ge 2 ]; then
echo "I: $f already exists in chroot, appending..." >&2
fi
# Add extra newline between old content and new content.
# This is required in case of deb822 files.
echo >> "$rootdir/$f"
fi
cat "$f" >> "$rootdir/$f"
if [ "${MMDEBSTRAP_VERBOSITY:-1}" -ge 3 ]; then
echo "D: contents of $f inside the chroot:" >&2
cat "$rootdir/$f" >&2
fi
done

12
hooks/eatmydata/customize.sh
View file

@ -1,10 +1,6 @@
#!/bin/sh
set -eu
if [ "${MMDEBSTRAP_VERBOSITY:-1}" -ge 3 ]; then
set -x
fi
set -exu
rootdir="$1"
@ -13,14 +9,14 @@ if [ -e "$rootdir/var/lib/dpkg/arch" ]; then
else
chrootarch=$(dpkg --print-architecture)
fi
libdir="/usr/lib/$(dpkg-architecture -a "$chrootarch" -q DEB_HOST_MULTIARCH)"
libdir="/usr/lib/$(dpkg-architecture -a $chrootarch -q DEB_HOST_MULTIARCH)"
# if eatmydata was actually installed properly, then we are not removing
# anything here
if ! chroot "$rootdir" dpkg-query --show eatmydata; then
if ! chroot "$rootdir" dpkg-query --list eatmydata; then
rm "$rootdir/usr/bin/eatmydata"
fi
if ! chroot "$rootdir" dpkg-query --show libeatmydata1; then
if ! chroot "$rootdir" dpkg-query --list libeatmydata1; then
rm "$rootdir$libdir"/libeatmydata.so*
fi

18
hooks/eatmydata/extract.sh
View file

@ -1,10 +1,6 @@
#!/bin/sh
set -eu
if [ "${MMDEBSTRAP_VERBOSITY:-1}" -ge 3 ]; then
set -x
fi
set -exu
rootdir="$1"
@ -14,10 +10,8 @@ else
chrootarch=$(dpkg --print-architecture)
fi
trusted=
eval "$(apt-config shell trusted Dir::Etc::trusted/f)"
trustedparts=
eval "$(apt-config shell trustedparts Dir::Etc::trustedparts/d)"
eval $(apt-config shell trusted Dir::Etc::trusted/f)
eval $(apt-config shell trustedparts Dir::Etc::trustedparts/d)
tmpfile=$(mktemp --tmpdir="$rootdir/tmp")
cat << END > "$tmpfile"
Apt::Architecture "$chrootarch";
@ -31,8 +25,8 @@ END
# nothing will be printed for them
tmpdir=$(mktemp --directory --tmpdir="$rootdir/tmp")
env --chdir="$tmpdir" APT_CONFIG="$tmpfile" apt-get download --print-uris eatmydata libeatmydata1 \
| sed -ne "s/^'\([^']\+\)'\s\+\(\S\+\)\s\+\([0-9]\+\)\s\+\(SHA256:[a-f0-9]\+\)$/\1 \2 \3 \4/p" \
| while read -r uri fname size hash; do
| sed -ne "s/^'\([^']\+\)'\s\+\([^\s]\+\)\s\+\([0-9]\+\)\s\+\(SHA256:[a-f0-9]\+\)$/\1 \2 \3 \4/p" \
| while read uri fname size hash; do
echo "processing $fname" >&2
if [ -e "$tmpdir/$fname" ]; then
echo "$tmpdir/$fname already exists" >&2
@ -47,7 +41,7 @@ env --chdir="$tmpdir" APT_CONFIG="$tmpfile" apt-get download --print-uris eatmyd
| tar --directory="$rootdir/usr/bin" --strip-components=3 --extract --verbose ./usr/bin/eatmydata
;;
libeatmydata1_*_$chrootarch.deb)
libdir="/usr/lib/$(dpkg-architecture -a "$chrootarch" -q DEB_HOST_MULTIARCH)"
libdir="/usr/lib/$(dpkg-architecture -a $chrootarch -q DEB_HOST_MULTIARCH)"
mkdir -p "$rootdir$libdir"
dpkg-deb --fsys-tarfile "$tmpdir/$fname" \
| tar --directory="$rootdir$libdir" --strip-components=4 --extract --verbose --wildcards ".$libdir/libeatmydata.so*"

41
hooks/file-mirror-automount/customize00.sh
View file

@ -1,41 +0,0 @@
#!/bin/sh
#
# shellcheck disable=SC2086
set -eu
if [ "${MMDEBSTRAP_VERBOSITY:-1}" -ge 3 ]; then
set -x
fi
rootdir="$1"
if [ ! -e "$rootdir/run/mmdebstrap/file-mirror-automount" ]; then
exit 0
fi
xargsopts="--null --no-run-if-empty -I {} --max-args=1"
case $MMDEBSTRAP_MODE in
root|unshare)
echo "unmounting the following mountpoints:" >&2 ;;
*)
echo "removing the following directories:" >&2 ;;
esac
< "$rootdir/run/mmdebstrap/file-mirror-automount" \
xargs $xargsopts echo " $rootdir/{}"
case $MMDEBSTRAP_MODE in
root|unshare)
< "$rootdir/run/mmdebstrap/file-mirror-automount" \
xargs $xargsopts umount "$rootdir/{}"
;;
*)
< "$rootdir/run/mmdebstrap/file-mirror-automount" \
xargs $xargsopts rm -r "$rootdir/{}"
;;
esac
rm "$rootdir/run/mmdebstrap/file-mirror-automount"
rmdir --ignore-fail-on-non-empty "$rootdir/run/mmdebstrap"

73
hooks/file-mirror-automount/setup00.sh
View file

@ -1,73 +0,0 @@
#!/bin/sh
set -eu
if [ "${MMDEBSTRAP_VERBOSITY:-1}" -ge 3 ]; then
set -x
fi
rootdir="$1"
# process all configured apt repositories
env APT_CONFIG="$MMDEBSTRAP_APT_CONFIG" apt-get indextargets --no-release-info --format '$(REPO_URI)' \
| sed -ne 's/^file:\/\+//p' \
| sort -u \
| while read -r path; do
mkdir -p "$rootdir/run/mmdebstrap"
if [ ! -d "/$path" ]; then
echo "W: /$path is not an existing directory" >&2
continue
fi
case $MMDEBSTRAP_MODE in
root|unshare)
echo "bind-mounting /$path into the chroot" >&2
mkdir -p "$rootdir/$path"
mount -o ro,bind "/$path" "$rootdir/$path"
;;
*)
echo "copying /$path into the chroot" >&2
mkdir -p "$rootdir/$path"
"$MMDEBSTRAP_ARGV0" --hook-helper "$rootdir" "$MMDEBSTRAP_MODE" "$MMDEBSTRAP_HOOK" env "$MMDEBSTRAP_VERBOSITY" sync-in "/$path" "/$path" <&"$MMDEBSTRAP_HOOKSOCK" >&"$MMDEBSTRAP_HOOKSOCK"
;;
esac
printf '/%s\0' "$path" >> "$rootdir/run/mmdebstrap/file-mirror-automount"
done
# process all files given via --include
set -f # turn off pathname expansion
IFS=',' # split by comma
for pkg in $MMDEBSTRAP_INCLUDE; do
set +f; unset IFS
case $pkg in
./*|../*|/*) : ;; # we are interested in this case
*) continue ;; # not a file
esac
# undo escaping
pkg="$(printf '%s' "$pkg" | sed 's/%2C/,/g; s/%25/%/g')"
# check for existance
if [ ! -f "$pkg" ]; then
echo "$pkg does not exist" >&2
continue
fi
# make path absolute
pkg="$(realpath "$pkg")"
case "$pkg" in
/*) : ;;
*) echo "path for $pkg is not absolute" >&2; continue;;
esac
mkdir -p "$rootdir/run/mmdebstrap"
mkdir -p "$rootdir/$(dirname "$pkg")"
case $MMDEBSTRAP_MODE in
root|unshare)
echo "bind-mounting $pkg into the chroot" >&2
touch "$rootdir/$pkg"
mount -o bind "$pkg" "$rootdir/$pkg"
;;
*)
echo "copying $pkg into the chroot" >&2
"$MMDEBSTRAP_ARGV0" --hook-helper "$rootdir" "$MMDEBSTRAP_MODE" "$MMDEBSTRAP_HOOK" env "$MMDEBSTRAP_VERBOSITY" upload "$pkg" "$pkg" <&"$MMDEBSTRAP_HOOKSOCK" >&"$MMDEBSTRAP_HOOKSOCK"
;;
esac
printf '/%s\0' "$pkg" >> "$rootdir/run/mmdebstrap/file-mirror-automount"
done
set +f; unset IFS

16
hooks/jessie-or-older/extract00.sh
View file

@ -1,16 +0,0 @@
#!/bin/sh
set -eu
if [ "${MMDEBSTRAP_VERBOSITY:-1}" -ge 3 ]; then
set -x
fi
TARGET="$1"
# not needed since dpkg 1.17.11
for f in available diversions cmethopt; do
if [ ! -e "$TARGET/var/lib/dpkg/$f" ]; then
touch "$TARGET/var/lib/dpkg/$f"
fi
done

47
hooks/jessie-or-older/extract01.sh
View file

@ -1,47 +0,0 @@
#!/bin/sh
#
# needed until init 1.33 which pre-depends on systemd-sysv
# starting with init 1.34, init is not Essential:yes anymore
#
# jessie has init 1.22
set -eu
if [ "${MMDEBSTRAP_VERBOSITY:-1}" -ge 3 ]; then
set -x
fi
TARGET="$1"
if [ -z "${MMDEBSTRAP_ESSENTIAL+x}" ]; then
MMDEBSTRAP_ESSENTIAL=
for f in "$TARGET/var/cache/apt/archives/"*.deb; do
[ -f "$f" ] || continue
f="${f#"$TARGET"}"
MMDEBSTRAP_ESSENTIAL="$MMDEBSTRAP_ESSENTIAL $f"
done
fi
fname_base_passwd=
fname_base_files=
fname_dpkg=
for pkg in $MMDEBSTRAP_ESSENTIAL; do
pkgname=$(dpkg-deb --show --showformat='${Package}' "$TARGET/$pkg")
# shellcheck disable=SC2034
case $pkgname in
base-passwd) fname_base_passwd=$pkg;;
base-files) fname_base_files=$pkg;;
dpkg) fname_dpkg=$pkg;;
esac
done
for var in base_passwd base_files dpkg; do
eval 'val=$fname_'"$var"
[ -z "$val" ] && continue
chroot "$TARGET" dpkg --install --force-depends "$val"
done
# shellcheck disable=SC2086
chroot "$TARGET" dpkg --unpack --force-depends $MMDEBSTRAP_ESSENTIAL
chroot "$TARGET" dpkg --configure --pending

37
hooks/maybe-jessie-or-older/extract00.sh
View file

@ -1,37 +0,0 @@
#!/bin/sh
set -eu
# we need to check the version of dpkg
# since at this point packages are just extracted but not installed, we cannot use dpkg-query
# since we want to support chrootless, we cannot run dpkg --version inside the chroot
# to avoid this hook depending on dpkg-dev being installed, we do not parse the extracted changelog with dpkg-parsechangelog
# we also want to avoid parsing the changelog because /usr/share/doc might've been added to dpkg --path-exclude
# instead, we just ask apt about the latest version of dpkg it knows of
# this should only fail in situations where there are multiple versions of dpkg in different suites
ver=$(env --chdir="$1" APT_CONFIG="$MMDEBSTRAP_APT_CONFIG" apt-cache show --no-all-versions dpkg 2>/dev/null | sed -ne 's/^Version: \(.*\)$/\1/p' || printf '')
if [ -z "$ver" ]; then
echo "no package called dpkg can be installed -- not running jessie-or-older extract00 hook" >&2
exit 0
fi
if dpkg --compare-versions "$ver" ge 1.17.11; then
echo "dpkg version $ver is >= 1.17.11 -- not running jessie-or-older extract00 hook" >&2
exit 0
else
echo "dpkg version $ver is << 1.17.11 -- running jessie-or-older extract00 hook" >&2
fi
# resolve the script path using several methods in order:
# 1. using dirname -- "$0"
# 2. using ./hooks
# 3. using /usr/share/mmdebstrap/hooks/
for p in "$(dirname -- "$0")/.." ./hooks /usr/share/mmdebstrap/hooks; do
if [ -x "$p/jessie-or-older/extract00.sh" ] && [ -x "$p/jessie-or-older/extract01.sh" ]; then
"$p/jessie-or-older/extract00.sh" "$1"
exit 0
fi
done
echo "cannot find jessie-or-older hook anywhere" >&2
exit 1

57
hooks/maybe-jessie-or-older/extract01.sh
View file

@ -1,57 +0,0 @@
#!/bin/sh
set -eu
# The jessie-or-older extract01 hook has to be run up to the point where the
# Essential:yes field was removed from the init package (with
# init-system-helpers 1.34). Since the essential packages have only been
# extracted but not installed, we cannot use dpkg-query to find out its
# version. Since /usr/share/doc might be missing due to dpkg --path-exclude, we
# also cannot check whether /usr/share/doc/init/copyright exists. There also
# was a time (before init-system-helpers 1.20) where there was no init package
# at all where we also want to apply this hook. So we just ask apt about the
# candidate version for init-system-helpers. This should only fail in
# situations where there are multiple versions of init-system-helpers in
# different suites.
ver=$(env --chdir="$1" APT_CONFIG="$MMDEBSTRAP_APT_CONFIG" apt-cache show --no-all-versions init-system-helpers 2>/dev/null | sed -ne 's/^Version: \(.*\)$/\1/p' || printf '')
if [ -z "$ver" ]; then
# there is no package called init-system-helpers, so either:
# - this is so old that init-system-helpers didn't exist yet
# - we are in a future where init-system-helpers doesn't exist anymore
# - something strange is going on
# we should only call the hook in the first case
ver=$(env --chdir="$1" APT_CONFIG="$MMDEBSTRAP_APT_CONFIG" apt-cache show --no-all-versions base-files 2>/dev/null | sed -ne 's/^Version: \(.*\)$/\1/p' || printf '')
if [ -z "$ver" ]; then
echo "neither init-system-helpers nor base-files can be installed -- not running jessie-or-older extract01 hook" >&2
exit 0
fi
# Jessie is Debian 8
if dpkg --compare-versions "$ver" ge 8; then
echo "there is no init-system-helpers but base-files version $ver is >= 8 -- not running jessie-or-older extract01 hook" >&2
exit 0
else
echo "there is no init-system-helpers but base-files version $ver is << 8 -- running jessie-or-older extract01 hook" >&2
fi
else
if dpkg --compare-versions "$ver" ge 1.34; then
echo "init-system-helpers version $ver is >= 1.34 -- not running jessie-or-older extract01 hook" >&2
exit 0
else
echo "init-system-helpers version $ver is << 1.34 -- running jessie-or-older extract01 hook" >&2
fi
fi
# resolve the script path using several methods in order:
# 1. using dirname -- "$0"
# 2. using ./hooks
# 3. using /usr/share/mmdebstrap/hooks/
for p in "$(dirname -- "$0")/.." ./hooks /usr/share/mmdebstrap/hooks; do
if [ -x "$p/jessie-or-older/extract00.sh" ] && [ -x "$p/jessie-or-older/extract01.sh" ]; then
"$p/jessie-or-older/extract01.sh" "$1"
exit 0
fi
done
echo "cannot find jessie-or-older hook anywhere" >&2
exit 1

40
hooks/maybe-merged-usr/essential00.sh
View file

@ -1,40 +0,0 @@
#!/bin/sh
set -eu
ver=$(dpkg-query --root="$1" -f '${db:Status-Status} ${Source} ${Version}' --show usr-is-merged 2>/dev/null || printf '')
case "$ver" in
'')
echo "no package called usr-is-merged is installed -- not running merged-usr essential hook" >&2
exit 0
;;
'installed mmdebstrap-dummy-usr-is-merged 1')
echo "dummy usr-is-merged package installed -- running merged-usr essential hook" >&2
;;
'installed usrmerge '*)
echo "usr-is-merged package from src:usrmerge installed -- not running merged-usr essential hook" >&2
exit 0
;;
'not-installed ')
echo "usr-is-merged was not installed in a previous hook -- not running merged-usr essential hook" >&2
exit 0
;;
*)
echo "unexpected situation for package usr-is-merged: $ver" >&2
exit 1
;;
esac
# resolve the script path using several methods in order:
# 1. using dirname -- "$0"
# 2. using ./hooks
# 3. using /usr/share/mmdebstrap/hooks/
for p in "$(dirname -- "$0")/.." ./hooks /usr/share/mmdebstrap/hooks; do
if [ -x "$p/merged-usr/setup00.sh" ] && [ -x "$p/merged-usr/extract00.sh" ] && [ -x "$p/merged-usr/essential00.sh" ]; then
"$p/merged-usr/essential00.sh" "$1"
exit 0
fi
done
echo "cannot find merged-usr hook anywhere" >&2
exit 1

37
hooks/maybe-merged-usr/extract00.sh
View file

@ -1,37 +0,0 @@
#!/bin/sh
set -eu
env --chdir="$1" APT_CONFIG="$MMDEBSTRAP_APT_CONFIG" apt-get update --error-on=any
if env --chdir="$1" APT_CONFIG="$MMDEBSTRAP_APT_CONFIG" apt-cache show --no-all-versions usr-is-merged > /dev/null 2>&1; then
# if apt-cache exited successfully, then usr-is-merged exists either as
# a real or virtual package
if env --chdir="$1" APT_CONFIG="$MMDEBSTRAP_APT_CONFIG" apt-cache show --no-all-versions usr-is-merged 2>/dev/null | grep -q "Package: usr-is-merged"; then
echo "usr-is-merged found -- running merged-usr extract hook" >&2
else
# The usr-is-merged must be virtual, so assume that nothing
# has to be done. This is the case with Debian Trixie or later
# or with Ubuntu Lunar or later
echo "usr-is-merged found but not real -- not running merged-usr extract hook" >&2
exit 0
fi
else
# if the usr-is-merged package cannot be installed with apt, do nothing
echo "no package providing usr-is-merged found -- not running merged-usr extract hook" >&2
exit 0
fi
# resolve the script path using several methods in order:
# 1. using dirname -- "$0"
# 2. using ./hooks
# 3. using /usr/share/mmdebstrap/hooks/
for p in "$(dirname -- "$0")/.." ./hooks /usr/share/mmdebstrap/hooks; do
if [ -x "$p/merged-usr/setup00.sh" ] && [ -x "$p/merged-usr/extract00.sh" ] && [ -x "$p/merged-usr/essential00.sh" ]; then
"$p/merged-usr/extract00.sh" "$1"
exit 0
fi
done
echo "cannot find merged-usr hook anywhere" >&2
exit 1

37
hooks/maybe-merged-usr/setup00.sh
View file

@ -1,37 +0,0 @@
#!/bin/sh
set -eu
env --chdir="$1" APT_CONFIG="$MMDEBSTRAP_APT_CONFIG" apt-get update --error-on=any
if env --chdir="$1" APT_CONFIG="$MMDEBSTRAP_APT_CONFIG" apt-cache show --no-all-versions usr-is-merged > /dev/null 2>&1; then
# if apt-cache exited successfully, then usr-is-merged exists either as
# a real or virtual package
if env --chdir="$1" APT_CONFIG="$MMDEBSTRAP_APT_CONFIG" apt-cache show --no-all-versions usr-is-merged 2>/dev/null | grep -q "Package: usr-is-merged"; then
echo "usr-is-merged found -- running merged-usr setup hook" >&2
else
# The usr-is-merged must be virtual, so assume that nothing
# has to be done. This is the case with Debian Trixie or later
# or with Ubuntu Lunar or later
echo "usr-is-merged found but not real -- not running merged-usr setup hook" >&2
exit 0
fi
else
# if the usr-is-merged package cannot be installed with apt, do nothing
echo "no package providing usr-is-merged found -- not running merged-usr setup hook" >&2
exit 0
fi
# resolve the script path using several methods in order:
# 1. using dirname -- "$0"
# 2. using ./hooks
# 3. using /usr/share/mmdebstrap/hooks/
for p in "$(dirname -- "$0")/.." ./hooks /usr/share/mmdebstrap/hooks; do
if [ -x "$p/merged-usr/setup00.sh" ] && [ -x "$p/merged-usr/extract00.sh" ] && [ -x "$p/merged-usr/essential00.sh" ]; then
"$p/merged-usr/setup00.sh" "$1"
exit 0
fi
done
echo "cannot find merged-usr hook anywhere" >&2
exit 1

28
hooks/merged-usr/essential00.sh
View file

@ -1,28 +0,0 @@
#!/bin/sh
set -eu
if [ "${MMDEBSTRAP_VERBOSITY:-1}" -ge 3 ]; then
set -x
fi
TARGET="$1"
if [ "${MMDEBSTRAP_MODE:-}" = "chrootless" ]; then
APT_CONFIG=$MMDEBSTRAP_APT_CONFIG apt-get --yes install \
-oDPkg::Chroot-Directory= \
-oDPkg::Options::=--force-not-root \
-oDPkg::Options::=--force-script-chrootless \
-oDPkg::Options::=--root="$TARGET" \
-oDPkg::Options::=--log="$TARGET/var/log/dpkg.log" \
usr-is-merged
export DPKG_ROOT="$TARGET"
dpkg-query --showformat '${db:Status-Status}\n' --show usr-is-merged | grep -q '^installed$'
dpkg-query --showformat '${Source}\n' --show usr-is-merged | grep -q '^usrmerge$'
dpkg --compare-versions "1" "lt" "$(dpkg-query --showformat '${Version}\n' --show usr-is-merged)"
else
APT_CONFIG=$MMDEBSTRAP_APT_CONFIG apt-get --yes install usr-is-merged
chroot "$TARGET" dpkg-query --showformat '${db:Status-Status}\n' --show usr-is-merged | grep -q '^installed$'
chroot "$TARGET" dpkg-query --showformat '${Source}\n' --show usr-is-merged | grep -q '^usrmerge$'
dpkg --compare-versions "1" "lt" "$(chroot "$TARGET" dpkg-query --showformat '${Version}\n' --show usr-is-merged)"
fi

85
hooks/merged-usr/extract00.sh
View file

@ -1,85 +0,0 @@
#!/bin/sh
set -eu
if [ "${MMDEBSTRAP_VERBOSITY:-1}" -ge 3 ]; then
set -x
fi
TARGET="$1"
# can_usrmerge_symlink() and can_usrmerge_symlink() are
# Copyright 2023 Helmut Grohne <helmut@subdivi.de>
# and part of the debootstrap source in /usr/share/debootstrap/functions
# https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/96
# https://bugs.debian.org/104989
can_usrmerge_symlink() {
# Absolute symlinks can be relocated without problems.
test "${2#/}" = "$2" || return 0
while :; do
if test "${2#/}" != "$2"; then
# Handle double-slashes.
set -- "$1" "${2#/}"
elif test "${2#./}" != "$2"; then
# Handle ./ inside a link target.
set -- "$1" "${2#./}"
elif test "$2" = ..; then
# A parent directory symlink is ok if it does not
# cross the top level directory.
test "${1%/*/*}" != "$1" -a -n "${1%/*/*}"
return $?
elif test "${2#../}" != "$2"; then
# Symbolic link crossing / cannot be moved safely.
# This is prohibited by Debian Policy 10.5.
test "${1%/*/*}" = "$1" -o -z "${1%/*/*}" && return 1
set -- "${1%/*}" "${2#../}"
else
# Consider the symlink ok if its target does not
# contain a parent directory. When we fail here,
# the link target is non-minimal and doesn't happen
# in the archive.
test "${2#*/../}" = "$2"
return $?
fi
done
}
merge_usr_entry() {
# shellcheck disable=SC3043
local entry canon
canon="$TARGET/usr/${1#"$TARGET/"}"
test -h "$canon" &&
error 1 USRMERGEFAIL "cannot move %s as its destination exists as a symlink" "${1#"$TARGET"}"
if ! test -e "$canon"; then
mv "$1" "$canon"
return 0
fi
test -d "$1" ||
error 1 USRMERGEFAIL "cannot move non-directory %s as its destination exists" "${1#"$TARGET"}"
test -d "$canon" ||
error 1 USRMERGEFAIL "cannot move directory %s as its destination is not a directory" "${1#"$TARGET"}"
for entry in "$1/"* "$1/."*; do
# Some shells return . and .. on dot globs.
test "${entry%/.}" != "${entry%/..}" && continue
if test -h "$entry" && ! can_usrmerge_symlink "${entry#"$TARGET"}" "$(readlink "$entry")"; then
error 1 USRMERGEFAIL "cannot move relative symlink crossing top-level directory" "${entry#"$TARGET"}"
fi
# Ignore glob match failures
if test "${entry%'/*'}" != "${entry%'/.*'}" && ! test -e "$entry"; then
continue
fi
merge_usr_entry "$entry"
done
rmdir "$1"
}
# This is list includes all possible multilib directories. It must be
# updated when new multilib directories are being added. Hopefully,
# all new architectures use multiarch instead, so we never get to
# update this.
for dir in bin lib lib32 lib64 libo32 libx32 sbin; do
test -h "$TARGET/$dir" && continue
test -e "$TARGET/$dir" || continue
merge_usr_entry "$TARGET/$dir"
ln -s "usr/$dir" "$TARGET/$dir"
done

79
hooks/merged-usr/setup00.sh
View file

@ -1,79 +0,0 @@
#!/bin/sh
#
# mmdebstrap does have a --merged-usr option but only as a no-op for
# debootstrap compatibility
#
# Using this hook script, you can emulate what debootstrap does to set up
# merged /usr via directory symlinks, even using the exact same shell function
# that debootstrap uses by running mmdebstrap with:
#
# --setup-hook=/usr/share/mmdebstrap/hooks/merged-usr/setup00.sh
#
# Alternatively, you can setup merged-/usr by installing the usrmerge package:
#
# --include=usrmerge
#
# mmdebstrap will not include this functionality via a --merged-usr option
# because there are many reasons against implementing merged-/usr that way:
#
# https://wiki.debian.org/Teams/Dpkg/MergedUsr
# https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Does_dpkg_support_merged-.2Fusr-via-aliased-dirs.3F
# https://lists.debian.org/20190219044924.GB21901@gaara.hadrons.org
# https://lists.debian.org/YAkLOMIocggdprSQ@thunder.hadrons.org
# https://lists.debian.org/20181223030614.GA8788@gaara.hadrons.org
#
# In addition, the merged-/usr-via-aliased-dirs approach violates an important
# principle of component based software engineering one of the core design
# ideas/goals of mmdebstrap: All the information to create a chroot of a Debian
# based distribution should be included in its packages and their metadata.
# Using directory symlinks as used by debootstrap contradicts this principle.
# The information whether a distribution uses this approach to merged-/usr or
# not is not anymore contained in its packages but in a tool from the outside.
#
# Example real world problem: I'm using debbisect to bisect Debian unstable
# between 2015 and today. For which snapshot.d.o timestamp should a merged-/usr
# chroot be created and for which ones not?
#
# The problem is not the idea of merged-/usr but the problem is the way how it
# got implemented in debootstrap via directory symlinks. That way of rolling
# out merged-/usr is bad from the dpkg point-of-view and completely opposite of
# the vision with which in mind I wrote mmdebstrap.
set -eu
if [ "${MMDEBSTRAP_VERBOSITY:-1}" -ge 3 ]; then
set -x
fi
TARGET="$1"
# now install an empty "usr-is-merged" package to avoid installing the
# usrmerge package on this system even after init-system-helpers starts
# depending on "usrmerge | usr-is-merged".
#
# This package will not end up in the final chroot because the essential
# hook replaces it with the actual usr-is-merged package from src:usrmerge.
tmpdir=$(mktemp --directory --tmpdir="$TARGET/tmp")
mkdir -p "$tmpdir/usr-is-merged/DEBIAN"
cat << END > "$tmpdir/usr-is-merged/DEBIAN/control"
Package: usr-is-merged
Priority: optional
Section: oldlibs
Maintainer: Johannes Schauer Marin Rodrigues <josch@debian.org>
Architecture: all
Multi-Arch: foreign
Source: mmdebstrap-dummy-usr-is-merged
Version: 1
Description: dummy package created by mmdebstrap merged-usr setup hook
This package was generated and installed by the mmdebstrap merged-usr
setup hook at /usr/share/mmdebstrap/hooks/merged-usr.
.
If this package is installed in the final chroot, then this is a bug
in mmdebstrap. Please report: https://gitlab.mister-muffin.de/josch/mmdebstrap
END
dpkg-deb --build "$tmpdir/usr-is-merged" "$tmpdir/usr-is-merged.deb"
dpkg --root="$TARGET" --log="$TARGET/var/log/dpkg.log" --install "$tmpdir/usr-is-merged.deb"
rm "$tmpdir/usr-is-merged.deb" "$tmpdir/usr-is-merged/DEBIAN/control"
rmdir "$tmpdir/usr-is-merged/DEBIAN" "$tmpdir/usr-is-merged" "$tmpdir"

1
hooks/no-merged-usr/essential00.sh
View file

@ -1 +0,0 @@
../merged-usr/essential00.sh

54
hooks/no-merged-usr/setup00.sh
View file

@ -1,54 +0,0 @@
#!/bin/sh
#
# mmdebstrap does have a --no-merged-usr option but only as a no-op for
# debootstrap compatibility
#
# Using this hook script, you can emulate what debootstrap does to set up
# a system without merged-/usr even after the essential init-system-helpers
# package added a dependency on "usrmerge | usr-is-merged". By installing
# a dummy usr-is-merged package, it avoids pulling in the dependencies of
# the usrmerge package.
set -eu
if [ "${MMDEBSTRAP_VERBOSITY:-1}" -ge 3 ]; then
set -x
fi
TARGET="$1"
echo "Warning: starting with Debian 12 (Bookworm), systems without merged-/usr are not supported anymore" >&2
echo "Warning: starting with Debian 13 (Trixie), merged-/usr symlinks are shipped by packages in the essential-set making this hook ineffective" >&2
echo "this system will not be supported in the future" > "$TARGET/etc/unsupported-skip-usrmerge-conversion"
# now install an empty "usr-is-merged" package to avoid installing the
# usrmerge package on this system even after init-system-helpers starts
# depending on "usrmerge | usr-is-merged".
#
# This package will not end up in the final chroot because the essential
# hook replaces it with the actual usr-is-merged package from src:usrmerge.
tmpdir=$(mktemp --directory --tmpdir="$TARGET/tmp")
mkdir -p "$tmpdir/usr-is-merged/DEBIAN"
cat << END > "$tmpdir/usr-is-merged/DEBIAN/control"
Package: usr-is-merged
Priority: optional
Section: oldlibs
Maintainer: Johannes Schauer Marin Rodrigues <josch@debian.org>
Architecture: all
Multi-Arch: foreign
Source: mmdebstrap-dummy-usr-is-merged
Version: 1
Description: dummy package created by mmdebstrap no-merged-usr setup hook
This package was generated and installed by the mmdebstrap no-merged-usr
setup hook at /usr/share/mmdebstrap/hooks/no-merged-usr.
.
If this package is installed in the final chroot, then this is a bug
in mmdebstrap. Please report: https://gitlab.mister-muffin.de/josch/mmdebstrap
END
dpkg-deb --build "$tmpdir/usr-is-merged" "$tmpdir/usr-is-merged.deb"
dpkg --root="$TARGET" --log="$TARGET/var/log/dpkg.log" --install "$tmpdir/usr-is-merged.deb"
rm "$tmpdir/usr-is-merged.deb" "$tmpdir/usr-is-merged/DEBIAN/control"
rmdir "$tmpdir/usr-is-merged/DEBIAN" "$tmpdir/usr-is-merged" "$tmpdir"

71
hooks/setup00-merged-usr.sh Executable file
View file

@ -0,0 +1,71 @@
#!/bin/sh
#
# mmdebstrap does have a --merged-usr option but only as a no-op for
# debootstrap compatibility
#
# Using this hook script, you can emulate what debootstrap does to set up
# merged /usr via directory symlinks, even using the exact same shell function
# that debootstrap uses.
#
# mmdebstrap will not include this functionality via a --merged-usr option
# because there are many reasons against implementing merged-/usr that way:
#
# https://wiki.debian.org/Teams/Dpkg/MergedUsr
# https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Does_dpkg_support_merged-.2Fusr-via-aliased-dirs.3F
# https://lists.debian.org/20190219044924.GB21901@gaara.hadrons.org
# https://lists.debian.org/YAkLOMIocggdprSQ@thunder.hadrons.org
# https://lists.debian.org/20181223030614.GA8788@gaara.hadrons.org
#
# In addition, the merged-/usr-via-aliased-dirs approach violates an important
# principle of component based software engineering one of the core design
# ideas/goals of mmdebstrap: All the information to create a chroot of a Debian
# based distribution should be included in its packages and their metadata.
# Using directory symlinks as used by debootstrap contradicts this principle.
# The information whether a distribution uses this approach to merged-/usr or
# not is not anymore contained in its packages but in a tool from the outside.
#
# The problem is not the idea of merged-/usr but the problem is the way how it
# got implemented in debootstrap via directory symlinks. That way of rolling
# out merged-/usr is bad from the dpkg point-of-view and completely opposite of
# the vision with which in mind I wrote mmdebstrap.
set -exu
TARGET="$1"
if [ -e "$TARGET/var/lib/dpkg/arch" ]; then
ARCH=$(head -1 "$TARGET/var/lib/dpkg/arch")
else
ARCH=$(dpkg --print-architecture)
fi
if [ -e /usr/share/debootstrap/functions ]; then
. /usr/share/debootstrap/functions
doing_variant () { [ $1 != "buildd" ]; }
MERGED_USR="yes"
# until https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/48 gets merged
link_dir=""
setup_merged_usr
else
link_dir=""
case $ARCH in
hurd-*) exit 0;;
amd64) link_dir="lib32 lib64 libx32" ;;
i386) link_dir="lib64 libx32" ;;
mips|mipsel) link_dir="lib32 lib64" ;;
mips64*|mipsn32*) link_dir="lib32 lib64 libo32" ;;
powerpc) link_dir="lib64" ;;
ppc64) link_dir="lib32 lib64" ;;
ppc64el) link_dir="lib64" ;;
s390x) link_dir="lib32" ;;
sparc) link_dir="lib64" ;;
sparc64) link_dir="lib32 lib64" ;;
x32) link_dir="lib32 lib64 libx32" ;;
esac
link_dir="bin sbin lib $link_dir"
for dir in $link_dir; do
ln -s usr/"$dir" "$TARGET/$dir"
mkdir -p "$TARGET/usr/$dir"
done
fi

131
ldconfig.fakechroot
View file

@ -1,131 +0,0 @@
#!/usr/bin/env python3
#
# This script is in the public domain
#
# Author: Johannes Schauer Marin Rodrigues <josch@mister-muffin.de>
#
# This is command substitution for ldconfig under fakechroot:
#
# export FAKECHROOT_CMD_SUBST=/sbin/ldconfig=/path/to/ldconfig.fakechroot
#
# Statically linked binaries cannot work with fakechroot and thus have to be
# replaced by either /bin/true or a more clever solution like this one. The
# ldconfig command supports the -r option which allows passing a chroot
# directory for ldconfig to work in. This can be used to run ldconfig without
# fakechroot but still let it create /etc/ld.so.cache inside the chroot.
#
# Since absolute symlinks are broken without fakechroot to translate them,
# we read /etc/ld.so.conf and turn all absolute symlink shared libraries into
# relative ones. At program exit, the original state is restored.
import os
import sys
import subprocess
import atexit
import glob
from pathlib import Path
symlinks = []
def restore_symlinks():
for (link, target, atime, mtime) in symlinks:
link.unlink()
link.symlink_to(target)
os.utime(link, times=None, ns=(atime, mtime), follow_symlinks=False)
atexit.register(restore_symlinks)
def get_libdirs(chroot, configs):
res = []
for conf in configs:
for line in (Path(conf)).read_text().splitlines():
line = line.strip()
if not line:
continue
if line.startswith("#"):
continue
if line.startswith("include "):
assert line.startswith("include /")
res.extend(
get_libdirs(chroot, chroot.glob(line.removeprefix("include /")))
)
continue
assert line.startswith("/"), line
line = line.lstrip("/")
if not (chroot / Path(line)).is_dir():
continue
for f in (chroot / Path(line)).iterdir():
if not f.is_symlink():
continue
linktarget = f.readlink()
# make sure that the linktarget is an absolute path inside the
# chroot
if not str(linktarget).startswith("/"):
continue
if chroot not in linktarget.parents:
continue
# store original link so that we can restore it later
symlinks.append(
(f, linktarget, f.lstat().st_atime_ns, f.lstat().st_mtime_ns)
)
# replace absolute symlink by relative link
relative = os.path.relpath(linktarget, f.parent)
f.unlink()
f.symlink_to(relative)
return res
def main():
if "FAKECHROOT_BASE_ORIG" not in os.environ:
print("FAKECHROOT_BASE_ORIG is not set", file=sys.stderr)
print(
"must be executed under fakechroot using FAKECHROOT_CMD_SUBST",
file=sys.stderr,
)
sys.exit(1)
chroot = Path(os.environ["FAKECHROOT_BASE_ORIG"])
# if chrootless mode is used from within a fakechroot chroot, then
# FAKECHROOT_BASE_ORIG will point at the outer chroot. We want to use
# the path from DPKG_ROOT inside of that instead
if os.environ.get("DPKG_ROOT", "") not in ["", "/"]:
chroot /= os.environ["DPKG_ROOT"].lstrip("/")
if not (chroot / "sbin" / "ldconfig").exists():
sys.exit(0)
(chroot / "var" / "cache" / "ldconfig").mkdir(
mode=0o700, parents=True, exist_ok=True
)
for d in get_libdirs(chroot, [chroot / "etc" / "ld.so.conf"]):
make_relative(d)
rootarg = chroot
argv = sys.argv[1:]
for arg in sys.argv[1:]:
if arg == "-r":
rootarg = None
elif rootarg is None:
argpath = Path(arg)
if argpath.is_absolute():
rootarg = chroot / argpath.relative_to("/")
else:
rootarg = Path.cwd() / argpath
if rootarg is None:
rootarg = chroot
# we add any additional arguments before "-r" such that any other "-r"
# option will be overwritten by the one we set
subprocess.check_call(
[chroot / "sbin" / "ldconfig"] + sys.argv[1:] + ["-r", rootarg]
)
if __name__ == "__main__":
main()

877
make_mirror.sh
View file

@ -10,160 +10,198 @@ set -eu
# the local repository.
deletecache() {
dir="$1"
echo "running deletecache $dir" >&2
if [ ! -e "$dir" ]; then
return
fi
if [ ! -e "$dir/mmdebstrapcache" ]; then
echo "$dir cannot be the mmdebstrap cache" >&2
return 1
fi
# be very careful with removing the old directory
# experimental is pulled in with USE_HOST_APT_CONFIG=yes on debci
# when testing a package from experimental
for dist in oldstable stable testing unstable experimental; do
# deleting artifacts from test "debootstrap"
for variant in minbase buildd -; do
if [ -e "$dir/debian-$dist-$variant.tar" ]; then
rm "$dir/debian-$dist-$variant.tar"
else
echo "does not exist: $dir/debian-$dist-$variant.tar" >&2
fi
done
# deleting artifacts from test "mmdebstrap"
for variant in essential apt minbase buildd - standard; do
for format in tar ext2 ext4 squashfs; do
if [ -e "$dir/mmdebstrap-$dist-$variant.$format" ]; then
# attempt to delete for all dists because DEFAULT_DIST might've been different the last time
rm "$dir/mmdebstrap-$dist-$variant.$format"
elif [ "$dist" = "$DEFAULT_DIST" ]; then
# only warn about non-existance when it's expected to exist
echo "does not exist: $dir/mmdebstrap-$dist-$variant.$format" >&2
fi
done
done
if [ -e "$dir/debian/dists/$dist" ]; then
rm --one-file-system --recursive "$dir/debian/dists/$dist"
else
echo "does not exist: $dir/debian/dists/$dist" >&2
fi
case "$dist" in oldstable | stable)
if [ -e "$dir/debian/dists/$dist-updates" ]; then
rm --one-file-system --recursive "$dir/debian/dists/$dist-updates"
else
echo "does not exist: $dir/debian/dists/$dist-updates" >&2
fi
;;
esac
case "$dist" in oldstable | stable)
if [ -e "$dir/debian-security/dists/$dist-security" ]; then
rm --one-file-system --recursive "$dir/debian-security/dists/$dist-security"
else
echo "does not exist: $dir/debian-security/dists/$dist-security" >&2
fi
;;
esac
done
for f in "$dir/debian-"*.ext4; do
if [ -e "$f" ]; then
rm --one-file-system "$f"
fi
done
# on i386 and amd64, the intel-microcode and amd64-microcode packages
# from non-free-firwame get pulled in because they are
# priority:standard with USE_HOST_APT_CONFIG=yes
for c in main non-free-firmware; do
if [ -e "$dir/debian/pool/$c" ]; then
rm --one-file-system --recursive "$dir/debian/pool/$c"
else
echo "does not exist: $dir/debian/pool/$c" >&2
fi
done
if [ -e "$dir/debian-security/pool/updates/main" ]; then
rm --one-file-system --recursive "$dir/debian-security/pool/updates/main"
else
echo "does not exist: $dir/debian-security/pool/updates/main" >&2
fi
for i in $(seq 1 6); do
if [ ! -e "$dir/debian$i" ]; then
continue
fi
rm "$dir/debian$i"
done
rm "$dir/mmdebstrapcache"
# remove all symlinks
find "$dir" -type l -delete
# now the rest should only be empty directories
if [ -e "$dir" ]; then
find "$dir" -depth -print0 | xargs -0 --no-run-if-empty rmdir
else
echo "does not exist: $dir" >&2
fi
dir="$1"
echo "running deletecache $dir">&2
if [ ! -e "$dir" ]; then
return
fi
if [ ! -e "$dir/mmdebstrapcache" ]; then
echo "$dir cannot be the mmdebstrap cache" >&2
return 1
fi
# be very careful with removing the old directory
for dist in stable testing unstable; do
for variant in minbase buildd -; do
if [ -e "$dir/debian-$dist-$variant.tar" ]; then
rm "$dir/debian-$dist-$variant.tar"
else
echo "does not exist: $dir/debian-$dist-$variant.tar" >&2
fi
done
if [ -e "$dir/debian/dists/$dist" ]; then
rm --one-file-system --recursive "$dir/debian/dists/$dist"
else
echo "does not exist: $dir/debian/dists/$dist" >&2
fi
if [ "$dist" = "stable" ]; then
if [ -e "$dir/debian/dists/stable-updates" ]; then
rm --one-file-system --recursive "$dir/debian/dists/stable-updates"
else
echo "does not exist: $dir/debian/dists/stable-updates" >&2
fi
if [ -e "$dir/debian-security/dists/stable/updates" ]; then
rm --one-file-system --recursive "$dir/debian-security/dists/stable/updates"
else
echo "does not exist: $dir/debian-security/dists/stable/updates" >&2
fi
fi
done
if [ -e $dir/debian-*.qcow ]; then
rm --one-file-system "$dir"/debian-*.qcow
else
echo "does not exist: $dir/debian-*.qcow" >&2
fi
if [ -e "$dir/debian/pool/main" ]; then
rm --one-file-system --recursive "$dir/debian/pool/main"
else
echo "does not exist: $dir/debian/pool/main" >&2
fi
if [ -e "$dir/debian-security/pool/updates/main" ]; then
rm --one-file-system --recursive "$dir/debian-security/pool/updates/main"
else
echo "does not exist: $dir/debian-security/pool/updates/main" >&2
fi
for i in $(seq 1 6); do
if [ ! -e "$dir/debian$i" ]; then
continue
fi
rm "$dir/debian$i"
done
rm "$dir/mmdebstrapcache"
# now the rest should only be empty directories
if [ -e "$dir" ]; then
find "$dir" -depth -print0 | xargs -0 --no-run-if-empty rmdir
else
echo "does not exist: $dir" >&2
fi
}
cleanup_newcachedir() {
echo "running cleanup_newcachedir"
deletecache "$newcachedir"
echo "running cleanup_newcachedir"
deletecache "$newcachedir"
}
get_oldaptnames() {
if [ ! -e "$1/$2" ]; then
return
fi
gzip -dc "$1/$2" \
| grep-dctrl --no-field-names --show-field=Package,Version,Architecture,Filename '' \
| paste -sd " \n" \
| while read name ver arch fname; do
if [ ! -e "$1/$fname" ]; then
continue
fi
# apt stores deb files with the colon encoded as %3a while
# mirrors do not contain the epoch at all #645895
case "$ver" in *:*) ver="${ver%%:*}%3a${ver#*:}";; esac
aptname="$rootdir/var/cache/apt/archives/${name}_${ver}_${arch}.deb"
# we have to cp and not mv because other
# distributions might still need this file
# we have to cp and not symlink because apt
# doesn't recognize symlinks
cp --link "$1/$fname" "$aptname"
echo "$aptname"
done
}
get_newaptnames() {
if [ ! -e "$1/$2" ]; then
return
fi
# skip empty files by trying to uncompress the first byte of the payload
if [ "$(gzip -dc "$1/$2" | head -c1 | wc -c)" -eq 0 ]; then
return
fi
gzip -dc "$1/$2" \
| grep-dctrl --no-field-names --show-field=Package,Version,Architecture,Filename,SHA256 '' \
| paste -sd " \n" \
| while read name ver arch fname hash; do
# sanity check for the hash because sometimes the
# archive switches the hash algorithm
if [ "${#hash}" -ne 64 ]; then
echo "expected hash length of 64 but got ${#hash} for: $hash" >&2
exit 1
fi
dir="${fname%/*}"
# apt stores deb files with the colon encoded as %3a while
# mirrors do not contain the epoch at all #645895
case "$ver" in *:*) ver="${ver%%:*}%3a${ver#*:}";; esac
aptname="$rootdir/var/cache/apt/archives/${name}_${ver}_${arch}.deb"
if [ -e "$aptname" ]; then
# make sure that we found the right file by checking its hash
echo "$hash $aptname" | sha256sum --check >&2
mkdir -p "$1/$dir"
# since we move hardlinks around, the same hardlink might've been
# moved already into the same place by another distribution.
# mv(1) refuses to copy A to B if both are hardlinks of each other.
if [ "$aptname" -ef "$1/$fname" ]; then
# both files are already the same so we just need to
# delete the source
rm "$aptname"
else
mv "$aptname" "$1/$fname"
fi
echo "$aptname"
fi
done
}
cleanupapt() {
echo "running cleanupapt" >&2
if [ ! -e "$rootdir" ]; then
return
fi
for f in \
"$rootdir/var/cache/apt/archives/"*.deb \
"$rootdir/var/cache/apt/archives/partial/"*.deb \
"$rootdir/var/cache/apt/"*.bin \
"$rootdir/var/lib/apt/lists/"* \
"$rootdir/var/lib/dpkg/status" \
"$rootdir/var/lib/dpkg/lock-frontend" \
"$rootdir/var/lib/dpkg/lock" \
"$rootdir/var/lib/apt/lists/lock" \
"$rootdir/etc/apt/apt.conf" \
"$rootdir/etc/apt/sources.list.d/"* \
"$rootdir/etc/apt/preferences.d/"* \
"$rootdir/etc/apt/sources.list" \
"$rootdir/var/cache/apt/archives/lock"; do
if [ ! -e "$f" ]; then
echo "does not exist: $f" >&2
continue
fi
if [ -d "$f" ]; then
rmdir "$f"
else
rm "$f"
fi
done
find "$rootdir" -depth -print0 | xargs -0 --no-run-if-empty rmdir
echo "running cleanupapt" >&2
if [ ! -e "$rootdir" ]; then
return
fi
for f in \
"$rootdir/var/cache/apt/archives/"*.deb \
"$rootdir/var/cache/apt/archives/partial/"*.deb \
"$rootdir/var/cache/apt/"*.bin \
"$rootdir/var/lib/apt/lists/"* \
"$rootdir/var/lib/dpkg/status" \
"$rootdir/var/lib/dpkg/lock-frontend" \
"$rootdir/var/lib/dpkg/lock" \
"$rootdir/etc/apt/apt.conf" \
"$rootdir/etc/apt/sources.list" \
"$rootdir/oldaptnames" \
"$rootdir/newaptnames" \
"$rootdir/var/cache/apt/archives/lock"; do
if [ ! -e "$f" ]; then
echo "does not exist: $f" >&2
continue
fi
if [ -d "$f" ]; then
rmdir "$f"
else
rm "$f"
fi
done
find "$rootdir" -depth -print0 | xargs -0 --no-run-if-empty rmdir
}
# note: this function uses brackets instead of curly braces, so that it's run
# in its own process and we can handle traps independent from the outside
update_cache() (
dist="$1"
nativearch="$2"
dist="$1"
nativearch="$2"
# use a subdirectory of $newcachedir so that we can use
# hardlinks
rootdir="$newcachedir/apt"
mkdir -p "$rootdir"
# use a subdirectory of $newcachedir so that we can use
# hardlinks
rootdir="$newcachedir/apt"
mkdir -p "$rootdir"
# we only set this trap here and overwrite the previous trap, because
# the update_cache function is run as part of a pipe and thus in its
# own process which will EXIT after it finished
trap 'kill "$PROXYPID" || :;cleanupapt' EXIT INT TERM
# we only set this trap here and overwrite the previous trap, because
# the update_cache function is run as part of a pipe and thus in its
# own process which will EXIT after it finished
trap "cleanupapt" EXIT INT TERM
for p in /etc/apt/apt.conf.d /etc/apt/sources.list.d /etc/apt/preferences.d /var/cache/apt/archives /var/lib/apt/lists/partial /var/lib/dpkg; do
mkdir -p "$rootdir/$p"
done
for p in /etc/apt/apt.conf.d /etc/apt/sources.list.d /etc/apt/preferences.d /var/cache/apt/archives /var/lib/apt/lists/partial /var/lib/dpkg; do
mkdir -p "$rootdir/$p"
done
# read sources.list content from stdin
cat >"$rootdir/etc/apt/sources.list"
# read sources.list content from stdin
cat > "$rootdir/etc/apt/sources.list"
cat <<END >"$rootdir/etc/apt/apt.conf"
cat << END > "$rootdir/etc/apt/apt.conf"
Apt::Architecture "$nativearch";
Apt::Architectures "$nativearch";
Dir::Etc "$rootdir/etc/apt";
@ -174,128 +212,116 @@ Apt::Get::Download-Only true;
Acquire::Languages "none";
Dir::Etc::Trusted "/etc/apt/trusted.gpg";
Dir::Etc::TrustedParts "/etc/apt/trusted.gpg.d";
Acquire::http::Proxy "http://127.0.0.1:8080/";
END
: >"$rootdir/var/lib/dpkg/status"
> "$rootdir/var/lib/dpkg/status"
if [ "$dist" = "$DEFAULT_DIST" ] && [ "$nativearch" = "$HOSTARCH" ] && [ "$USE_HOST_APT_CONFIG" = "yes" ]; then
# we append sources and settings instead of overwriting after
# an empty line
for f in /etc/apt/sources.list /etc/apt/sources.list.d/*; do
[ -e "$f" ] || continue
[ -e "$rootdir/$f" ] && echo >>"$rootdir/$f"
# Filter out file:// repositories as they are added
# to each mmdebstrap call verbatim by
# debian/tests/copy_host_apt_config
# Also filter out all mirrors that are not of suite
# $DEFAULT_DIST, except experimental if the suite
# is unstable. This prevents packages from
# unstable entering a testing mirror.
if [ "$dist" = unstable ]; then
grep -v ' file://' "$f" \
| grep -E " (unstable|experimental) " \
>>"$rootdir/$f" || :
else
grep -v ' file://' "$f" \
| grep " $DEFAULT_DIST " \
>>"$rootdir/$f" || :
fi
done
for f in /etc/apt/preferences.d/*; do
[ -e "$f" ] || continue
[ -e "$rootdir/$f" ] && echo >>"$rootdir/$f"
cat "$f" >>"$rootdir/$f"
done
fi
echo "creating mirror for $dist" >&2
for f in /etc/apt/sources.list /etc/apt/sources.list.d/* /etc/apt/preferences.d/*; do
[ -e "$rootdir/$f" ] || continue
echo "contents of $f:" >&2
cat "$rootdir/$f" >&2
done
APT_CONFIG="$rootdir/etc/apt/apt.conf" apt-get update
APT_CONFIG="$rootdir/etc/apt/apt.conf" apt-get update --error-on=any
# before downloading packages and before replacing the old Packages
# file, copy all old *.deb packages from the mirror to
# /var/cache/apt/archives so that apt will not re-download *.deb
# packages that we already have
{
get_oldaptnames "$oldmirrordir" "dists/$dist/main/binary-$nativearch/Packages.gz"
if grep --quiet security.debian.org "$rootdir/etc/apt/sources.list"; then
get_oldaptnames "$oldmirrordir" "dists/stable-updates/main/binary-$nativearch/Packages.gz"
get_oldaptnames "$oldcachedir/debian-security" "dists/stable/updates/main/binary-$nativearch/Packages.gz"
fi
} | sort -u > "$rootdir/oldaptnames"
pkgs=$(APT_CONFIG="$rootdir/etc/apt/apt.conf" apt-get indextargets \
--format '$(FILENAME)' 'Created-By: Packages' "Architecture: $nativearch" \
| xargs --delimiter='\n' /usr/lib/apt/apt-helper cat-file \
| grep-dctrl --no-field-names --show-field=Package --exact-match \
\( --field=Essential yes --or --field=Priority required \
--or --field=Priority important --or --field=Priority standard \
\))
pkgs=$(APT_CONFIG="$rootdir/etc/apt/apt.conf" apt-get indextargets \
--format '$(FILENAME)' 'Created-By: Packages' "Architecture: $nativearch" \
| xargs --delimiter='\n' /usr/lib/apt/apt-helper cat-file \
| grep-dctrl --no-field-names --show-field=Package --exact-match \
\( --field=Essential yes --or --field=Priority required \
--or --field=Priority important --or --field=Priority standard \
\))
pkgs="$pkgs build-essential busybox gpg eatmydata fakechroot fakeroot"
pkgs="$(echo $pkgs) build-essential busybox gpg eatmydata"
# we need usr-is-merged to simulate debootstrap behaviour for all dists
# starting from Debian 12 (Bullseye)
case "$dist" in
oldstable) : ;;
*) pkgs="$pkgs usr-is-merged usrmerge" ;;
esac
APT_CONFIG="$rootdir/etc/apt/apt.conf" apt-get --yes install $pkgs
# shellcheck disable=SC2086
APT_CONFIG="$rootdir/etc/apt/apt.conf" apt-get --yes install $pkgs \
|| APT_CONFIG="$rootdir/etc/apt/apt.conf" apt-get --yes install \
-oDebug::pkgProblemResolver=true -oDebug::pkgDepCache::Marker=1 \
-oDebug::pkgDepCache::AutoInstall=1 \
$pkgs
# to be able to also test gpg verification, we need to create a mirror
mkdir -p "$newmirrordir/dists/$dist/main/binary-$nativearch/"
curl --location "$mirror/dists/$dist/Release" > "$newmirrordir/dists/$dist/Release"
curl --location "$mirror/dists/$dist/Release.gpg" > "$newmirrordir/dists/$dist/Release.gpg"
curl --location "$mirror/dists/$dist/main/binary-$nativearch/Packages.gz" > "$newmirrordir/dists/$dist/main/binary-$nativearch/Packages.gz"
if grep --quiet security.debian.org "$rootdir/etc/apt/sources.list"; then
mkdir -p "$newmirrordir/dists/stable-updates/main/binary-$nativearch/"
curl --location "$mirror/dists/stable-updates/Release" > "$newmirrordir/dists/stable-updates/Release"
curl --location "$mirror/dists/stable-updates/Release.gpg" > "$newmirrordir/dists/stable-updates/Release.gpg"
curl --location "$mirror/dists/stable-updates/main/binary-$nativearch/Packages.gz" > "$newmirrordir/dists/stable-updates/main/binary-$nativearch/Packages.gz"
mkdir -p "$newcachedir/debian-security/dists/stable/updates/main/binary-$nativearch/"
curl --location "$security_mirror/dists/stable/updates/Release" > "$newcachedir/debian-security/dists/stable/updates/Release"
curl --location "$security_mirror/dists/stable/updates/Release.gpg" > "$newcachedir/debian-security/dists/stable/updates/Release.gpg"
curl --location "$security_mirror/dists/stable/updates/main/binary-$nativearch/Packages.gz" > "$newcachedir/debian-security/dists/stable/updates/main/binary-$nativearch/Packages.gz"
fi
rm "$rootdir/var/cache/apt/archives/lock"
rmdir "$rootdir/var/cache/apt/archives/partial"
APT_CONFIG="$rootdir/etc/apt/apt.conf" apt-get --option Dir::Etc::SourceList=/dev/null update
APT_CONFIG="$rootdir/etc/apt/apt.conf" apt-get clean
# the deb files downloaded by apt must be moved to their right locations in the
# pool directory
#
# Instead of parsing the Packages file, we could also attempt to move the deb
# files ourselves to the appropriate pool directories. But that approach
# requires re-creating the heuristic by which the directory is chosen, requires
# stripping the epoch from the filename and will break once mirrors change.
# This way, it doesn't matter where the mirror ends up storing the package.
{
get_newaptnames "$newmirrordir" "dists/$dist/main/binary-$nativearch/Packages.gz";
if grep --quiet security.debian.org "$rootdir/etc/apt/sources.list"; then
get_newaptnames "$newmirrordir" "dists/stable-updates/main/binary-$nativearch/Packages.gz"
get_newaptnames "$newcachedir/debian-security" "dists/stable/updates/main/binary-$nativearch/Packages.gz"
fi
} | sort -u > "$rootdir/newaptnames"
cleanupapt
rm "$rootdir/var/cache/apt/archives/lock"
rmdir "$rootdir/var/cache/apt/archives/partial"
# remove all packages that were in the old Packages file but not in the
# new one anymore
comm -23 "$rootdir/oldaptnames" "$rootdir/newaptnames" | xargs --delimiter="\n" --no-run-if-empty rm
# now the apt cache should be empty
if [ ! -z "$(ls -1qA "$rootdir/var/cache/apt/archives/")" ]; then
echo "$rootdir/var/cache/apt/archives not empty:"
ls -la "$rootdir/var/cache/apt/archives/"
exit 1
fi
# this function is run in its own process, so we unset all traps before
# returning
trap "-" EXIT INT TERM
APT_CONFIG="$rootdir/etc/apt/apt.conf" apt-get --option Dir::Etc::SourceList=/dev/null update
APT_CONFIG="$rootdir/etc/apt/apt.conf" apt-get clean
cleanupapt
# this function is run in its own process, so we unset all traps before
# returning
trap "-" EXIT INT TERM
)
check_proxy_running() {
if timeout 1 bash -c 'exec 3<>/dev/tcp/127.0.0.1/8080 && printf "GET http://deb.debian.org/debian/dists/'"$DEFAULT_DIST"'/InRelease HTTP/1.1\nHost: deb.debian.org\n\n" >&3 && grep "Suite: '"$DEFAULT_DIST"'" <&3 >/dev/null' 2>/dev/null; then
return 0
elif timeout 1 env http_proxy="http://127.0.0.1:8080/" wget --quiet -O - "http://deb.debian.org/debian/dists/$DEFAULT_DIST/InRelease" | grep "Suite: $DEFAULT_DIST" >/dev/null; then
return 0
elif timeout 1 curl --proxy "http://127.0.0.1:8080/" --silent "http://deb.debian.org/debian/dists/$DEFAULT_DIST/InRelease" | grep "Suite: $DEFAULT_DIST" >/dev/null; then
return 0
fi
return 1
}
if [ -e "./shared/cache.A" ] && [ -e "./shared/cache.B" ]; then
echo "both ./shared/cache.A and ./shared/cache.B exist" >&2
echo "was a former run of the script aborted?" >&2
if [ -e ./shared/cache ]; then
echo "cache symlink points to $(readlink ./shared/cache)" >&2
case "$(readlink ./shared/cache)" in
cache.A)
echo "removing ./shared/cache.B" >&2
rm -r ./shared/cache.B
;;
cache.B)
echo "removing ./shared/cache.A" >&2
rm -r ./shared/cache.A
;;
*)
echo "unexpected" >&2
exit 1
;;
esac
else
echo "./shared/cache doesn't exist" >&2
exit 1
fi
echo "both ./shared/cache.A and ./shared/cache.B exist" >&2
echo "was a former run of the script aborted?" >&2
if [ -e ./shared/cache ]; then
echo "cache symlink points to $(readlink ./shared/cache)" >&2
case "$(readlink ./shared/cache)" in
cache.A)
echo "maybe rm -r ./shared/cache.B" >&2
;;
cache.B)
echo "maybe rm -r ./shared/cache.A" >&2
;;
*)
echo "unexpected" >&2
esac
fi
exit 1
fi
if [ -e "./shared/cache.A" ]; then
oldcache=cache.A
newcache=cache.B
oldcache=cache.A
newcache=cache.B
else
oldcache=cache.B
newcache=cache.A
oldcache=cache.B
newcache=cache.A
fi
oldcachedir="./shared/$oldcache"
@ -309,175 +335,141 @@ security_mirror="http://security.debian.org/debian-security"
components=main
: "${DEFAULT_DIST:=unstable}"
: "${ONLY_DEFAULT_DIST:=no}"
: "${ONLY_HOSTARCH:=no}"
: "${HAVE_QEMU:=yes}"
: "${RUN_MA_SAME_TESTS:=yes}"
: "${HAVE_PROOT:=yes}"
# by default, use the mmdebstrap executable in the current directory
: "${CMD:=./mmdebstrap}"
: "${USE_HOST_APT_CONFIG:=no}"
: "${FORCE_UPDATE:=no}"
if [ "$FORCE_UPDATE" != "yes" ] && [ -e "$oldmirrordir/dists/$DEFAULT_DIST/InRelease" ]; then
http_code=$(curl --output /dev/null --silent --location --head --time-cond "$oldmirrordir/dists/$DEFAULT_DIST/InRelease" --write-out '%{http_code}' "$mirror/dists/$DEFAULT_DIST/InRelease")
case "$http_code" in
200) ;; # need update
304)
echo up-to-date
exit 0
;;
*)
echo "unexpected status: $http_code"
exit 1
;;
esac
if [ -e "$oldmirrordir/dists/$DEFAULT_DIST/Release" ]; then
http_code=$(curl --output /dev/null --silent --location --head --time-cond "$oldmirrordir/dists/$DEFAULT_DIST/Release" --write-out '%{http_code}' "$mirror/dists/$DEFAULT_DIST/Release")
case "$http_code" in
200) ;; # need update
304) echo up-to-date; exit 0;;
*) echo "unexpected status: $http_code"; exit 1;;
esac
fi
./caching_proxy.py "$oldcachedir" "$newcachedir" &
PROXYPID=$!
trap 'kill "$PROXYPID" || :' EXIT INT TERM
for i in $(seq 10); do
check_proxy_running && break
sleep 1
done
if [ ! -s "$newmirrordir/dists/$DEFAULT_DIST/InRelease" ]; then
echo "failed to start proxy" >&2
kill $PROXYPID
exit 1
fi
trap 'kill "$PROXYPID" || :;cleanup_newcachedir' EXIT INT TERM
trap "cleanup_newcachedir" EXIT INT TERM
mkdir -p "$newcachedir"
touch "$newcachedir/mmdebstrapcache"
HOSTARCH=$(dpkg --print-architecture)
arches="$HOSTARCH"
if [ "$HOSTARCH" = amd64 ]; then
arches="$arches arm64 i386"
elif [ "$HOSTARCH" = arm64 ]; then
arches="$arches amd64 armhf"
arches="amd64 arm64 i386"
else
arches="$HOSTARCH"
fi
# we need the split_inline_sig() function
# shellcheck disable=SC1091
. /usr/share/debootstrap/functions
for dist in oldstable stable testing unstable; do
for nativearch in $arches; do
# non-host architectures are only downloaded for $DEFAULT_DIST
if [ "$nativearch" != "$HOSTARCH" ] && [ "$DEFAULT_DIST" != "$dist" ]; then
continue
fi
# if ONLY_DEFAULT_DIST is set, only download DEFAULT_DIST
if [ "$ONLY_DEFAULT_DIST" = "yes" ] && [ "$DEFAULT_DIST" != "$dist" ]; then
continue
fi
if [ "$ONLY_HOSTARCH" = "yes" ] && [ "$nativearch" != "$HOSTARCH" ]; then
continue
fi
# we need a first pass without updates and security patches
# because otherwise, old package versions needed by
# debootstrap will not get included
echo "deb [arch=$nativearch] $mirror $dist $components" | update_cache "$dist" "$nativearch"
# we need to include the base mirror again or otherwise
# packages like build-essential will be missing
case "$dist" in oldstable | stable)
cat <<END | update_cache "$dist" "$nativearch"
for nativearch in $arches; do
for dist in stable testing unstable; do
# non-host architectures are only downloaded for $DEFAULT_DIST
if [ $nativearch != $HOSTARCH ] && [ $DEFAULT_DIST != $dist ]; then
continue
fi
cat << END | update_cache "$dist" "$nativearch"
deb [arch=$nativearch] $mirror $dist $components
deb [arch=$nativearch] $mirror $dist-updates main
deb [arch=$nativearch] $security_mirror $dist-security main
END
;;
esac
done
codename=$(awk '/^Codename: / { print $2; }' <"$newmirrordir/dists/$dist/InRelease")
ln -s "$dist" "$newmirrordir/dists/$codename"
# split the InRelease file into Release and Release.gpg not because apt
# or debootstrap need it that way but because grep-dctrl does
split_inline_sig \
"$newmirrordir/dists/$dist/InRelease" \
"$newmirrordir/dists/$dist/Release" \
"$newmirrordir/dists/$dist/Release.gpg"
touch --reference="$newmirrordir/dists/$dist/InRelease" "$newmirrordir/dists/$dist/Release" "$newmirrordir/dists/$dist/Release.gpg"
if [ "$dist" = "stable" ]; then
# starting wit bullseye, stable/updates becomes stable-security
cat << END | update_cache "$dist" "$nativearch"
deb [arch=$nativearch] $mirror $dist $components
deb [arch=$nativearch] $mirror stable-updates main
deb [arch=$nativearch] $security_mirror stable/updates main
END
fi
done
done
kill $PROXYPID
# Create some symlinks so that we can trick apt into accepting multiple apt
# lines that point to the same repository but look different. This is to
# avoid the warning:
# W: Target Packages (main/binary-all/Packages) is configured multiple times...
for i in $(seq 1 6); do
ln -s debian "$newcachedir/debian$i"
ln -s debian "$newcachedir/debian$i"
done
tmpdir=""
cleanuptmpdir() {
if [ -z "$tmpdir" ]; then
return
fi
if [ ! -e "$tmpdir" ]; then
return
fi
for f in "$tmpdir/worker.sh" "$tmpdir/mmdebstrap.service"; do
if [ ! -e "$f" ]; then
echo "does not exist: $f" >&2
continue
fi
rm "$f"
done
rmdir "$tmpdir"
if [ -z "$tmpdir" ]; then
return
fi
if [ ! -e "$tmpdir" ]; then
return
fi
for f in "$tmpdir/extlinux.conf" \
"$tmpdir/worker.sh" \
"$tmpdir/mini-httpd" "$tmpdir/hosts" \
"$tmpdir/debian-chroot.tar" \
"$tmpdir/mmdebstrap.service" \
"$tmpdir/debian-$DEFAULT_DIST.img"; do
if [ ! -e "$f" ]; then
echo "does not exist: $f" >&2
continue
fi
rm "$f"
done
rmdir "$tmpdir"
}
SOURCE_DATE_EPOCH="$(date --date="$(grep-dctrl -s Date -n '' "$newmirrordir/dists/$DEFAULT_DIST/Release")" +%s)"
export SOURCE_DATE_EPOCH
export SOURCE_DATE_EPOCH=$(date --date="$(grep-dctrl -s Date -n '' "$newmirrordir/dists/$DEFAULT_DIST/Release")" +%s)
if [ "$HAVE_QEMU" = "yes" ]; then
# we use the caching proxy again when building the qemu image
# - we can re-use the packages that were already downloaded earlier
# - we make sure that the qemu image uses the same Release file even
# if a mirror push happened between now and earlier
# - we avoid polluting the mirror with the additional packages by
# using --readonly
./caching_proxy.py --readonly "$oldcachedir" "$newcachedir" &
PROXYPID=$!
# We must not use any --dpkgopt here because any dpkg options still
# leak into the chroot with chrootless mode.
# We do not use our own package cache here because
# - it doesn't (and shouldn't) contain the extra packages
# - it doesn't matter if the base system is from a different mirror timestamp
# procps is needed for /sbin/sysctl
tmpdir="$(mktemp -d)"
trap "cleanuptmpdir; cleanup_newcachedir" EXIT INT TERM
for i in $(seq 10); do
check_proxy_running && break
sleep 1
done
if [ ! -s "$newmirrordir/dists/$DEFAULT_DIST/InRelease" ]; then
echo "failed to start proxy" >&2
kill $PROXYPID
exit 1
fi
pkgs=perl-doc,systemd-sysv,perl,arch-test,fakechroot,fakeroot,mount,uidmap,qemu-user-static,binfmt-support,qemu-user,dpkg-dev,mini-httpd,libdevel-cover-perl,libtemplate-perl,debootstrap,procps,apt-cudf,aspcud,python3,libcap2-bin,gpg,debootstrap,distro-info-data,iproute2,ubuntu-keyring,apt-utils
if [ "$DEFAULT_DIST" != "stable" ]; then
pkgs="$pkgs,squashfs-tools-ng,genext2fs"
fi
if [ "$HAVE_PROOT" = "yes" ]; then
pkgs="$pkgs,proot"
fi
if [ ! -e ./mmdebstrap ]; then
pkgs="$pkgs,mmdebstrap"
fi
case "$HOSTARCH" in
amd64|arm64)
pkgs="$pkgs,linux-image-$HOSTARCH"
;;
i386)
pkgs="$pkgs,linux-image-686"
;;
ppc64el)
pkgs="$pkgs,linux-image-powerpc64le"
;;
*)
echo "no kernel image for $HOSTARCH" >&2
exit 1
;;
esac
if [ "$HOSTARCH" = amd64 ] && [ "$RUN_MA_SAME_TESTS" = "yes" ]; then
arches=amd64,arm64
pkgs="$pkgs,libfakechroot:arm64,libfakeroot:arm64"
else
arches=$HOSTARCH
fi
$CMD --variant=apt --architectures=$arches --include="$pkgs" \
$DEFAULT_DIST - "$mirror" > "$tmpdir/debian-chroot.tar"
tmpdir="$(mktemp -d)"
trap 'kill "$PROXYPID" || :;cleanuptmpdir; cleanup_newcachedir' EXIT INT TERM
cat << END > "$tmpdir/extlinux.conf"
default linux
timeout 0
pkgs=perl-doc,systemd-sysv,perl,arch-test,fakechroot,fakeroot,mount,uidmap,qemu-user-binfmt,dpkg-dev,mini-httpd,libdevel-cover-perl,libtemplate-perl,debootstrap,procps,apt-cudf,aspcud,python3,libcap2-bin,gpg,debootstrap,distro-info-data,iproute2,ubuntu-keyring,apt-utils,squashfs-tools-ng,genext2fs,linux-image-generic,passwd,e2fsprogs,uuid-runtime
if [ ! -e ./mmdebstrap ]; then
pkgs="$pkgs,mmdebstrap"
fi
arches=$HOSTARCH
if [ "$RUN_MA_SAME_TESTS" = "yes" ]; then
case "$HOSTARCH" in
amd64)
arches=amd64,arm64
pkgs="$pkgs,libfakechroot:arm64,libfakeroot:arm64"
;;
arm64)
arches=arm64,amd64
pkgs="$pkgs,libfakechroot:amd64,libfakeroot:amd64"
;;
esac
fi
cat <<END >"$tmpdir/mmdebstrap.service"
label linux
kernel /vmlinuz
append initrd=/initrd.img root=/dev/vda1 rw console=ttyS0,115200
serial 0 115200
END
cat << END > "$tmpdir/mmdebstrap.service"
[Unit]
Description=mmdebstrap worker script
@ -488,23 +480,21 @@ ExecStart=/worker.sh
[Install]
WantedBy=multi-user.target
END
# here is something crazy:
# as we run mmdebstrap, the process ends up being run by different users with
# different privileges (real or fake). But for being able to collect
# Devel::Cover data, they must all share a single directory. The only way that
# I found to make this work is to mount the database directory with a
# filesystem that doesn't support ownership information at all and a umask that
# gives read/write access to everybody.
# https://github.com/pjcj/Devel--Cover/issues/223
cat <<'END' >"$tmpdir/worker.sh"
# here is something crazy:
# as we run mmdebstrap, the process ends up being run by different users with
# different privileges (real or fake). But for being able to collect
# Devel::Cover data, they must all share a single directory. The only way that
# I found to make this work is to mount the database directory with a
# filesystem that doesn't support ownership information at all and a umask that
# gives read/write access to everybody.
# https://github.com/pjcj/Devel--Cover/issues/223
cat << 'END' > "$tmpdir/worker.sh"
#!/bin/sh
echo 'root:root' | chpasswd
mount -t 9p -o trans=virtio,access=any,msize=128k mmdebstrap /mnt
mount -t 9p -o trans=virtio,access=any mmdebstrap /mnt
# need to restart mini-httpd because we mounted different content into www-root
systemctl restart mini-httpd
ip link set enp0s1 down || :
handler () {
while IFS= read -r line || [ -n "$line" ]; do
printf "%s %s: %s\n" "$(date -u -d "0 $(date +%s.%3N) seconds - $2 seconds" +"%T.%3N")" "$1" "$line"
@ -527,46 +517,103 @@ handler () {
} 2>&1;
} | { read xs; exit $xs; };
} 3>&1 || ret=$?
echo $ret > /mnt/exitstatus.txt
if [ -e cover_db.img ]; then
df -h cover_db
umount cover_db
fi
) > /mnt/output.txt 2>&1
echo $ret
) > /mnt/result.txt 2>&1
umount /mnt
systemctl poweroff
END
chmod +x "$tmpdir/worker.sh"
if [ -z ${DISK_SIZE+x} ]; then
DISK_SIZE=10G
fi
# set PATH to pick up the correct mmdebstrap variant
env PATH="$(dirname "$(realpath --canonicalize-existing "$CMD")"):$PATH" \
debvm-create --skip=usrmerge,systemdnetwork \
--size="$DISK_SIZE" --release="$DEFAULT_DIST" \
--output="$newcachedir/debian-$DEFAULT_DIST.ext4" -- \
--architectures="$arches" --include="$pkgs" \
--setup-hook='echo "Acquire::http::Proxy \"http://127.0.0.1:8080/\";" > "$1/etc/apt/apt.conf.d/00proxy"' \
--hook-dir=/usr/share/mmdebstrap/hooks/maybe-merged-usr \
--customize-hook='rm "$1/etc/apt/apt.conf.d/00proxy"' \
--customize-hook='mkdir -p "$1/etc/systemd/system/multi-user.target.wants"' \
--customize-hook='ln -s ../mmdebstrap.service "$1/etc/systemd/system/multi-user.target.wants/mmdebstrap.service"' \
--customize-hook='touch "$1/mmdebstrap-testenv"' \
--customize-hook='copy-in "'"$tmpdir"'/mmdebstrap.service" /etc/systemd/system/' \
--customize-hook='copy-in "'"$tmpdir"'/worker.sh" /' \
--customize-hook='echo 127.0.0.1 localhost > "$1/etc/hosts"' \
--customize-hook='printf "START=1\nDAEMON_OPTS=\"-h 127.0.0.1 -p 80 -u nobody -dd /mnt/cache -i /var/run/mini-httpd.pid -T UTF-8\"\n" > "$1/etc/default/mini-httpd"' \
--customize-hook='touch "$1/etc/systemd/system/tmp.mount"' \
"$mirror"
chmod +x "$tmpdir/worker.sh"
# initially we serve from the new cache so that debootstrap can grab
# the new package repository and not the old
cat << END > "$tmpdir/mini-httpd"
START=1
DAEMON_OPTS="-h 127.0.0.1 -p 80 -u nobody -dd /mnt/$newcache -i /var/run/mini-httpd.pid -T UTF-8"
END
cat << 'END' > "$tmpdir/hosts"
127.0.0.1 localhost
END
#libguestfs-test-tool
#export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
#
# In case the rootfs was prepared in fakechroot mode, ldconfig has to
# run to populate /etc/ld.so.cache or otherwise fakechroot tests will
# fail to run.
#
# The disk size is sufficient in most cases. Sometimes, gcc will do
# an upload with unstripped executables to make tracking down ICEs much
# easier (see #872672, #894014). During times with unstripped gcc, the
# buildd variant will not be 400MB but 1.3GB large and needs a 10G
# disk.
if [ -z ${DISK_SIZE+x} ]; then
DISK_SIZE=3G
fi
guestfish -N "$tmpdir/debian-$DEFAULT_DIST.img"=disk:$DISK_SIZE -- \
part-disk /dev/sda mbr : \
mkfs ext2 /dev/sda1 : \
mount /dev/sda1 / : \
tar-in "$tmpdir/debian-chroot.tar" / : \
command /sbin/ldconfig : \
copy-in "$tmpdir/extlinux.conf" / : \
mkdir-p /etc/systemd/system/multi-user.target.wants : \
ln-s ../mmdebstrap.service /etc/systemd/system/multi-user.target.wants/mmdebstrap.service : \
copy-in "$tmpdir/mmdebstrap.service" /etc/systemd/system/ : \
copy-in "$tmpdir/worker.sh" / : \
copy-in "$tmpdir/mini-httpd" /etc/default : \
copy-in "$tmpdir/hosts" /etc/ : \
touch /mmdebstrap-testenv : \
upload /usr/lib/SYSLINUX/mbr.bin /mbr.bin : \
copy-file-to-device /mbr.bin /dev/sda size:440 : \
rm /mbr.bin : \
extlinux / : \
sync : \
umount / : \
part-set-bootable /dev/sda 1 true : \
shutdown
qemu-img convert -O qcow2 "$tmpdir/debian-$DEFAULT_DIST.img" "$newcachedir/debian-$DEFAULT_DIST.qcow"
cleanuptmpdir
trap "cleanup_newcachedir" EXIT INT TERM
fi
kill $PROXYPID
cleanuptmpdir
trap "cleanup_newcachedir" EXIT INT TERM
mirror="http://127.0.0.1/debian"
for dist in stable testing unstable; do
for variant in minbase buildd -; do
echo "running debootstrap --no-merged-usr --variant=$variant $dist \${TEMPDIR} $mirror"
cat << END > shared/test.sh
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
export SOURCE_DATE_EPOCH=$SOURCE_DATE_EPOCH
tmpdir="\$(mktemp -d)"
chmod 755 "\$tmpdir"
debootstrap --no-merged-usr --variant=$variant $dist "\$tmpdir" $mirror
tar --sort=name --mtime=@$SOURCE_DATE_EPOCH --clamp-mtime --numeric-owner --one-file-system --xattrs -C "\$tmpdir" -c . > "$newcache/debian-$dist-$variant.tar"
rm -r "\$tmpdir"
END
if [ "$HAVE_QEMU" = "yes" ]; then
cachedir=$newcachedir ./run_qemu.sh
else
./run_null.sh SUDO
fi
done
done
if [ "$HAVE_QEMU" = "yes" ]; then
# now replace the minihttpd config with one that serves the new repository
guestfish -a "$newcachedir/debian-$DEFAULT_DIST.qcow" -i <<EOF
upload -<<END /etc/default/mini-httpd
START=1
DAEMON_OPTS="-h 127.0.0.1 -p 80 -u nobody -dd /mnt/cache -i /var/run/mini-httpd.pid -T UTF-8"
END
EOF
fi
# delete possibly leftover symlink
if [ -e ./shared/cache.tmp ]; then
rm ./shared/cache.tmp
rm ./shared/cache.tmp
fi
# now atomically switch the symlink to point to the other directory
ln -s $newcache ./shared/cache.tmp
@ -575,5 +622,3 @@ mv --no-target-directory ./shared/cache.tmp ./shared/cache
deletecache "$oldcachedir"
trap - EXIT INT TERM
echo "$0 finished successfully" >&2

6237
mmdebstrap
View file

File diff suppressed because it is too large Load diff

460
mmdebstrap-autopkgtest-build-qemu
View file

@ -1,460 +0,0 @@
#!/bin/sh
# Copyright 2023 Johannes Schauer Marin Rodrigues <josch@debian.org>
# Copyright 2023 Helmut Grohne <helmut@subdivi.de>
# SPDX-License-Identifier: MIT
# We generally use single quotes to avoid variable expansion:
# shellcheck disable=SC2016
# Replacement for autopkgtest-build-qemu and vmdb2 for all architectures
# supporting EFI booting (amd64, arm64, armhf, i386, riscv64).
# For use as replacement for autopkgtest-build-qemu and vmdb2 on ppc64el which
# neither supports extlinux nor efi booting there is an unmaintained script
# which uses grub instead to boot:
#
# https://gitlab.mister-muffin.de/josch/mmdebstrap/src/commit/
# e523741610a4ed8579642bfc755956f64c847ef3/mmdebstrap-autopkgtest-build-qemu
: <<'POD2MAN'
=head1 NAME
mmdebstrap-autopkgtest-build-qemu - autopkgtest-build-qemu without vmdb2 but mmdebstrap and EFI boot
=head1 SYNOPSIS
B<mmdebstrap-autopkgtest-build-qemu> [I<OPTIONS>] B<--boot>=B<efi> I<RELEASE> I<IMAGE>
=head1 DESCRIPTION
B<mmdebstrap-autopkgtest-build-qemu> is a mostly compatible drop-in replacement
for L<autopkgtest-build-qemu(1)> with two main differences: Firstly, it uses
L<mmdebstrap(1)> instead of L<vmdb2(1)> and thus is able to create QEMU disk
images without requiring superuser privileges and with bit-by-bit reproducible
output. Secondly, it uses L<systemd-boot(7)> and thus only supports booting via
EFI. For architectures for which L<autopkgtest-virt-qemu(1)> does not default
to EFI booting you must pass B<--boot=efi> when invoking the autopkgtest virt
backend.
=head1 POSITIONAL PARAMETERS
=over 8
=item I<RELEASE>
The release to download from the I<MIRROR>. This parameter is required.
=item I<IMAGE>
The file to write, in raw format. This parameter is required.
=back
=head1 OPTIONS
=over 8
=item B<--mirror>=I<MIRROR>
Specify which distribution to install. It defaults to
http://deb.debian.org/debian (i.e. Debian), but you can pass a mirror of any
Debian derivative.
=item B<--architecture>=I<ARCHITECTURE>
Set the architecture for the virtual machine image, specified as a L<dpkg(1)>
architecture. If omitted, the host architecture is assumed.
B<--arch>=I<ARCH> is an alias for this option.
=item B<--script>=I<SCRIPT>
Specifies a user script that will be called with the root filesystem of the
image as its first parameter. This script can them make any necesssary
modifications to the root filesystem.
The script must be a POSIX shell script, and should not depend on bash-specific
features. This script will be executed inside a L<chroot(1)> call in the
virtual machine root filesystem.
=item B<--size>=I<SIZE>
Specifies the image size for the virtual machine, defaulting to 25G.
=item B<--apt-proxy>=I<PROXY>
Specify an apt proxy to use in the virtual machine. By default, if you have
an apt proxy configured on the host, the virtual machine will automatically use
this, otherwise there is no default.
=item B<--boot>=B<efi>, B<--efi>
Select the way the generated image will expect to be booted. Unless you
explicitly select --boot=efi, operation will fail.
=item B<--keyring>=I<KEYRING>
Passes an additional B<--keyring> parameter to B<mmdebstrap>.
=back
=head1 EXAMPLES
Make sure, that F</path/to/debian-unstable.img> is a path that the unshared
user has access to. This can be done by ensuring world-execute permissions on
all path components or by creating the image in a world-readable directory like
/tmp before copying it into its final location.
$ mmdebstrap-autopkgtest-build-qemu --boot=efi --arch=amd64 unstable /path/to/debian-unstable.img
[...]
$ autopkgtest mypackage -- qemu --boot=efi --dpkg-architecture=amd64 /path/to/debian-unstable.img
Make sure to add B<--boot=efi> to both the B<mmdebstrap-autopkgtest-build-qemu>
as well as the B<autopkgtest-virt-qemu> invocation.
Create bit-by-bit reproducible images from a given snapshot.d.o timestamp.
SOURCE_DATE_EPOCH=1612543740 mmdebstrap-autopkgtest-build-qemu --boot=efi \
--mirror=http://snapshot.debian.org/archive/debian/20210205T164900Z/ \
unstable /path/to/debian-unstable.img
=head1 SEE ALSO
L<autopkgtest-build-qemu(1)>, L<autopkgtest-virt-qemu(1)>, L<mmdebstrap(1)>, L<autopkgtest(1)>
=cut
POD2MAN
set -eu
die() {
echo "$*" 1>&2
exit 1
}
usage() {
die "usage: $0 [--architecture=|--apt-proxy=|--keyring=|--mirror=|--script=|--size=] --boot=efi <RELEASE> <IMAGE>"
}
usage_error() {
echo "error: $*" 1>&2
usage
}
BOOT=auto
ARCHITECTURE=$(dpkg --print-architecture)
IMAGE=
MIRROR=
KEYRING=
RELEASE=
SIZE=25G
SCRIPT=
# consumed by setup-testbed
export AUTOPKGTEST_BUILD_QEMU=1
opt_boot() {
BOOT="$1"
}
opt_architecture() {
ARCHITECTURE="$1"
}
opt_arch() {
ARCHITECTURE="$1"
}
opt_apt_proxy() {
# consumed by setup-testbed
export AUTOPKGTEST_APT_PROXY="$1"
# consumed by mmdebstrap
if test "$1" = DIRECT; then
unset http_proxy
else
export http_proxy="$1"
fi
}
opt_keyring() {
KEYRING="$1"
}
opt_mirror() {
# consumed by setup-testbed
export MIRROR="$1"
}
opt_script() {
test -f "$1" || die "passed script '$1' does not refer to a file"
SCRIPT="$1"
}
opt_size() {
SIZE="$1"
}
positional=1
positional_1() {
# consumed by setup-testbed
export RELEASE="$1"
}
positional_2() {
IMAGE="$1"
}
positional_3() { opt_mirror "$@"; }
positional_4() { opt_architecture "$@"; }
positional_5() { opt_script "$@"; }
positional_6() { opt_size "$@"; }
positional_7() {
die "too many positional options"
}
while test "$#" -gt 0; do
case "$1" in
--architecture=* | --arch=* | --boot=* | --keyring=* | --mirror=* | --script=* | --size=*)
optname="${1%%=*}"
"opt_${optname#--}" "${1#*=}"
;;
--apt-proxy=*)
opt_apt_proxy "${1#*=}"
;;
--architecture | --arch | --boot | --keyring | --mirror | --script | --size)
test "$#" -ge 2 || usage_error "missing argument for $1"
"opt_${1#--}" "$2"
shift
;;
--apt-proxy)
test "$#" -ge 2 || usage_error "missing argument for $1"
opt_apt_proxy "$2"
shift
;;
--efi)
opt_boot efi
;;
--*)
usage_error "unrecognized argument $1"
;;
*)
"positional_$positional" "$1"
positional=$((positional + 1))
;;
esac
shift
done
test -z "$RELEASE" -o -z "$IMAGE" && usage_error "missing positional arguments"
test "$BOOT" = efi \
|| die "this tool does not support boot modes other than efi"
case "$ARCHITECTURE" in
amd64)
EFIIMG=bootx64.efi
QEMUARCH=x86_64
VMFPKG=ovmf
LINUXIMAGE=linux-image-amd64
;;
arm64)
EFIIMG=bootaa64.efi
QEMUARCH=aarch64
VMFPKG=qemu-efi-aarch64
LINUXIMAGE=linux-image-arm64
;;
armhf)
EFIIMG=bootarm.efi
QEMUARCH=arm
VMFPKG=qemu-efi-arm
LINUXIMAGE=linux-image-armmp
;;
i386)
EFIIMG=bootia32.efi
QEMUARCH=i386
VMFPKG=ovmf-ia32
LINUXIMAGE=linux-image-686-pae
;;
riscv64)
EFIIMG=bootriscv64.efi
QEMUARCH=riscv64
VMFPKG=
LINUXIMAGE=linux-image-riscv64
;;
*)
die "unsupported architecture: $ARCHITECTURE"
;;
esac
if test "$(dpkg-query -f '${db:Status-Status}' -W binutils-multiarch)" = installed; then
GNU_PREFIX=
BINUTILS=
else
GNU_ARCHITECTURE="$(dpkg-architecture "-a$ARCHITECTURE" -qDEB_HOST_GNU_TYPE)"
GNU_PREFIX="$GNU_ARCHITECTURE-"
GNU_SUFFIX="-$(echo "$GNU_ARCHITECTURE" | tr _ -)"
BINUTILS=", binutils$GNU_SUFFIX | binutils-multiarch"
fi
arches=" $(dpkg --print-architecture) $(dpkg --print-foreign-architectures | tr '\n' ' ') "
case $arches in
*" $ARCHITECTURE "*) : ;; # nothing to do
*) die "enable $ARCHITECTURE by running: sudo dpkg --add-architecture $ARCHITECTURE && sudo apt update" ;;
esac
test "$(dpkg-query -f '${db:Status-Status}' -W "dpkg-dev")" = installed \
|| die "please install dpkg-dev"
dpkg-checkbuilddeps -d "autopkgtest, dosfstools, e2fsprogs, fdisk, mount, mtools, passwd, uidmap, libarchive13, systemd-boot-efi:$ARCHITECTURE $BINUTILS" /dev/null \
|| die "please install the required packages listed above"
BOOTSTUB="/usr/lib/systemd/boot/efi/linux${EFIIMG#boot}.stub"
WORKDIR=
cleanup() {
test -n "$WORKDIR" && rm -Rf "$WORKDIR"
}
trap cleanup EXIT INT TERM QUIT
WORKDIR=$(mktemp -d)
FAT_OFFSET_SECTORS=$((1024 * 2))
FAT_SIZE_SECTORS=$((1024 * 254))
# The image is raw and not in qcow2 format because:
# - faster run-time as the "qemu-image convert" step is not needed
# - image can be used independent of qemu tooling
# - modifying the image just with "mount" instead of requiring qemu-nbd
# - sparse images make the file just as small as with qcow2
# - trim support is more difficult on qcow2
# - snapshots and overlays work just as well with raw images
# - users who prefer qcow2 get to choose to run it themselves with their own
# custom options like compression
set -- \
--mode=unshare \
--format=tar \
--variant=important \
--architecture="$ARCHITECTURE"
case $MIRROR in http://snapshot.debian.org/archive/* | https://snapshot.debian.org/archive/*)
set -- "$@" --aptopt='Acquire::Check-Valid-Until "false"'
;;
esac
EXT_FEATURES=
if test "$RELEASE" = jessie; then
set -- "$@" --keyring=/usr/share/keyrings/debian-archive-removed-keys.gpg
set -- "$@" --aptopt='Apt::Key::gpgvcommand "/usr/libexec/mmdebstrap/gpgvnoexpkeysig"'
set -- "$@" --hook-dir=/usr/share/mmdebstrap/hooks/jessie-or-older
EXT_FEATURES="^metadata_csum,^metadata_csum_seed,^orphan_file"
fi
set -- "$@" \
"--include=init,$LINUXIMAGE,python3" \
'--customize-hook=echo host >"$1/etc/hostname"' \
'--customize-hook=echo 127.0.0.1 localhost host >"$1/etc/hosts"' \
'--customize-hook=passwd --root "$1" --delete root' \
'--customize-hook=useradd --root "$1" --home-dir /home/user --create-home user' \
'--customize-hook=passwd --root "$1" --delete user' \
'--customize-hook=/usr/share/autopkgtest/setup-commands/setup-testbed'
if test -n "$SCRIPT"; then
set -- "$@" \
"--customize-hook=upload '$SCRIPT' /userscript" \
"--chrooted-customize-hook=sh /userscript" \
'--customize-hook=rm -f "$1/userscript"'
fi
set -- "$@" \
"--customize-hook=download vmlinuz '$WORKDIR/kernel'" \
"--customize-hook=download initrd.img '$WORKDIR/initrd'" \
"$RELEASE" \
-
test -n "$MIRROR" && set -- "$@" "$MIRROR"
test -n "$KEYRING" && set -- "$@" "--keyring=$KEYRING"
echo "+ mmdebstrap $*" >&2
# https://github.com/koalaman/shellcheck/issues/2555
# shellcheck disable=SC3040
set -o pipefail
mmdebstrap "$@" | {
set -- -t ext4 -L autopkgtestvm -d -
if test -n "$EXT_FEATURES"; then
set -- "$@" -O "$EXT_FEATURES"
fi
EXTOPTS="offset=$(((FAT_OFFSET_SECTORS + FAT_SIZE_SECTORS) * 512))"
if test -n "${SOURCE_DATE_EPOCH-}"; then
uuid="$(uuidgen --sha1 --namespace="$(uuidgen --sha1 --namespace='@dns' --name mister-muffin.de)" --name "$SOURCE_DATE_EPOCH")"
set -- "$@" -U "$uuid"
EXTOPTS="$EXTOPTS,hash_seed=$uuid"
fi
set -- "$@" -E "$EXTOPTS" "$IMAGE" "$SIZE"
echo "+ mke2fs $*" >&2
/sbin/mke2fs "$@"
}
echo "root=LABEL=autopkgtestvm rw console=ttyS0" >"$WORKDIR/cmdline"
align_size() {
echo "$((($1) + ($2) - 1 - (($1) + ($2) - 1) % ($2)))"
}
alignment=$("${GNU_PREFIX}objdump" -p "$BOOTSTUB" | sed 's/^SectionAlignment\s\+\([0-9]\)/0x/;t;d')
test -z "$alignment" && die "failed to discover the alignment of the efi stub"
echo "determined efi vma alignment as $alignment"
test "$RELEASE" = jessie -a "$((alignment))" -lt "$((1024 * 1024))" && {
echo "increasing efi vma alignment for jessie"
alignment=$((1024 * 1024))
}
lastoffset=0
# shellcheck disable=SC2034 # unused variables serve documentation
lastoffset="$("${GNU_PREFIX}objdump" -h "$BOOTSTUB" \
| while read -r idx name size vma lma fileoff algn behind; do
test -z "$behind" -a "${algn#"2**"}" != "$algn" || continue
offset=$((0x$vma + 0x$size))
test "$offset" -gt "$lastoffset" || continue
lastoffset="$offset"
echo "$lastoffset"
done | tail -n1)"
lastoffset=$(align_size "$lastoffset" "$alignment")
echo "determined minimum efi vma offset as $lastoffset"
cmdline_size="$(stat -Lc%s "$WORKDIR/cmdline")"
cmdline_size="$(align_size "$cmdline_size" "$alignment")"
linux_size="$(stat -Lc%s "$WORKDIR/kernel")"
linux_size="$(align_size "$linux_size" "$alignment")"
cmdline_offset="$lastoffset"
linux_offset=$((cmdline_offset + cmdline_size))
initrd_offset=$((linux_offset + linux_size))
SOURCE_DATE_EPOCH=0 \
"${GNU_PREFIX}objcopy" \
--enable-deterministic-archives \
--add-section .cmdline="$WORKDIR/cmdline" \
--change-section-vma .cmdline="$(printf 0x%x "$cmdline_offset")" \
--add-section .linux="$WORKDIR/kernel" \
--change-section-vma .linux="$(printf 0x%x "$linux_offset")" \
--add-section .initrd="$WORKDIR/initrd" \
--change-section-vma .initrd="$(printf 0x%x "$initrd_offset")" \
"$BOOTSTUB" "$WORKDIR/efiimg"
rm -f "$WORKDIR/kernel" "$WORKDIR/initrd"
truncate -s "$((FAT_SIZE_SECTORS * 512))" "$WORKDIR/fat"
/sbin/mkfs.fat -F 32 --invariant "$WORKDIR/fat"
mmd -i "$WORKDIR/fat" EFI EFI/BOOT
mcopy -i "$WORKDIR/fat" "$WORKDIR/efiimg" "::EFI/BOOT/$EFIIMG"
rm -f "$WORKDIR/efiimg"
truncate --size="+$((34 * 512))" "$IMAGE"
/sbin/sfdisk "$IMAGE" <<EOF
label: gpt
unit: sectors
start=$FAT_OFFSET_SECTORS, size=$FAT_SIZE_SECTORS, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B
start=$((FAT_OFFSET_SECTORS + FAT_SIZE_SECTORS)), type=0FC63DAF-8483-4772-8E79-3D69D8477DE4
EOF
dd if="$WORKDIR/fat" of="$IMAGE" conv=notrunc,sparse bs=512 "seek=$FAT_OFFSET_SECTORS" status=none
if test "$(dpkg --print-architecture)" != "$ARCHITECTURE" && test "$(dpkg-query -f '${db:Status-Status}' -W "qemu-system-$QEMUARCH")" != installed; then
echo "I: you might need to install a package providing qemu-system-$QEMUARCH to use this image with autopkgtest-virt-qemu" >&2
fi
if test -n "$VMFPKG" && test "$(dpkg-query -f '${db:Status-Status}' -W "$VMFPKG")" != installed; then
echo "I: you might need to install $VMFPKG to use this image with autopkgtest-virt-qemu" >&2
fi
echo "I: SUCCESS! Your new image can be found here: $IMAGE" >&2
echo "I: Don't forget to pass --boot=efi when running autopkgtest-virt-qemu with this image" >&2

6
proxysolver
View file

@ -1,9 +1,5 @@
#!/usr/bin/env python3
#
# This script is in the public domain
#
# Author: Johannes Schauer Marin Rodrigues <josch@mister-muffin.de>
#
# thin layer around /usr/lib/apt/solvers/apt, so that we can capture the solver
# result
#

61
run_null.sh
View file

@ -4,46 +4,27 @@ set -eu
SUDO=
while [ "$#" -gt 0 ]; do
key="$1"
case "$key" in
SUDO)
SUDO=sudo
;;
*)
echo "Unknown argument: $key"
exit 1
;;
esac
shift
key="$1"
case "$key" in
SUDO)
SUDO=sudo
;;
*)
echo "Unknown argument: $key"
exit 1
;;
esac
shift
done
# - Run command with fds 3 and 4 closed so that whatever test.sh does it
# cannot interfere with these.
# - Both stdin and stderr of test.sh are written to stdout
# - Write exit status of test.sh to fd 3
# - Write stdout to shared/output.txt as well as to fd 4
# - Redirect fd 3 to stdout
# - Read fd 3 and let the group exit with that value
# - Redirect fd 4 to stdout
ret=0
{
{
{
{
ret=0
(
exec 3>&- 4>&-
env --chdir=./shared $SUDO sh -x ./test.sh 2>&1
) || ret=$?
echo $ret >&3
} | tee shared/output.txt >&4
} 3>&1
} | {
read -r xs
exit "$xs"
}
} 4>&1 || ret=$?
if [ "$ret" -ne 0 ]; then
echo "test.sh failed"
exit 1
# subshell so that we can cd without effecting the rest
(
set +e
cd ./shared;
$SUDO sh -x ./test.sh;
echo $?;
) 2>&1 | tee shared/result.txt | head --lines=-1
if [ "$(tail --lines=1 shared/result.txt)" -ne 0 ]; then
echo "test.sh failed"
exit 1
fi

88
run_qemu.sh
View file

@ -4,69 +4,45 @@ set -eu
: "${DEFAULT_DIST:=unstable}"
: "${cachedir:=./shared/cache}"
: "${MMDEBSTRAP_TESTS_DEBUG:=no}"
tmpdir="$(mktemp -d)"
cleanup() {
rv=$?
rm -f "$tmpdir/log"
[ -e "$tmpdir" ] && rmdir "$tmpdir"
if [ -e shared/output.txt ]; then
res="$(cat shared/exitstatus.txt)"
if [ "$res" != "0" ]; then
# this might possibly overwrite another non-zero rv
rv=1
fi
fi
exit $rv
rv=$?
rm -f "$tmpdir/debian-$DEFAULT_DIST-overlay.qcow"
rm -f "$tmpdir/log"
[ -e "$tmpdir" ] && rmdir "$tmpdir"
if [ -e shared/result.txt ]; then
head --lines=-1 shared/result.txt
res="$(tail --lines=1 shared/result.txt)"
rm shared/result.txt
if [ "$res" != "0" ]; then
# this might possibly overwrite another non-zero rv
rv=1
fi
fi
exit $rv
}
trap cleanup INT TERM EXIT
echo 1 >shared/exitstatus.txt
if [ -e shared/output.txt ]; then
rm shared/output.txt
fi
touch shared/output.txt
setpriv --pdeathsig TERM tail -f shared/output.txt &
set -- timeout --foreground 40m \
debvm-run --image="$(realpath "$cachedir")/debian-$DEFAULT_DIST.ext4" \
--
cpuname=$(lscpu | awk '/Model name:/ {print $3}' | tr '\n' '+')
ncpu=$(lscpu | awk '/Core\(s\) per socket:/ {print $4}' | tr '\n' '+')
if [ "$cpuname" = "Cortex-A53+Cortex-A73+" ] && [ "$ncpu" = "2+4+" ]; then
# crude detection of the big.LITTLE heterogeneous setup of cores on the
# amlogic a311d bananapi
#
# https://lists.nongnu.org/archive/html/qemu-devel/2020-10/msg08494.html
# https://gitlab.com/qemu-project/qemu/-/issues/239
# https://segments.zhan.science/posts/kvm_on_pinehone_pro/#trouble-with-heterogeneous-architecture
set -- taskset --cpu-list 2,3,4,5 "$@" -smp 4
fi
set -- "$@" -nic none -m 4G -snapshot
if [ "$MMDEBSTRAP_TESTS_DEBUG" = "no" ]; then
# to connect to serial use:
# minicom -D 'unix#/tmp/ttyS0'
# or this (quit with ctrl+q):
# socat stdin,raw,echo=0,escape=0x11 unix-connect:/tmp/ttyS0
set -- "$@" \
-monitor unix:/tmp/monitor,server,nowait \
-serial unix:/tmp/ttyS0,server,nowait \
-serial unix:/tmp/ttyS1,server,nowait
fi
set -- "$@" -virtfs local,id=mmdebstrap,path="$(pwd)/shared",security_model=none,mount_tag=mmdebstrap
# the path to debian-$DEFAULT_DIST.qcow must be absolute or otherwise qemu will
# look for the path relative to debian-$DEFAULT_DIST-overlay.qcow
qemu-img create -f qcow2 -b "$(realpath $cachedir)/debian-$DEFAULT_DIST.qcow" -F qcow2 "$tmpdir/debian-$DEFAULT_DIST-overlay.qcow"
# to connect to serial use:
# minicom -D 'unix#/tmp/ttyS0'
ret=0
if [ "$MMDEBSTRAP_TESTS_DEBUG" = "no" ]; then
"$@" >"$tmpdir/log" 2>&1 || ret=$?
else
"$@" 2>&1 | tee "$tmpdir/log" || ret=$?
fi
timeout 20m qemu-system-x86_64 \
-no-user-config \
-M accel=kvm:tcg -m 1G -nographic \
-object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0 \
-monitor unix:/tmp/monitor,server,nowait \
-serial unix:/tmp/ttyS0,server,nowait \
-serial unix:/tmp/ttyS1,server,nowait \
-net nic,model=virtio -net user \
-virtfs local,id=mmdebstrap,path="$(pwd)/shared",security_model=none,mount_tag=mmdebstrap \
-drive file="$tmpdir/debian-$DEFAULT_DIST-overlay.qcow",cache=unsafe,index=0,if=virtio \
>"$tmpdir/log" 2>&1 || ret=$?
if [ "$ret" -ne 0 ]; then
cat "$tmpdir/log"
exit $ret
cat "$tmpdir/log"
exit $ret
fi

187
tarfilter
View file

@ -2,8 +2,6 @@
#
# This script is in the public domain
#
# Author: Johannes Schauer Marin Rodrigues <josch@mister-muffin.de>
#
# This script accepts a tarball on standard input and filters it according to
# the same rules used by dpkg --path-exclude and --path-include, using command
# line options of the same name. The result is then printed on standard output.
@ -43,80 +41,17 @@ class PaxFilterAction(argparse.Action):
setattr(namespace, "paxfilter", items)
class TypeFilterAction(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
items = getattr(namespace, "typefilter", [])
match values:
case "REGTYPE" | "0":
items.append(tarfile.REGTYPE)
case "LNKTYPE" | "1":
items.append(tarfile.LNKTYPE)
case "SYMTYPE" | "2":
items.append(tarfile.SYMTYPE)
case "CHRTYPE" | "3":
items.append(tarfile.CHRTYPE)
case "BLKTYPE" | "4":
items.append(tarfile.BLKTYPE)
case "DIRTYPE" | "5":
items.append(tarfile.DIRTYPE)
case "FIFOTYPE" | "6":
items.append(tarfile.FIFOTYPE)
case _:
raise ValueError("invalid type: %s" % values)
setattr(namespace, "typefilter", items)
class TransformAction(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
items = getattr(namespace, "trans", [])
# This function mimics what src/transform.c from tar does
if not values.startswith("s"):
raise ValueError("regex must start with an 's'")
if len(values) <= 4:
# minimum regex: s/x//
raise ValueError("invalid regex (too short)")
d = values[1]
if values.startswith(f"s{d}{d}"):
raise ValueError("empty regex")
values = values.removeprefix(f"s{d}")
flags = 0
if values.endswith(f"{d}i"):
# trailing flags
flags = re.IGNORECASE
values = values.removesuffix(f"{d}i")
# This regex only finds non-empty tokens.
# Finding empty tokens would require a variable length look-behind
# or \K in order to find escaped delimiters which is not supported by
# the python re module.
tokens = re.findall(rf"(?:\\[\\{d}]|[^{d}])+", values)
match len(tokens):
case 0:
raise ValueError("invalid regex: not enough terms")
case 1:
repl = ""
case 2:
repl = tokens[1]
case _:
raise ValueError("invalid regex: too many terms: %s" % tokens)
items.append((re.compile(tokens[0], flags), repl))
setattr(namespace, "trans", items)
def main():
parser = argparse.ArgumentParser(
formatter_class=argparse.RawDescriptionHelpFormatter,
description="""\
Filters a tarball on standard input by the same rules as the dpkg --path-exclude
and --path-include options and writes resulting tarball to standard output. See
dpkg(1) for information on how these two options work in detail. To reuse the
exact same semantics as used by dpkg, paths must be given as /path and not as
./path even though they might be stored as such in the tarball.
dpkg(1) for information on how these two options work in detail.
Secondly, filter out unwanted pax extended headers using --pax-exclude and
--pax-include. This is useful in cases where a tool only accepts certain xattr
prefixes. For example tar2sqfs only supports SCHILY.xattr.user.*,
SCHILY.xattr.trusted.* and SCHILY.xattr.security.* but not
SCHILY.xattr.system.posix_acl_default.*.
Similarly, filter out unwanted pax extended headers. This is useful in cases
where a tool only accepts certain xattr prefixes. For example tar2sqfs only
supports SCHILY.xattr.user.*, SCHILY.xattr.trusted.* and
SCHILY.xattr.security.* but not SCHILY.xattr.system.posix_acl_default.*.
Both types of options use Unix shell-style wildcards:
@ -124,88 +59,34 @@ Both types of options use Unix shell-style wildcards:
? matches any single character
[seq] matches any character in seq
[!seq] matches any character not in seq
Thirdly, filter out files matching a specific tar archive member type using
--type-exclude. Valid type names are REGTYPE (regular file), LNKTYPE
(hardlink), SYMTYPE (symlink), CHRTYPE (character special), BLKTYPE (block
special), DIRTYPE (directory), FIFOTYPE (fifo) or their tar format flag value
(0-6, respectively).
Fourthly, transform the path of tar members using a sed expression just as with
GNU tar --transform.
Fifthly, strip leading directory components off of tar members. Just as with
GNU tar --strip-components, tar members that have less or equal components in
their path are not passed through.
Lastly, shift user id and group id of each entry by the value given by the
--idshift argument. The resulting uid or gid must not be negative.
""",
"""
)
parser.add_argument(
"--path-exclude",
metavar="pattern",
action=PathFilterAction,
help="Exclude path matching the given shell pattern. "
"This option can be specified multiple times.",
help="Exclude path matching the given shell pattern.",
)
parser.add_argument(
"--path-include",
metavar="pattern",
action=PathFilterAction,
help="Re-include a pattern after a previous exclusion. "
"This option can be specified multiple times.",
help="Re-include a pattern after a previous exclusion.",
)
parser.add_argument(
"--pax-exclude",
metavar="pattern",
action=PaxFilterAction,
help="Exclude pax header matching the given globbing pattern. "
"This option can be specified multiple times.",
help="Exclude pax header matching the given globbing pattern.",
)
parser.add_argument(
"--pax-include",
metavar="pattern",
action=PaxFilterAction,
help="Re-include a pax header after a previous exclusion. "
"This option can be specified multiple times.",
)
parser.add_argument(
"--type-exclude",
metavar="type",
action=TypeFilterAction,
help="Exclude certain member types by their type. Choose types either "
"by their name (REGTYPE, LNKTYPE, SYMTYPE, CHRTYPE, BLKTYPE, DIRTYPE, "
"FIFOTYPE) or by their tar format flag values (0-6, respectively). "
"This option can be specified multiple times.",
)
parser.add_argument(
"--transform",
"--xform",
metavar="EXPRESSION",
action=TransformAction,
help="Use sed replace EXPRESSION to transform file names. "
"This option can be specified multiple times.",
)
parser.add_argument(
"--strip-components",
metavar="NUMBER",
type=int,
help="Strip NUMBER leading components from file names",
)
parser.add_argument(
"--idshift",
metavar="NUM",
type=int,
help="Integer value by which to shift the uid and gid of each entry",
help="Re-include a pax header after a previous exclusion.",
)
args = parser.parse_args()
if (
not hasattr(args, "pathfilter")
and not hasattr(args, "paxfilter")
and not hasattr(args, "typefilter")
and not hasattr(args, "strip_components")
):
if not hasattr(args, "pathfilter") and not hasattr(args, "paxfilter"):
from shutil import copyfileobj
copyfileobj(sys.stdin.buffer, sys.stdout.buffer)
@ -218,22 +99,19 @@ Lastly, shift user id and group id of each entry by the value given by the
skip = False
if not hasattr(args, "pathfilter"):
return False
# normalize path and make it absolute by stripping off all leading
# dots and slashes and then prepending a slash
name = "/" + member.name.lstrip("./")
for t, r in args.pathfilter:
if r.match(name) is not None:
for (t, r) in args.pathfilter:
if r.match(member.name[1:]) is not None:
if t == "path_include":
skip = False
else:
skip = True
if skip and (member.isdir() or member.issym()):
for t, r in args.pathfilter:
for (t, r) in args.pathfilter:
if t != "path_include":
continue
prefix = prefix_prog.sub(r"\1", r.pattern)
prefix = prefix.rstrip("/")
if name.startswith(prefix):
if member.name[1:].startswith(prefix):
return False
return skip
@ -241,7 +119,7 @@ Lastly, shift user id and group id of each entry by the value given by the
if not hasattr(args, "paxfilter"):
return False
skip = False
for t, r in args.paxfilter:
for (t, r) in args.paxfilter:
if r.match(header) is None:
continue
if t == "pax_include":
@ -250,48 +128,19 @@ Lastly, shift user id and group id of each entry by the value given by the
skip = True
return skip
def type_filter_should_skip(member):
if not hasattr(args, "typefilter"):
return False
for t in args.typefilter:
if member.type == t:
return True
return False
# starting with Python 3.8, the default format became PAX_FORMAT but we
# are still explicit here in case of future changes.
# starting with Python 3.8, the default format became PAX_FORMAT, so this
# is only for compatibility with older versions of Python 3
with tarfile.open(fileobj=sys.stdin.buffer, mode="r|*") as in_tar, tarfile.open(
fileobj=sys.stdout.buffer, mode="w|", format=tarfile.PAX_FORMAT
) as out_tar:
for member in in_tar:
if path_filter_should_skip(member):
continue
if type_filter_should_skip(member):
continue
if args.strip_components:
comps = member.name.split("/")
# just as with GNU tar, archive members with less or equal
# number of components are not passed through at all
if len(comps) <= args.strip_components:
continue
member.name = "/".join(comps[args.strip_components :])
member.pax_headers = {
k: v
for k, v in member.pax_headers.items()
if not pax_filter_should_skip(k)
}
if args.idshift:
if args.idshift < 0 and -args.idshift > member.uid:
print("uid cannot be negative", file=sys.stderr)
exit(1)
if args.idshift < 0 and -args.idshift > member.gid:
print("gid cannot be negative", file=sys.stderr)
exit(1)
member.uid += args.idshift
member.gid += args.idshift
if hasattr(args, "trans"):
for r, s in args.trans:
member.name = r.sub(s, member.name)
if member.isfile():
with in_tar.extractfile(member) as file:
out_tar.addfile(member, file)

65
taridshift Executable file
View file

@ -0,0 +1,65 @@
#!/usr/bin/env python3
#
# This script is in the public domain
#
# This script accepts a tarball on standard input and prints a tarball on
# standard output with the same contents but all uid and gid ownership
# information shifted by the value given as first command line argument.
#
# A tool like this should be written in C but libarchive has issues:
# https://github.com/libarchive/libarchive/issues/587
# https://github.com/libarchive/libarchive/pull/1288/ (needs 3.4.1)
# Should these issues get fixed, then a good template is tarfilter.c in the
# examples directory of libarchive.
#
# We are not using Perl either, because Archive::Tar slurps the whole tarball
# into memory.
#
# We could also use Go but meh...
# https://stackoverflow.com/a/59542307/784669
import tarfile
import sys
import argparse
def main():
parser = argparse.ArgumentParser(
description="""\
Accepts a tarball on standard input and prints a tarball on standard output
with the same contents but all uid and gid ownership information shifted by the
value given as first command line argument.
"""
)
parser.add_argument(
"idshift",
metavar="NUM",
type=int,
help="Integer value by which to shift the uid and gid of each entry",
)
args = parser.parse_args()
# starting with Python 3.8, the default format became PAX_FORMAT, so this
# is only for compatibility with older versions of Python 3
with tarfile.open(fileobj=sys.stdin.buffer, mode="r|*") as in_tar, tarfile.open(
fileobj=sys.stdout.buffer, mode="w|", format=tarfile.PAX_FORMAT
) as out_tar:
for member in in_tar:
if args.idshift < 0 and -args.idshift > member.uid:
print("uid cannot be negative", file=sys.stderr)
exit(1)
if args.idshift < 0 and -args.idshift > member.gid:
print("gid cannot be negative", file=sys.stderr)
exit(1)
member.uid += args.idshift
member.gid += args.idshift
if member.isfile():
with in_tar.extractfile(member) as file:
out_tar.addfile(member, file)
else:
out_tar.addfile(member)
if __name__ == "__main__":
main()

8
tests/apt-patterns
View file

@ -1,8 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
trap "rm -f /tmp/debian-chroot.tar" EXIT INT TERM
{{ CMD }} --mode={{ MODE }} --variant=essential \
--include '?or(?exact-name(dummy-does-not-exist),?exact-name(apt))' \
{{ DIST }} /tmp/debian-chroot.tar {{ MIRROR }}
tar -tf /tmp/debian-chroot.tar | sort | diff -u tar1.txt -

12
tests/apt-patterns-custom
View file

@ -1,12 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
trap "rm -f /tmp/debian-chroot.tar" EXIT INT TERM
{{ CMD }} --mode={{ MODE }} --variant=custom \
--include '?narrow(?archive(^{{ DIST }}$),?essential)' \
--include apt \
{{ DIST }} /tmp/debian-chroot.tar {{ MIRROR }}
{
tar -tf /tmp/debian-chroot.tar
echo ./var/lib/apt/extended_states
} | sort | diff -u tar1.txt -

9
tests/aptopt
View file

@ -1,9 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
trap "rm -rf /tmp/debian-chroot; rm -f /tmp/config" EXIT INT TERM
echo 'Acquire::Languages "none";' >/tmp/config
{{ CMD }} --mode=root --variant=apt --aptopt='Acquire::Check-Valid-Until "false"' --aptopt=/tmp/config {{ DIST }} /tmp/debian-chroot {{ MIRROR }}
printf 'Acquire::Check-Valid-Until "false";\nAcquire::Languages "none";\n' | cmp /tmp/debian-chroot/etc/apt/apt.conf.d/99mmdebstrap -
rm /tmp/debian-chroot/etc/apt/apt.conf.d/99mmdebstrap
tar -C /tmp/debian-chroot --one-file-system -c . | tar -t | sort | diff -u tar1.txt -

18
tests/arm64-without-qemu-support
View file

@ -1,18 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
if [ ! -e /mmdebstrap-testenv ]; then
echo "this test modifies the system and should only be run inside a container" >&2
exit 1
fi
apt-get remove --yes qemu-user-binfmt binfmt-support qemu-user
# the following is not necessary anymore since systemd-binfmt
# successfully disables support upon removal of qemu-user with
# the upload of src:systemd 251.2-4: https://bugs.debian.org/1012163
#echo 0 > /proc/sys/fs/binfmt_misc/qemu-aarch64
ret=0
{{ CMD }} --mode={{ MODE }} --variant=apt --architectures=arm64 {{ DIST }} /tmp/debian-chroot.tar {{ MIRROR }} || ret=$?
if [ "$ret" = 0 ]; then
echo expected failure but got exit $ret >&2
exit 1
fi

133
tests/as-debootstrap-unshare-wrapper
View file

@ -1,133 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
export SOURCE_DATE_EPOCH={{ SOURCE_DATE_EPOCH }}
prefix=
if [ "$(id -u)" -eq 0 ] && [ "{{ MODE }}" != "root" ] && [ "{{ MODE }}" != "auto" ]; then
if ! id "${SUDO_USER:-user}" >/dev/null 2>&1; then
if [ ! -e /mmdebstrap-testenv ]; then
echo "this test modifies the system and should only be run inside a container" >&2
exit 1
fi
useradd --home-dir "/home/${SUDO_USER:-user}" --create-home "${SUDO_USER:-user}"
fi
prefix="runuser -u ${SUDO_USER:-user} --"
fi
# debootstrap uses apt-config to figure out whether the system running it has
# any proxies configured and then runs the binary to set the http_proxy
# environment variable. This will fail if debootstrap is run in a linux user
# namespace because auto-apt-proxy will see /tmp/.auto-apt-proxy-0 as being
# owned by the user "nobody" and group "nogroup" and fail with:
# insecure cache dir /tmp/.auto-apt-proxy-0. Must be owned by UID 0 and have permissions 700
# We cannot overwrite a configuration item using the APT_CONFIG environment
# variable, so instead we use it to set the Dir configuration option
# to /dev/null to force all apt settings to their defaults.
# There is currently no better way to disable this behavior. See also:
# https://bugs.debian.org/1031105
# https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/90
AUTOPROXY=
eval "$(apt-config shell AUTOPROXY Acquire::http::Proxy-Auto-Detect)"
if [ -n "$AUTOPROXY" ] && [ -x "$AUTOPROXY" ] && [ -e /tmp/.auto-apt-proxy-0 ]; then
TMP_APT_CONFIG=$(mktemp)
echo 'Dir "/dev/null";' >"$TMP_APT_CONFIG"
chmod 644 "$TMP_APT_CONFIG"
fi
$prefix {{ CMD }} --variant=custom --mode={{ MODE }} \
--setup-hook='env '"${AUTOPROXY:+APT_CONFIG='$TMP_APT_CONFIG'}"' debootstrap --variant={{ VARIANT }} unstable "$1" {{ MIRROR }}' \
- /tmp/debian-mm.tar {{ MIRROR }}
if [ -n "$AUTOPROXY" ] && [ -x "$AUTOPROXY" ] && [ -e /tmp/.auto-apt-proxy-0 ]; then
rm "$TMP_APT_CONFIG"
fi
mkdir /tmp/debian-mm
tar --xattrs --xattrs-include='*' -C /tmp/debian-mm -xf /tmp/debian-mm.tar
mkdir /tmp/debian-debootstrap
tar --xattrs --xattrs-include='*' -C /tmp/debian-debootstrap -xf "cache/debian-unstable-{{ VARIANT }}.tar"
# diff cannot compare device nodes, so we use tar to do that for us and then
# delete the directory
tar -C /tmp/debian-debootstrap -cf dev1.tar ./dev
tar -C /tmp/debian-mm -cf dev2.tar ./dev
cmp dev1.tar dev2.tar >&2
rm dev1.tar dev2.tar
rm -r /tmp/debian-debootstrap/dev /tmp/debian-mm/dev
# remove downloaded deb packages
rm /tmp/debian-debootstrap/var/cache/apt/archives/*.deb
# remove aux-cache
rm /tmp/debian-debootstrap/var/cache/ldconfig/aux-cache
# remove logs
rm /tmp/debian-debootstrap/var/log/dpkg.log \
/tmp/debian-debootstrap/var/log/bootstrap.log \
/tmp/debian-debootstrap/var/log/alternatives.log \
/tmp/debian-mm/var/log/bootstrap.log
# clear out /run except for /run/lock
find /tmp/debian-debootstrap/run/ -mindepth 1 -maxdepth 1 ! -name lock -print0 | xargs --no-run-if-empty -0 rm -r
# debootstrap doesn't clean apt
rm /tmp/debian-debootstrap/var/lib/apt/lists/127.0.0.1_debian_dists_unstable_main_binary-{{ HOSTARCH }}_Packages \
/tmp/debian-debootstrap/var/lib/apt/lists/127.0.0.1_debian_dists_unstable_InRelease \
/tmp/debian-debootstrap/var/lib/apt/lists/127.0.0.1_debian_dists_unstable_Release \
/tmp/debian-debootstrap/var/lib/apt/lists/127.0.0.1_debian_dists_unstable_Release.gpg
if [ -e /tmp/debian-debootstrap/etc/machine-id ]; then
rm /tmp/debian-debootstrap/etc/machine-id /tmp/debian-mm/etc/machine-id
fi
rm /tmp/debian-mm/var/cache/apt/archives/lock
rm /tmp/debian-mm/var/lib/apt/lists/lock
rm /tmp/debian-mm/var/lib/dpkg/arch
# also needed for users that are created by systemd-sysusers before systemd 252
# https://github.com/systemd/systemd/pull/24534
for f in shadow shadow-; do
if [ ! -e /tmp/debian-debootstrap/etc/$f ]; then
continue
fi
if ! cmp /tmp/debian-debootstrap/etc/$f /tmp/debian-mm/etc/$f >&2; then
echo patching /etc/$f >&2
awk -v FS=: -v OFS=: -v SDE={{ SOURCE_DATE_EPOCH }} '{ print $1,$2,int(SDE/60/60/24),$4,$5,$6,$7,$8,$9 }' </tmp/debian-mm/etc/$f >/tmp/debian-mm/etc/$f.bak
cat /tmp/debian-mm/etc/$f.bak >/tmp/debian-mm/etc/$f
rm /tmp/debian-mm/etc/$f.bak
else
echo no difference for /etc/$f >&2
fi
done
# isc-dhcp-client postinst doesn't create this file in debootstrap run with
# unshared wrapper. The responsible postinst snippet was automatically added
# by dh_apparmor since isc-dhcp-client 4.4.3-P1-1.1
if [ -e /tmp/debian-debootstrap/etc/apparmor.d/local/sbin.dhclient ] && [ ! -s /tmp/debian-debootstrap/etc/apparmor.d/local/sbin.dhclient ]; then
echo /sbin/setcap >/tmp/debian-debootstrap/etc/apparmor.d/local/sbin.dhclient
fi
# check if the file content differs
diff --unified --no-dereference --recursive /tmp/debian-debootstrap /tmp/debian-mm >&2
# check permissions, ownership, symlink targets, modification times using tar
# mtimes of directories created by mmdebstrap will differ, thus we equalize them first
for d in etc/apt/preferences.d/ etc/apt/sources.list.d/ etc/dpkg/dpkg.cfg.d/ var/log/apt/; do
touch --date="@{{ SOURCE_DATE_EPOCH }}" /tmp/debian-debootstrap/$d /tmp/debian-mm/$d
done
# debootstrap never ran apt -- fixing permissions
for d in ./var/lib/apt/lists/partial ./var/cache/apt/archives/partial; do
chroot /tmp/debian-debootstrap chmod 0700 $d
chroot /tmp/debian-debootstrap chown _apt:root $d
done
tar -C /tmp/debian-debootstrap --numeric-owner --xattrs --xattrs-include='*' --sort=name --clamp-mtime --mtime="$(date --utc --date=@{{ SOURCE_DATE_EPOCH }} --iso-8601=seconds)" -cf /tmp/root1.tar .
tar -C /tmp/debian-mm --numeric-owner --xattrs --xattrs-include='*' --sort=name --clamp-mtime --mtime="$(date --utc --date=@{{ SOURCE_DATE_EPOCH }} --iso-8601=seconds)" -cf /tmp/root2.tar .
tar --full-time --verbose -tf /tmp/root1.tar >/tmp/root1.tar.list
tar --full-time --verbose -tf /tmp/root2.tar >/tmp/root2.tar.list
# despite SOURCE_DATE_EPOCH and --clamp-mtime, the timestamps in the tarball
# will slightly differ from each other in the sub-second precision (last
# decimals) so the tarballs will not be identical, so we use diff to compare
# content and tar to compare attributes
diff -u /tmp/root1.tar.list /tmp/root2.tar.list >&2
rm /tmp/root1.tar /tmp/root2.tar /tmp/root1.tar.list /tmp/root2.tar.list
rm /tmp/debian-mm.tar
rm -r /tmp/debian-debootstrap /tmp/debian-mm

21
tests/ascii-armored-keys
View file

@ -1,21 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
if [ ! -e /mmdebstrap-testenv ]; then
echo "this test modifies the system and should only be run inside a container" >&2
exit 1
fi
for f in /etc/apt/trusted.gpg.d/*.gpg /etc/apt/trusted.gpg.d/*.asc; do
[ -e "$f" ] || continue
rm "$f"
done
rmdir /etc/apt/trusted.gpg.d
mkdir /etc/apt/trusted.gpg.d
for f in /usr/share/keyrings/*.gpg; do
name=$(basename "$f" .gpg)
gpg --no-default-keyring --keyring="/usr/share/keyrings/$name.gpg" --armor --output="/etc/apt/trusted.gpg.d/$name.asc" --export
rm "/usr/share/keyrings/$name.gpg"
done
{{ CMD }} --mode=root --variant=apt {{ DIST }} /tmp/debian-chroot.tar {{ MIRROR }}
tar -tf /tmp/debian-chroot.tar | sort | diff -u tar1.txt -
rm -r /tmp/debian-chroot.tar

14
tests/aspcud-apt-solver
View file

@ -1,14 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
trap "rm -f /tmp/debian-chroot.tar" EXIT INT TERM
{{ CMD }} --mode={{ MODE }} --variant=custom \
--include "$(tr '\n' ',' <pkglist.txt)" \
--aptopt='APT::Solver "aspcud"' \
{{ DIST }} /tmp/debian-chroot.tar {{ MIRROR }}
{
tar -tf /tmp/debian-chroot.tar
echo ./var/lib/apt/extended_states
} | sort \
| grep -v '^./etc/apt/apt.conf.d/99mmdebstrap$' \
| diff -u tar1.txt -

22
tests/auto-mode-as-normal-user
View file

@ -1,22 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
trap "rm -f /tmp/debian-chroot.tar.gz" EXIT INT TERM
[ {{ MODE }} = "auto" ]
prefix=
if [ "$(id -u)" -eq 0 ]; then
if ! id "${SUDO_USER:-user}" >/dev/null 2>&1; then
if [ ! -e /mmdebstrap-testenv ]; then
echo "this test modifies the system and should only be run inside a container" >&2
exit 1
fi
useradd --home-dir "/home/${SUDO_USER:-user}" --create-home "${SUDO_USER:-user}"
fi
prefix="runuser -u ${SUDO_USER:-user} --"
fi
$prefix {{ CMD }} --mode={{ MODE }} --variant=apt {{ DIST }} /tmp/debian-chroot.tar.gz {{ MIRROR }}
tar -tf /tmp/debian-chroot.tar.gz | sort | diff -u tar1.txt -

14
tests/auto-mode-without-unshare-capabilities
View file

@ -1,14 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
if [ ! -e /mmdebstrap-testenv ]; then
echo "this test modifies the system and should only be run inside a container" >&2
exit 1
fi
useradd --home-dir /home/user --create-home user
if [ -e /proc/sys/kernel/unprivileged_userns_clone ] && [ "$(sysctl -n kernel.unprivileged_userns_clone)" = "1" ]; then
sysctl -w kernel.unprivileged_userns_clone=0
fi
runuser -u user -- {{ CMD }} --mode=auto --variant=apt {{ DIST }} /tmp/debian-chroot.tar.gz {{ MIRROR }}
tar -tf /tmp/debian-chroot.tar.gz | sort | diff -u tar1.txt -
rm /tmp/debian-chroot.tar.gz

14
tests/automatic-mirror-from-suite
View file

@ -1,14 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
if [ ! -e /mmdebstrap-testenv ]; then
echo "this test modifies the system and should only be run inside a container" >&2
exit 1
fi
cat <<HOSTS >>/etc/hosts
127.0.0.1 deb.debian.org
127.0.0.1 security.debian.org
HOSTS
{{ CMD }} --mode={{ MODE }} --variant=apt {{ DIST }} /tmp/debian-chroot.tar
tar -tf /tmp/debian-chroot.tar | sort | diff -u tar1.txt -
rm /tmp/debian-chroot.tar

257
tests/check-against-debootstrap-dist
View file

@ -1,257 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
export SOURCE_DATE_EPOCH={{ SOURCE_DATE_EPOCH }}
echo "SOURCE_DATE_EPOCH=$SOURCE_DATE_EPOCH"
# we create the apt user ourselves or otherwise its uid/gid will differ
# compared to the one chosen in debootstrap because of different installation
# order in comparison to the systemd users
# https://bugs.debian.org/969631
# we cannot use useradd because passwd is not Essential:yes
{{ CMD }} --variant={{ VARIANT }} --mode={{ MODE }} \
--essential-hook='[ {{ DIST }} = oldstable ] && [ {{ VARIANT }} = - ] && echo _apt:*:100:65534::/nonexistent:/usr/sbin/nologin >> "$1"/etc/passwd || :' \
"$(if [ {{ DIST }} = oldstable ]; then echo --merged-usr; else echo --hook-dir=./hooks/merged-usr; fi)" \
"$(case {{ DIST }} in oldstable) echo --include=e2fsprogs,mount,tzdata,gcc-9-base ;; stable) echo --include=e2fsprogs,mount,tzdata ;; *) echo --include=base-files ;; esac)" \
{{ DIST }} /tmp/debian-{{ DIST }}-mm.tar {{ MIRROR }}
mkdir /tmp/debian-{{ DIST }}-mm
tar --xattrs --xattrs-include='*' -C /tmp/debian-{{ DIST }}-mm -xf /tmp/debian-{{ DIST }}-mm.tar
rm /tmp/debian-{{ DIST }}-mm.tar
mkdir /tmp/debian-{{ DIST }}-debootstrap
tar --xattrs --xattrs-include='*' -C /tmp/debian-{{ DIST }}-debootstrap -xf "cache/debian-{{ DIST }}-{{ VARIANT }}.tar"
# diff cannot compare device nodes, so we use tar to do that for us and then
# delete the directory
tar -C /tmp/debian-{{ DIST }}-debootstrap -cf /tmp/dev1.tar ./dev
tar -C /tmp/debian-{{ DIST }}-mm -cf /tmp/dev2.tar ./dev
ret=0
cmp /tmp/dev1.tar /tmp/dev2.tar >&2 || ret=$?
if [ "$ret" -ne 0 ]; then
if type diffoscope >/dev/null; then
diffoscope /tmp/dev1.tar /tmp/dev2.tar
exit 1
else
echo "no diffoscope installed" >&2
fi
if type base64 >/dev/null; then
base64 /tmp/dev1.tar
base64 /tmp/dev2.tar
exit 1
else
echo "no base64 installed" >&2
fi
if type xxd >/dev/null; then
xxd /tmp/dev1.tar
xxd /tmp/dev2.tar
exit 1
else
echo "no xxd installed" >&2
fi
exit 1
fi
rm /tmp/dev1.tar /tmp/dev2.tar
rm -r /tmp/debian-{{ DIST }}-debootstrap/dev /tmp/debian-{{ DIST }}-mm/dev
# remove downloaded deb packages
rm /tmp/debian-{{ DIST }}-debootstrap/var/cache/apt/archives/*.deb
# remove aux-cache
rm /tmp/debian-{{ DIST }}-debootstrap/var/cache/ldconfig/aux-cache
# remove logs
rm /tmp/debian-{{ DIST }}-debootstrap/var/log/dpkg.log \
/tmp/debian-{{ DIST }}-debootstrap/var/log/bootstrap.log \
/tmp/debian-{{ DIST }}-debootstrap/var/log/alternatives.log
# remove *-old files
rm /tmp/debian-{{ DIST }}-debootstrap/var/cache/debconf/config.dat-old \
/tmp/debian-{{ DIST }}-mm/var/cache/debconf/config.dat-old
rm /tmp/debian-{{ DIST }}-debootstrap/var/cache/debconf/templates.dat-old \
/tmp/debian-{{ DIST }}-mm/var/cache/debconf/templates.dat-old
rm /tmp/debian-{{ DIST }}-debootstrap/var/lib/dpkg/status-old \
/tmp/debian-{{ DIST }}-mm/var/lib/dpkg/status-old
rm -f /tmp/debian-{{ DIST }}-debootstrap/var/lib/dpkg/diversions-old \
/tmp/debian-{{ DIST }}-mm/var/lib/dpkg/diversions-old
# remove dpkg files
rm /tmp/debian-{{ DIST }}-debootstrap/var/lib/dpkg/available
rm /tmp/debian-{{ DIST }}-debootstrap/var/lib/dpkg/cmethopt
# remove /var/lib/dpkg/arch
rm /tmp/debian-{{ DIST }}-mm/var/lib/dpkg/arch
# since we installed packages directly from the .deb files, Priorities differ
# thus we first check for equality and then remove the files
chroot /tmp/debian-{{ DIST }}-debootstrap dpkg --list >/tmp/dpkg1
chroot /tmp/debian-{{ DIST }}-mm dpkg --list >/tmp/dpkg2
diff -u /tmp/dpkg1 /tmp/dpkg2 >&2
rm /tmp/dpkg1 /tmp/dpkg2
grep -v '^Priority: ' /tmp/debian-{{ DIST }}-debootstrap/var/lib/dpkg/status >/tmp/status1
grep -v '^Priority: ' /tmp/debian-{{ DIST }}-mm/var/lib/dpkg/status >/tmp/status2
diff -u /tmp/status1 /tmp/status2 >&2
rm /tmp/status1 /tmp/status2
rm /tmp/debian-{{ DIST }}-debootstrap/var/lib/dpkg/status /tmp/debian-{{ DIST }}-mm/var/lib/dpkg/status
# debootstrap exposes the hosts's kernel version
if [ -e /tmp/debian-{{ DIST }}-debootstrap/etc/apt/apt.conf.d/01autoremove-kernels ]; then
rm /tmp/debian-{{ DIST }}-debootstrap/etc/apt/apt.conf.d/01autoremove-kernels
fi
if [ -e /tmp/debian-{{ DIST }}-mm/etc/apt/apt.conf.d/01autoremove-kernels ]; then
rm /tmp/debian-{{ DIST }}-mm/etc/apt/apt.conf.d/01autoremove-kernels
fi
# clear out /run except for /run/lock
find /tmp/debian-{{ DIST }}-debootstrap/run/ -mindepth 1 -maxdepth 1 ! -name lock -print0 | xargs --no-run-if-empty -0 rm -r
# debootstrap doesn't clean apt
rm /tmp/debian-{{ DIST }}-debootstrap/var/lib/apt/lists/127.0.0.1_debian_dists_{{ DIST }}_main_binary-{{ HOSTARCH }}_Packages \
/tmp/debian-{{ DIST }}-debootstrap/var/lib/apt/lists/127.0.0.1_debian_dists_{{ DIST }}_InRelease \
/tmp/debian-{{ DIST }}-debootstrap/var/lib/apt/lists/127.0.0.1_debian_dists_{{ DIST }}_Release \
/tmp/debian-{{ DIST }}-debootstrap/var/lib/apt/lists/127.0.0.1_debian_dists_{{ DIST }}_Release.gpg
if [ "{{ VARIANT }}" = "-" ]; then
rm /tmp/debian-{{ DIST }}-debootstrap/etc/machine-id
rm /tmp/debian-{{ DIST }}-mm/etc/machine-id
rm /tmp/debian-{{ DIST }}-debootstrap/var/lib/systemd/catalog/database
rm /tmp/debian-{{ DIST }}-mm/var/lib/systemd/catalog/database
case {{ DIST }} in oldstable | stable)
cap=$(chroot /tmp/debian-{{ DIST }}-debootstrap /sbin/getcap /bin/ping)
expected="/bin/ping cap_net_raw=ep"
if [ "$cap" != "$expected" ]; then
echo "expected bin/ping to have capabilities $expected" >&2
echo "but debootstrap produced: $cap" >&2
exit 1
fi
cap=$(chroot /tmp/debian-{{ DIST }}-mm /sbin/getcap /bin/ping)
if [ "$cap" != "$expected" ]; then
echo "expected bin/ping to have capabilities $expected" >&2
echo "but mmdebstrap produced: $cap" >&2
exit 1
fi
;;
esac
fi
rm /tmp/debian-{{ DIST }}-mm/var/cache/apt/archives/lock
rm /tmp/debian-{{ DIST }}-mm/var/lib/apt/extended_states
rm /tmp/debian-{{ DIST }}-mm/var/lib/apt/lists/lock
# the list of shells might be sorted wrongly
# /var/lib/dpkg/triggers/File might be sorted wrongly
for f in "/var/lib/dpkg/triggers/File" "/etc/shells"; do
f1="/tmp/debian-{{ DIST }}-debootstrap/$f"
f2="/tmp/debian-{{ DIST }}-mm/$f"
# both chroots must have the file
if [ ! -e "$f1" ] || [ ! -e "$f2" ]; then
continue
fi
# the file must be different
if cmp "$f1" "$f2" >&2; then
continue
fi
# then sort both
sort -o "$f1" "$f1"
sort -o "$f2" "$f2"
done
# Because of unreproducible uids (#969631) we created the _apt user ourselves
# and because passwd is not Essential:yes we didn't use useradd. But newer
# versions of adduser and shadow will create a different /etc/shadow
if [ "{{ VARIANT }}" = "-" ] && [ "{{ DIST}}" = oldstable ]; then
for f in shadow shadow-; do
if grep -q '^_apt:!:' /tmp/debian-{{ DIST }}-debootstrap/etc/$f; then
sed -i 's/^_apt:\*:\([^:]\+\):0:99999:7:::$/_apt:!:\1::::::/' /tmp/debian-{{ DIST }}-mm/etc/$f
fi
done
fi
for log in faillog lastlog; do
f1="/tmp/debian-{{ DIST }}-debootstrap/var/log/$log"
f2="/tmp/debian-{{ DIST }}-mm/var/log/$log"
# skip cmp if file is absent in both chroots
if [ ! -e "$f1" ] && [ ! -e "$f2" ]; then
continue
fi
if ! cmp "$f1" "$f2" >&2; then
# if the files differ, make sure they are all zeroes
cmp -n "$(stat -c %s "$f1")" "$f1" /dev/zero >&2
cmp -n "$(stat -c %s "$f2")" "$f2" /dev/zero >&2
# then delete them
rm "$f1" "$f2"
fi
done
if [ "{{ VARIANT }}" = "-" ]; then
# the order in which systemd and cron get installed differ and thus the order
# of lines in /etc/group and /etc/gshadow differs
for f in group group- gshadow gshadow-; do
for d in mm debootstrap; do
sort /tmp/debian-{{ DIST }}-$d/etc/$f >/tmp/debian-{{ DIST }}-$d/etc/$f.bak
mv /tmp/debian-{{ DIST }}-$d/etc/$f.bak /tmp/debian-{{ DIST }}-$d/etc/$f
done
done
# the order in which systemd and passwd get installed differ and thus
# the order of lines in /etc/shadow and /etc/shadow- differs
for f in shadow shadow-; do
for d in mm debootstrap; do
sort /tmp/debian-{{ DIST }}-$d/etc/$f >/tmp/debian-{{ DIST }}-$d/etc/$f.bak
mv /tmp/debian-{{ DIST }}-$d/etc/$f.bak /tmp/debian-{{ DIST }}-$d/etc/$f
done
done
# and since the order was different, ignore the *- files
for f in shadow- passwd-; do
for d in mm debootstrap; do
rm /tmp/debian-{{ DIST }}-$d/etc/$f
done
done
fi
# since debootstrap 1.0.133 there is no tzdata in the buildd variant and thus
# debootstrap creates its own /etc/localtime
if [ "{{ VARIANT }}" = "buildd" ] && [ "{{ DIST }}" != "stable" ] && [ "{{ DIST }}" != "oldstable" ]; then
[ "$(readlink /tmp/debian-{{ DIST }}-debootstrap/etc/localtime)" = /usr/share/zoneinfo/UTC ]
rm /tmp/debian-{{ DIST }}-debootstrap/etc/localtime
fi
# starting with systemd 255 upstream dropped splitusr support and depending on
# the installation order, symlink targets are prefixed with /usr or not
# See #1060000 and #1054137
case {{ DIST }} in testing | unstable)
for f in multi-user.target.wants/e2scrub_reap.service timers.target.wants/apt-daily-upgrade.timer timers.target.wants/apt-daily.timer timers.target.wants/e2scrub_all.timer; do
for d in mm debootstrap; do
[ -L "/tmp/debian-{{ DIST }}-$d/etc/systemd/system/$f" ] || continue
oldlink="$(readlink "/tmp/debian-{{ DIST }}-$d/etc/systemd/system/$f")"
case $oldlink in
/usr/*) : ;;
/*) oldlink="/usr$oldlink" ;;
*)
echo unexpected >&2
exit 1
;;
esac
ln -sf "$oldlink" "/tmp/debian-{{ DIST }}-$d/etc/systemd/system/$f"
done
done
;;
esac
# check if the file content differs
diff --unified --no-dereference --recursive /tmp/debian-{{ DIST }}-debootstrap /tmp/debian-{{ DIST }}-mm >&2
# check permissions, ownership, symlink targets, modification times using tar
# directory mtimes will differ, thus we equalize them first
find /tmp/debian-{{ DIST }}-debootstrap /tmp/debian-{{ DIST }}-mm -type d -print0 | xargs -0 touch --date="@{{ SOURCE_DATE_EPOCH }}"
# debootstrap never ran apt -- fixing permissions
for d in ./var/lib/apt/lists/partial ./var/cache/apt/archives/partial; do
unmergedPATH="$PATH$(if [ "{{ DIST }}" = oldstable ]; then echo :/bin:/sbin; fi)"
PATH="$unmergedPATH" chroot /tmp/debian-{{ DIST }}-debootstrap chmod 0700 $d
PATH="$unmergedPATH" chroot /tmp/debian-{{ DIST }}-debootstrap chown "$(id -u _apt):root" $d
done
tar -C /tmp/debian-{{ DIST }}-debootstrap --numeric-owner --sort=name --clamp-mtime --mtime="$(date --utc --date=@{{ SOURCE_DATE_EPOCH }} --iso-8601=seconds)" -cf /tmp/root1.tar .
tar -C /tmp/debian-{{ DIST }}-mm --numeric-owner --sort=name --clamp-mtime --mtime="$(date --utc --date=@{{ SOURCE_DATE_EPOCH }} --iso-8601=seconds)" -cf /tmp/root2.tar .
tar --full-time --verbose -tf /tmp/root1.tar >/tmp/root1.tar.list
tar --full-time --verbose -tf /tmp/root2.tar >/tmp/root2.tar.list
diff -u /tmp/root1.tar.list /tmp/root2.tar.list >&2
rm /tmp/root1.tar /tmp/root2.tar /tmp/root1.tar.list /tmp/root2.tar.list
# check if file properties (permissions, ownership, symlink names, modification time) differ
#
# we cannot use this (yet) because it cannot cope with paths that have [ or @ in them
#fmtree -c -p /tmp/debian-{{ DIST }}-debootstrap -k flags,gid,link,mode,size,time,uid | sudo fmtree -p /tmp/debian-{{ DIST }}-mm
rm -r /tmp/debian-{{ DIST }}-debootstrap /tmp/debian-{{ DIST }}-mm

28
tests/check-for-bit-by-bit-identical-format-output
View file

@ -1,28 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
export SOURCE_DATE_EPOCH={{ SOURCE_DATE_EPOCH }}
trap "rm -f /tmp/debian-chroot-{{ MODE }}.{{ FORMAT }}" EXIT INT TERM
case {{ MODE }} in unshare | fakechroot) : ;; *) exit 1 ;; esac
prefix=
if [ "$(id -u)" -eq 0 ] && [ "{{ MODE }}" != "root" ] && [ "{{ MODE }}" != "auto" ]; then
if ! id "${SUDO_USER:-user}" >/dev/null 2>&1; then
if [ ! -e /mmdebstrap-testenv ]; then
echo "this test modifies the system and should only be run inside a container" >&2
exit 1
fi
useradd --home-dir "/home/${SUDO_USER:-user}" --create-home "${SUDO_USER:-user}"
fi
prefix="runuser -u ${SUDO_USER:-user} --"
fi
$prefix {{ CMD }} --mode={{ MODE }} --variant={{ VARIANT }} {{ DIST }} /tmp/debian-chroot-{{ MODE }}.{{ FORMAT }} {{ MIRROR }}
cmp ./cache/mmdebstrap-{{ DIST }}-{{ VARIANT }}.{{ FORMAT }} /tmp/debian-chroot-{{ MODE }}.{{ FORMAT }} \
|| diffoscope ./cache/mmdebstrap-{{ DIST }}-{{ VARIANT }}.{{ FORMAT }} /tmp/debian-chroot-{{ MODE }}.{{ FORMAT }}
# we cannot test chrootless mode here, because mmdebstrap relies on the
# usrmerge package to set up merged-/usr and that doesn't work in chrootless
# mode

8
tests/chroot-directory-not-accessible-by-apt-user
View file

@ -1,8 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
trap "rm -rf /tmp/debian-chroot" EXIT INT TERM
mkdir /tmp/debian-chroot
chmod 700 /tmp/debian-chroot
{{ CMD }} --mode=root --variant=apt {{ DIST }} /tmp/debian-chroot {{ MIRROR }}
tar -C /tmp/debian-chroot --one-file-system -c . | tar -t | sort | diff -u tar1.txt -

16
tests/chrootless
View file

@ -1,16 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
export SOURCE_DATE_EPOCH={{ SOURCE_DATE_EPOCH }}
trap "rm -f /tmp/chrootless.tar /tmp/root.tar" EXIT INT TERM
# we need --hook-dir=./hooks/merged-usr because usrmerge does not understand
# DPKG_ROOT
for INCLUDE in '' 'apt' 'apt,build-essential' 'systemd-sysv'; do
for MODE in root chrootless; do
{{ CMD }} --mode=$MODE --variant={{ VARIANT }} --hook-dir=./hooks/merged-usr \
${INCLUDE:+--include="$INCLUDE"} --skip=check/chrootless \
{{ DIST }} "/tmp/$MODE.tar" {{ MIRROR }}
done
cmp /tmp/root.tar /tmp/chrootless.tar || diffoscope /tmp/root.tar /tmp/chrootless.tar
rm /tmp/chrootless.tar /tmp/root.tar
done

43
tests/chrootless-fakeroot
View file

@ -1,43 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
export SOURCE_DATE_EPOCH={{ SOURCE_DATE_EPOCH }}
trap "rm -f /tmp/chrootless.tar /tmp/root.tar" EXIT INT TERM
[ {{ MODE }} = chrootless ]
prefix=
if [ "$(id -u)" -eq 0 ] && [ "{{ MODE }}" != "root" ] && [ "{{ MODE }}" != "auto" ]; then
if ! id "${SUDO_USER:-user}" >/dev/null 2>&1; then
if [ ! -e /mmdebstrap-testenv ]; then
echo "this test modifies the system and should only be run inside a container" >&2
exit 1
fi
useradd --home-dir "/home/${SUDO_USER:-user}" --create-home "${SUDO_USER:-user}"
fi
prefix="runuser -u ${SUDO_USER:-user} --"
fi
MMTARFILTER=
[ -x /usr/bin/mmtarfilter ] && MMTARFILTER=/usr/bin/mmtarfilter
[ -x ./tarfilter ] && MMTARFILTER=./tarfilter
# we need --hook-dir=./hooks/merged-usr because usrmerge does not understand
# DPKG_ROOT
# permissions drwxr-sr-x and extended attributes of ./var/log/journal/ cannot
# be preserved under fakeroot
# this applies to 'z' lines in files in /usr/lib/tmpfiles.d/
for INCLUDE in '' 'apt' 'apt,build-essential' 'systemd-sysv'; do
{{ CMD }} --variant={{ VARIANT }} --hook-dir=./hooks/merged-usr \
${INCLUDE:+--include="$INCLUDE"} \
{{ DIST }} - {{ MIRROR }} \
| "$MMTARFILTER" --path-exclude="/var/log/journal" --path-exclude="/etc/credstore*" \
>/tmp/root.tar
$prefix fakeroot {{ CMD }} --mode={{ MODE }} --variant={{ VARIANT }} --hook-dir=./hooks/merged-usr \
${INCLUDE:+--include="$INCLUDE"} \
{{ DIST }} - {{ MIRROR }} \
| "$MMTARFILTER" --path-exclude="/var/log/journal" --path-exclude="/etc/credstore*" \
>/tmp/chrootless.tar
cmp /tmp/root.tar /tmp/chrootless.tar || diffoscope /tmp/root.tar /tmp/chrootless.tar
rm /tmp/chrootless.tar /tmp/root.tar
done

66
tests/chrootless-foreign
View file

@ -1,66 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
export SOURCE_DATE_EPOCH={{ SOURCE_DATE_EPOCH }}
trap "rm -f /tmp/chrootless.tar /tmp/root.tar" EXIT INT TERM
if [ ! -e /mmdebstrap-testenv ]; then
echo "this test modifies the system and should only be run inside a container" >&2
exit 1
fi
deb2qemu() {
case "$1" in
amd64) echo x86_64 ;;
arm64) echo aarch64 ;;
armel | armhf) echo arm ;;
ppc64el) echo ppc64le ;;
*) echo "$1" ;;
esac
}
if [ "$(dpkg --print-architecture)" = "arm64" ]; then
arch=amd64
else
arch=arm64
fi
[ "$(id -u)" -eq 0 ]
[ -e "/proc/sys/fs/binfmt_misc/qemu-$(deb2qemu "$arch")" ]
# we need --hook-dir=./hooks/merged-usr because usrmerge does not understand
# DPKG_ROOT
#
# dpkg is unable to install architecture arch:all packages with a
# dependency on an arch:any package (perl-modules-5.34 in this case)
# inside foreign architecture chrootless chroots, because dpkg will use
# its own architecture as the native architecture, see #825385 and #1020533
# So we are not testing the installation of apt,build-essential here.
for INCLUDE in '' 'apt' 'systemd-sysv'; do
echo 1 >"/proc/sys/fs/binfmt_misc/qemu-$(deb2qemu "$arch")"
arch-test "$arch"
{{ CMD }} --mode=root --architecture="$arch" --variant={{ VARIANT }} \
--hook-dir=./hooks/merged-usr ${INCLUDE:+--include="$INCLUDE"} \
{{ DIST }} "/tmp/root.tar" {{ MIRROR }}
echo 0 >"/proc/sys/fs/binfmt_misc/qemu-$(deb2qemu "$arch")"
arch-test "$arch" && exit 1
{{ CMD }} --mode=chrootless --architecture="$arch" --variant={{ VARIANT }} \
--hook-dir=./hooks/merged-usr ${INCLUDE:+--include="$INCLUDE"} \
--skip=check/chrootless {{ DIST }} "/tmp/chrootless.tar" {{ MIRROR }}
# when creating a foreign architecture chroot, the tarballs are not
# bit-by-bit identical but contain a few remaining differences:
#
# * /etc/ld.so.cache -- hard problem, must be solved in glibc upstream
# * /var/lib/dpkg/triggers -- #990712
# * /var/cache/debconf/*.dat-old -- needs investigation
for tar in root chrootless; do
./tarfilter <"/tmp/$tar.tar" \
--path-exclude=/var/cache/debconf/config.dat-old \
--path-exclude=/var/cache/debconf/templates.dat-old \
--path-exclude=/etc/ld.so.cache \
--path-exclude=/var/lib/dpkg/triggers/File \
--path-exclude=/var/lib/dpkg/triggers/ldconfig \
>"/tmp/$tar.tar.tmp"
mv "/tmp/$tar.tar.tmp" "/tmp/$tar.tar"
done
cmp /tmp/root.tar /tmp/chrootless.tar || diffoscope /tmp/root.tar /tmp/chrootless.tar
rm /tmp/chrootless.tar /tmp/root.tar
done

44
tests/compare-output-with-pre-seeded-var-cache-apt-archives
View file

@ -1,44 +0,0 @@
#!/bin/sh
#
# test that the user can drop archives into /var/cache/apt/archives as well as
# into /var/cache/apt/archives/partial
set -eu
export LC_ALL=C.UTF-8
export SOURCE_DATE_EPOCH={{ SOURCE_DATE_EPOCH }}
if [ ! -e /mmdebstrap-testenv ]; then
echo "this test requires the cache directory to be mounted on /mnt and should only be run inside a container" >&2
exit 1
fi
tmpdir=$(mktemp -d)
trap 'rm -f "$tmpdir"/*.deb /tmp/orig.tar /tmp/test1.tar /tmp/test2.tar; rmdir "$tmpdir"' EXIT INT TERM
include="--include=doc-debian"
if [ "{{ VARIANT }}" = "custom" ]; then
include="$include,base-files,base-passwd,coreutils,dash,diffutils,dpkg,libc-bin,sed"
fi
{{ CMD }} $include --mode={{ MODE }} --variant={{ VARIANT }} \
--setup-hook='mkdir -p "$1"/var/cache/apt/archives/partial' \
--setup-hook='touch "$1"/var/cache/apt/archives/lock' \
--setup-hook='chmod 0640 "$1"/var/cache/apt/archives/lock' \
{{ DIST }} - {{ MIRROR }} >/tmp/orig.tar
# somehow, when trying to create a tarball from the 9p mount, tar throws the
# following error: tar: ./doc-debian_6.4_all.deb: File shrank by 132942 bytes; padding with zeros
# to reproduce, try: tar --directory /mnt/cache/debian/pool/main/d/doc-debian/ --create --file - . | tar --directory /tmp/ --extract --file -
# this will be different:
# md5sum /mnt/cache/debian/pool/main/d/doc-debian/*.deb /tmp/*.deb
# another reason to copy the files into a new directory is, that we can use shell globs
cp /mnt/cache/debian/pool/main/b/busybox/busybox_*"_{{ HOSTARCH }}.deb" /mnt/cache/debian/pool/main/a/apt/apt_*"_{{ HOSTARCH }}.deb" "$tmpdir"
{{ CMD }} $include --mode={{ MODE }} --variant={{ VARIANT }} \
--setup-hook='mkdir -p "$1"/var/cache/apt/archives/partial' \
--setup-hook='sync-in "'"$tmpdir"'" /var/cache/apt/archives/partial' \
{{ DIST }} - {{ MIRROR }} >/tmp/test1.tar
cmp /tmp/orig.tar /tmp/test1.tar
{{ CMD }} $include --mode={{ MODE }} --variant={{ VARIANT }} \
--customize-hook='touch "$1"/var/cache/apt/archives/partial' \
--setup-hook='mkdir -p "$1"/var/cache/apt/archives/' \
--setup-hook='sync-in "'"$tmpdir"'" /var/cache/apt/archives/' \
--setup-hook='chmod 0755 "$1"/var/cache/apt/archives/' \
--customize-hook='find "'"$tmpdir"'" -type f -exec md5sum "{}" \; | sed "s|"'"$tmpdir"'"|$1/var/cache/apt/archives|" | md5sum --check' \
{{ DIST }} - {{ MIRROR }} >/tmp/test2.tar
cmp /tmp/orig.tar /tmp/test2.tar

10
tests/copy-mirror
View file

@ -1,10 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
if [ ! -e /mmdebstrap-testenv ]; then
echo "this test requires the cache directory to be mounted on /mnt and should only be run inside a container" >&2
exit 1
fi
{{ CMD }} --mode={{ MODE }} --variant=apt {{ DIST }} /tmp/debian-chroot.tar "deb copy:///mnt/cache/debian {{ DIST }} main"
tar -tf /tmp/debian-chroot.tar | sort | diff -u tar1.txt -
rm /tmp/debian-chroot.tar

9
tests/create-directory
View file

@ -1,9 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
trap "rm -rf /tmp/debian-chroot" EXIT INT TERM
{{ CMD }} --mode=root --variant=apt {{ DIST }} /tmp/debian-chroot {{ MIRROR }}
chroot /tmp/debian-chroot dpkg-query --showformat '${binary:Package}\n' --show >pkglist.txt
tar -C /tmp/debian-chroot --one-file-system -c . | tar -t | sort >tar1.txt

29
tests/create-directory-dry-run
View file

@ -1,29 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
{{ CMD }} --mode={{ MODE }} --dry-run --variant=apt \
--setup-hook="exit 1" \
--essential-hook="exit 1" \
--customize-hook="exit 1" \
{{ DIST }} /tmp/debian-chroot {{ MIRROR }}
rm /tmp/debian-chroot/dev/console
rm /tmp/debian-chroot/dev/fd
rm /tmp/debian-chroot/dev/full
rm /tmp/debian-chroot/dev/null
rm /tmp/debian-chroot/dev/ptmx
rm /tmp/debian-chroot/dev/random
rm /tmp/debian-chroot/dev/stderr
rm /tmp/debian-chroot/dev/stdin
rm /tmp/debian-chroot/dev/stdout
rm /tmp/debian-chroot/dev/tty
rm /tmp/debian-chroot/dev/urandom
rm /tmp/debian-chroot/dev/zero
rm /tmp/debian-chroot/etc/apt/sources.list
rm /tmp/debian-chroot/etc/fstab
rm /tmp/debian-chroot/etc/hostname
rm /tmp/debian-chroot/etc/resolv.conf
rm /tmp/debian-chroot/var/lib/apt/lists/lock
rm /tmp/debian-chroot/var/lib/dpkg/status
rm /tmp/debian-chroot/var/lib/dpkg/arch
# the rest should be empty directories that we can rmdir recursively
find /tmp/debian-chroot -depth -print0 | xargs -0 rmdir

78
tests/create-foreign-tarball
View file

@ -1,78 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
prefix=
if [ "$(id -u)" -eq 0 ] && [ "{{ MODE }}" != "root" ] && [ "{{ MODE }}" != "auto" ]; then
if ! id "${SUDO_USER:-user}" >/dev/null 2>&1; then
if [ ! -e /mmdebstrap-testenv ]; then
echo "this test modifies the system and should only be run inside a container" >&2
exit 1
fi
useradd --home-dir "/home/${SUDO_USER:-user}" --create-home "${SUDO_USER:-user}"
fi
prefix="runuser -u ${SUDO_USER:-user} --"
fi
case "$(dpkg --print-architecture)" in
arm64)
native_arch=arm64
native_gnu=aarch64-linux-gnu
foreign_arch=amd64
foreign_gnu=x86_64-linux-gnu
;;
amd64)
native_arch=amd64
native_gnu=x86_64-linux-gnu
foreign_arch=arm64
foreign_gnu=aarch64-linux-gnu
;;
*)
echo "unsupported native architecture" >&2
exit 1
;;
esac
[ "{{ MODE }}" = "fakechroot" ] && prefix="$prefix fakechroot fakeroot"
$prefix {{ CMD }} --mode={{ MODE }} --variant=apt --architectures="$foreign_arch" \
{{ DIST }} /tmp/debian-chroot.tar {{ MIRROR }}
# we ignore differences between architectures by ignoring some files
# and renaming others
{
tar -tf /tmp/debian-chroot.tar \
| grep -v '^\./usr/bin/i386$' \
| grep -v '^\./usr/bin/x86_64$' \
| grep -v '^\./lib64$' \
| grep -v '^\./usr/lib64/$' \
| grep -v '^\./usr/lib64/ld-linux-x86-64\.so\.2$' \
| grep -v '^\./usr/lib/ld-linux-aarch64\.so\.1$' \
| grep -v "^\\./usr/lib/$foreign_gnu/ld-linux-aarch64\\.so\\.1$" \
| grep -v "^\\./usr/lib/$foreign_gnu/ld-linux-x86-64\\.so\\.2$" \
| grep -v "^\\./usr/lib/$foreign_gnu/perl/5\\.[0-9][.0-9]\\+/.*\\.ph$" \
| grep -v "^\\./usr/lib/$foreign_gnu/libmvec\\.so\\.1$" \
| grep -v "^\\./usr/share/doc/[^/]\\+/changelog\\(\\.Debian\\)\\?\\.$foreign_arch\\.gz$" \
| grep -v '^\./usr/share/man/man8/i386\.8\.gz$' \
| grep -v '^\./usr/share/man/man8/x86_64\.8\.gz$' \
| sed "s/$foreign_gnu/$native_gnu/" \
| sed "s/$foreign_arch/$native_arch/"
} | sort >/tmp/tar2.txt
{
grep <tar1.txt -v '^\./usr/bin/i386$' \
| grep -v '^\./usr/bin/x86_64$' \
| grep -v '^\./lib32$' \
| grep -v '^\./lib64$' \
| grep -v '^\./libx32$' \
| grep -v '^\./usr/lib32/$' \
| grep -v '^\./usr/libx32/$' \
| grep -v '^\./usr/lib64/$' \
| grep -v '^\./usr/lib64/ld-linux-x86-64\.so\.2$' \
| grep -v '^\./usr/lib/ld-linux-aarch64\.so\.1$' \
| grep -v "^\\./usr/lib/$native_gnu/ld-linux-x86-64\\.so\\.2$" \
| grep -v "^\\./usr/lib/$native_gnu/ld-linux-aarch64\\.so\\.1$" \
| grep -v "^\\./usr/lib/$native_gnu/libmvec\\.so\\.1$" \
| grep -v "^\\./usr/lib/$native_gnu/perl/5\\.[0-9][.0-9]\\+/.*\\.ph$" \
| grep -v "^\\./usr/share/doc/[^/]\\+/changelog\\(\\.Debian\\)\\?\\.$native_arch\\.gz$" \
| grep -v '^\./usr/share/man/man8/i386\.8\.gz$' \
| grep -v '^\./usr/share/man/man8/x86_64\.8\.gz$'
} | sort | diff -u - /tmp/tar2.txt >&2
rm /tmp/debian-chroot.tar /tmp/tar2.txt

20
tests/create-gzip-compressed-tarball
View file

@ -1,20 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
prefix=
if [ "$(id -u)" -eq 0 ] && [ "{{ MODE }}" != "root" ] && [ "{{ MODE }}" != "auto" ]; then
if ! id "${SUDO_USER:-user}" >/dev/null 2>&1; then
if [ ! -e /mmdebstrap-testenv ]; then
echo "this test modifies the system and should only be run inside a container" >&2
exit 1
fi
useradd --home-dir "/home/${SUDO_USER:-user}" --create-home "${SUDO_USER:-user}"
fi
prefix="runuser -u ${SUDO_USER:-user} --"
fi
$prefix {{ CMD }} --mode={{ MODE }} --variant=apt {{ DIST }} /tmp/debian-chroot.tar.gz {{ MIRROR }}
printf '\037\213\010' | cmp --bytes=3 /tmp/debian-chroot.tar.gz -
tar -tf /tmp/debian-chroot.tar.gz | sort | diff -u tar1.txt -
rm /tmp/debian-chroot.tar.gz

27
tests/create-tarball-dry-run
View file

@ -1,27 +0,0 @@
#!/bin/sh
#
# we are testing all variants here because with 0.7.5 we had a bug:
# mmdebstrap sid /dev/null --simulate ==> E: cannot read /var/cache/apt/archives/
set -eu
export LC_ALL=C.UTF-8
prefix=
include=,
if [ "$(id -u)" -eq 0 ] && [ "{{ MODE }}" != root ] && [ "{{ MODE }}" != auto ]; then
if ! id "${SUDO_USER:-user}" >/dev/null 2>&1; then
if [ ! -e /mmdebstrap-testenv ]; then
echo "this test modifies the system and should only be run inside a container" >&2
exit 1
fi
useradd --home-dir "/home/${SUDO_USER:-user}" --create-home "${SUDO_USER:-user}"
fi
prefix="runuser -u ${SUDO_USER:-user} --"
if [ "{{ VARIANT }}" = extract ] || [ "{{ VARIANT }}" = custom ]; then
include="$(tr '\n' ',' <pkglist.txt)"
fi
fi
$prefix {{ CMD }} --mode={{ MODE }} --include="$include" --dry-run --variant={{ VARIANT }} {{ DIST }} /tmp/debian-chroot.tar {{ MIRROR }}
if [ -e /tmp/debian-chroot.tar ]; then
echo "/tmp/debian-chroot.tar must not be created with --dry-run" >&2
exit 1
fi

12
tests/create-tarball-with-tmp-mounted-nodev
View file

@ -1,12 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
if [ ! -e /mmdebstrap-testenv ]; then
echo "this test modifies the system and should only be run inside a container" >&2
exit 1
fi
mount -t tmpfs -o nodev,nosuid,size=400M tmpfs /tmp
# use --customize-hook to exercise the mounting/unmounting code of block devices in root mode
{{ CMD }} --mode=root --variant=apt --customize-hook='mount | grep /dev/full' --customize-hook='test "$(echo foo | tee /dev/full 2>&1 1>/dev/null)" = "tee: /dev/full: No space left on device"' {{ DIST }} /tmp/debian-chroot.tar {{ MIRROR }}
tar -tf /tmp/debian-chroot.tar | sort | diff -u tar1.txt -
rm /tmp/debian-chroot.tar

33
tests/custom-tmpdir
View file

@ -1,33 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
[ "$(id -u)" -eq 0 ]
[ {{ MODE }} = "unshare" ]
if ! id "${SUDO_USER:-user}" >/dev/null 2>&1; then
if [ ! -e /mmdebstrap-testenv ]; then
echo "this test modifies the system and should only be run inside a container" >&2
exit 1
fi
useradd --home-dir "/home/${SUDO_USER:-user}" --create-home "${SUDO_USER:-user}"
fi
prefix="runuser -u ${SUDO_USER:-user} --"
# https://www.etalabs.net/sh_tricks.html
quote() { printf %s\\n "$1" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/'/"; }
homedir=$($prefix sh -c 'cd && pwd')
# apt:test/integration/test-apt-key
TMPDIR_ADD='This is fü$$ing cràzy, $(apt -v)$!'
$prefix mkdir "$homedir/$TMPDIR_ADD"
# make sure the unshared user can traverse into the TMPDIR
chmod 711 "$homedir"
# set permissions and sticky bit like the real /tmp
chmod 1777 "$homedir/$TMPDIR_ADD"
$prefix env TMPDIR="$homedir/$TMPDIR_ADD" {{ CMD }} --mode={{ MODE }} --variant=apt \
--setup-hook='case "$1" in '"$(quote "$homedir/$TMPDIR_ADD/mmdebstrap.")"'??????????) exit 0;; *) echo "$1"; exit 1;; esac' \
{{ DIST }} /tmp/debian-chroot.tar {{ MIRROR }}
tar -tf /tmp/debian-chroot.tar | sort | diff -u tar1.txt -
# use rmdir as a quick check that nothing is remaining in TMPDIR
$prefix rmdir "$homedir/$TMPDIR_ADD"
rm /tmp/debian-chroot.tar

16
tests/customize-hook
View file

@ -1,16 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
trap "rm -rf /tmp/debian-chroot; rm -f /tmp/customize.sh" EXIT INT TERM
cat <<'SCRIPT' >/tmp/customize.sh
#!/bin/sh
chroot "$1" whoami > "$1/output2"
chroot "$1" pwd >> "$1/output2"
SCRIPT
chmod +x /tmp/customize.sh
{{ CMD }} --mode=root --variant=apt --customize-hook='chroot "$1" sh -c "whoami; pwd" > "$1/output1"' --customize-hook=/tmp/customize.sh {{ DIST }} /tmp/debian-chroot {{ MIRROR }}
printf "root\n/\n" | cmp /tmp/debian-chroot/output1
printf "root\n/\n" | cmp /tmp/debian-chroot/output2
rm /tmp/debian-chroot/output1
rm /tmp/debian-chroot/output2
tar -C /tmp/debian-chroot --one-file-system -c . | tar -t | sort | diff -u tar1.txt -

30
tests/cwd-directory-not-accessible-by-unshared-user
View file

@ -1,30 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
[ "$(id -u)" -eq 0 ]
[ {{ MODE }} = "unshare" ]
if ! id "${SUDO_USER:-user}" >/dev/null 2>&1; then
if [ ! -e /mmdebstrap-testenv ]; then
echo "this test modifies the system and should only be run inside a container" >&2
exit 1
fi
useradd --home-dir "/home/${SUDO_USER:-user}" --create-home "${SUDO_USER:-user}"
fi
prefix="runuser -u ${SUDO_USER:-user} --"
mkdir /tmp/debian-chroot
chmod 700 /tmp/debian-chroot
chown "${SUDO_USER:-user}:${SUDO_USER:-user}" /tmp/debian-chroot
set -- env --chdir=/tmp/debian-chroot
if [ "{{ CMD }}" = "./mmdebstrap" ]; then
set -- "$@" "$(realpath --canonicalize-existing ./mmdebstrap)"
elif [ "{{ CMD }}" = "perl -MDevel::Cover=-silent,-nogcov ./mmdebstrap" ]; then
set -- "$@" perl -MDevel::Cover=-silent,-nogcov "$(realpath --canonicalize-existing ./mmdebstrap)"
else
set -- "$@" {{ CMD }}
fi
$prefix "$@" --mode={{ MODE }} --variant=apt {{ DIST }} /tmp/debian-chroot.tar {{ MIRROR }}
tar -tf /tmp/debian-chroot.tar | sort | diff -u tar1.txt -
rm /tmp/debian-chroot.tar

45
tests/deb822-1-2
View file

@ -1,45 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
trap "rm -rf /tmp/debian-chroot; rm -f /tmp/sources.list /tmp/deb822.sources" EXIT INT TERM
cat <<SOURCES >/tmp/deb822.sources
Types: deb
URIs: {{ MIRROR }}1
Suites: {{ DIST }}
Components: main
SOURCES
echo "deb {{ MIRROR }}2 {{ DIST }} main" >/tmp/sources.list
echo "deb {{ MIRROR }}3 {{ DIST }} main" \
| {{ CMD }} --mode={{ MODE }} --variant=apt {{ DIST }} \
/tmp/debian-chroot \
/tmp/deb822.sources \
{{ MIRROR }}4 \
- \
"deb {{ MIRROR }}5 {{ DIST }} main" \
{{ MIRROR }}6 \
/tmp/sources.list
test ! -e /tmp/debian-chroot/etc/apt/sources.list
cat <<SOURCES | cmp /tmp/debian-chroot/etc/apt/sources.list.d/0000deb822.sources -
Types: deb
URIs: {{ MIRROR }}1
Suites: {{ DIST }}
Components: main
SOURCES
cat <<SOURCES | cmp /tmp/debian-chroot/etc/apt/sources.list.d/0001main.list -
deb {{ MIRROR }}4 {{ DIST }} main
deb {{ MIRROR }}3 {{ DIST }} main
deb {{ MIRROR }}5 {{ DIST }} main
deb {{ MIRROR }}6 {{ DIST }} main
SOURCES
echo "deb {{ MIRROR }}2 {{ DIST }} main" | cmp /tmp/debian-chroot/etc/apt/sources.list.d/0002sources.list -
tar -C /tmp/debian-chroot --one-file-system -c . \
| {
tar -t \
| grep -v "^./etc/apt/sources.list.d/0000deb822.sources$" \
| grep -v "^./etc/apt/sources.list.d/0001main.list$" \
| grep -v "^./etc/apt/sources.list.d/0002sources.list"
printf "./etc/apt/sources.list\n"
} | sort | diff -u tar1.txt -

44
tests/deb822-2-2
View file

@ -1,44 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
trap "rm -rf /tmp/debian-chroot; rm -f /tmp/sources /tmp/deb822" EXIT INT TERM
cat <<SOURCES >/tmp/deb822
Types: deb
URIs: {{ MIRROR }}1
Suites: {{ DIST }}
Components: main
SOURCES
echo "deb {{ MIRROR }}2 {{ DIST }} main" >/tmp/sources
cat <<SOURCES | {{ CMD }} --mode={{ MODE }} --variant=apt {{ DIST }} \
/tmp/debian-chroot \
/tmp/deb822 \
- \
/tmp/sources
Types: deb
URIs: {{ MIRROR }}3
Suites: {{ DIST }}
Components: main
SOURCES
test ! -e /tmp/debian-chroot/etc/apt/sources.list
ls -lha /tmp/debian-chroot/etc/apt/sources.list.d/
cat <<SOURCES | cmp /tmp/debian-chroot/etc/apt/sources.list.d/0000deb822.sources -
Types: deb
URIs: {{ MIRROR }}1
Suites: {{ DIST }}
Components: main
SOURCES
cat <<SOURCES | cmp /tmp/debian-chroot/etc/apt/sources.list.d/0001main.sources -
Types: deb
URIs: {{ MIRROR }}3
Suites: {{ DIST }}
Components: main
SOURCES
echo "deb {{ MIRROR }}2 {{ DIST }} main" | cmp /tmp/debian-chroot/etc/apt/sources.list.d/0002sources.list -
tar -C /tmp/debian-chroot --one-file-system -c . \
| {
tar -t \
| grep -v "^./etc/apt/sources.list.d/0000deb822.sources$" \
| grep -v "^./etc/apt/sources.list.d/0001main.sources$" \
| grep -v "^./etc/apt/sources.list.d/0002sources.list$"
printf "./etc/apt/sources.list\n"
} | sort | diff -u tar1.txt -

16
tests/debootstrap
View file

@ -1,16 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
export SOURCE_DATE_EPOCH={{ SOURCE_DATE_EPOCH }}
tmpdir="$(mktemp -d)"
chmod 755 "$tmpdir"
ret=0
debootstrap "$([ "{{ DIST }}" = oldstable ] && echo --no-merged-usr || echo --merged-usr)" --variant={{ VARIANT }} {{ DIST }} "$tmpdir" {{ MIRROR }} || ret=$?
if [ "$ret" -ne 0 ]; then
echo "E: debootstrap failed, dumping $tmpdir/debootstrap/debootstrap.log"
cat "$tmpdir/debootstrap/debootstrap.log"
exit 1
fi
tar --sort=name --mtime=@$SOURCE_DATE_EPOCH --clamp-mtime --numeric-owner --one-file-system --xattrs -C "$tmpdir" -c . >"./cache/debian-{{ DIST }}-{{ VARIANT }}.tar"
rm -r "$tmpdir"

6
tests/debootstrap-no-op-options
View file

@ -1,6 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
{{ CMD }} --mode=root --variant=apt --resolve-deps --merged-usr --no-merged-usr --force-check-gpg {{ DIST }} /tmp/debian-chroot {{ MIRROR }}
tar -C /tmp/debian-chroot --one-file-system -c . | tar -t | sort | diff -u tar1.txt -
rm -r /tmp/debian-chroot

15
tests/debug
View file

@ -1,15 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
export SOURCE_DATE_EPOCH={{ SOURCE_DATE_EPOCH }}
trap "rm -f /tmp/debian-chroot.tar" EXIT INT TERM
# we use variant standard in verbose mode to see the maximum number of packages
# that was chosen in case of USE_HOST_APT_CONFIG=yes
case {{ VARIANT }} in standard) : ;; *) exit 1 ;; esac
{{ CMD }} --variant={{ VARIANT }} --debug {{ DIST }} /tmp/debian-chroot.tar {{ MIRROR }}
cmp ./cache/mmdebstrap-{{ DIST }}-{{ VARIANT }}.tar /tmp/debian-chroot.tar \
|| diffoscope ./cache/mmdebstrap-{{ DIST }}-{{ VARIANT }}.tar /tmp/debian-chroot.tar

6
tests/debug-output-on-fake-tty
View file

@ -1,6 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
trap "rm -f /tmp/debian-chroot.tar" EXIT INT TERM
script -qfc "{{ CMD }} --mode={{ MODE }} --debug --variant=apt {{ DIST }} /tmp/debian-chroot.tar {{ MIRROR }}" /dev/null
tar -tf /tmp/debian-chroot.tar | sort | diff -u tar1.txt -

149
tests/dev-ptmx
View file

@ -1,149 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
if [ {{ MODE }} != unshare ] && [ {{ MODE }} != root ]; then
echo "test requires root or unshare mode" >&2
exit 1
fi
prefix=
if [ "$(id -u)" -eq 0 ] && [ "{{ MODE }}" != "root" ] && [ "{{ MODE }}" != "auto" ]; then
if ! id "${SUDO_USER:-user}" >/dev/null 2>&1; then
if [ ! -e /mmdebstrap-testenv ]; then
echo "this test modifies the system and should only be run inside a container" >&2
exit 1
fi
useradd --home-dir "/home/${SUDO_USER:-user}" --create-home "${SUDO_USER:-user}"
fi
prefix="runuser -u ${SUDO_USER:-user} --"
fi
# this mimics what apt does in apt-pkg/deb/dpkgpm.cc/pkgDPkgPM::StartPtyMagic()
cat >/tmp/test.c <<'END'
#define _GNU_SOURCE
#include <stdlib.h>
#include <fcntl.h>
#include <termios.h>
#include <unistd.h>
#include <stdio.h>
#include <sys/ioctl.h>
#include <signal.h>
int main() {
int ret;
int fd = posix_openpt(O_RDWR | O_NOCTTY);
if (fd < 0) {
perror("posix_openpt");
return 1;
}
char buf[64]; // 64 is used by apt
ret = ptsname_r(fd, buf, sizeof(buf));
if (ret != 0) {
perror("ptsname_r");
return 1;
}
ret = grantpt(fd);
if (ret == -1) {
perror("grantpt");
return 1;
}
struct termios origtt;
ret = tcgetattr(STDIN_FILENO, &origtt);
if (ret != 0) {
perror("tcgetattr1");
return 1;
}
struct termios tt;
ret = tcgetattr(STDOUT_FILENO, &tt);
if (ret != 0) {
perror("tcgetattr2");
return 1;
}
struct winsize win;
ret = ioctl(STDOUT_FILENO, TIOCGWINSZ, &win);
if (ret < 0) {
perror("ioctl stdout TIOCGWINSZ");
return 1;
}
ret = ioctl(fd, TIOCSWINSZ, &win);
if (ret < 0) {
perror("ioctl fd TIOCGWINSZ");
return 1;
}
ret = tcsetattr(fd, TCSANOW, &tt);
if (ret != 0) {
perror("tcsetattr1");
return 1;
}
cfmakeraw(&tt);
tt.c_lflag &= ~ECHO;
tt.c_lflag |= ISIG;
sigset_t sigmask;
sigset_t sigmask_old;
ret = sigemptyset(&sigmask);
if (ret != 0) {
perror("sigemptyset");
return 1;
}
ret = sigaddset(&sigmask, SIGTTOU);
if (ret != 0) {
perror("sigaddset");
return 1;
}
ret = sigprocmask(SIG_BLOCK,&sigmask, &sigmask_old);
if (ret != 0) {
perror("sigprocmask1");
return 1;
}
ret = tcsetattr(STDIN_FILENO, TCSAFLUSH, &tt);
if (ret != 0) {
perror("tcsetattr2");
return 1;
}
ret = sigprocmask(SIG_BLOCK,&sigmask_old, NULL);
if (ret != 0) {
perror("sigprocmask2");
return 1;
}
ret = tcsetattr(STDIN_FILENO, TCSAFLUSH, &origtt);
if (ret != 0) {
perror("tcsetattr3");
return 1;
}
return 0;
}
END
# use script to create a fake tty
# run all tests as root and as a normal user (the latter requires ptmxmode=666)
script -qfec "$prefix {{ CMD }} --mode={{ MODE }} --variant=apt \
--include=gcc,libc6-dev,python3,passwd \
--customize-hook='chroot \"\$1\" useradd --home-dir /home/user --create-home user' \
--customize-hook='chroot \"\$1\" python3 -c \"import pty; print(pty.openpty())\"' \
--customize-hook='chroot \"\$1\" runuser -u user -- python3 -c \"import pty; print(pty.openpty())\"' \
--customize-hook='chroot \"\$1\" script -c \"echo foobar\"' \
--customize-hook='chroot \"\$1\" runuser -u user -- env --chdir=/home/user script -c \"echo foobar\"' \
--customize-hook='chroot \"\$1\" apt-get install --yes doc-debian 2>&1 | tee \"\$1\"/tmp/log' \
--customize-hook=\"copy-in /tmp/test.c /tmp\" \
--customize-hook='chroot \"\$1\" gcc /tmp/test.c -o /tmp/test' \
--customize-hook='chroot \"\$1\" /tmp/test' \
--customize-hook='chroot \"\$1\" runuser -u user -- /tmp/test' \
--customize-hook='rm \"\$1\"/tmp/test \"\$1\"/tmp/test.c' \
--customize-hook=\"copy-out /tmp/log /tmp\" \
{{ DIST }} /dev/null {{ MIRROR }}" /dev/null
fail=0
[ -r /tmp/log ] || fail=1
grep '^E:' /tmp/log && fail=1
grep 'Can not write log' /tmp/log && fail=1
grep 'posix_openpt' /tmp/log && fail=1
grep 'No such file or directory' /tmp/log && fail=1
if [ $fail -eq 1 ]; then
echo "apt failed to write log:" >&2
cat /tmp/log >&2
exit 1
fi
rm /tmp/test.c /tmp/log

12
tests/directory-ending-in-tar
View file

@ -1,12 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
[ "$(whoami)" = "root" ]
trap "rm -rf /tmp/debian-chroot.tar" EXIT INT TERM
{{ CMD }} --mode={{ MODE }} --variant=apt --format=directory {{ DIST }} /tmp/debian-chroot.tar {{ MIRROR }}
ftype=$(stat -c %F /tmp/debian-chroot.tar)
if [ "$ftype" != directory ]; then
echo "expected directory but got: $ftype" >&2
exit 1
fi
tar -C /tmp/debian-chroot.tar --one-file-system -c . | tar -t | sort | diff -u tar1.txt -

13
tests/dist-using-codename
View file

@ -1,13 +0,0 @@
#!/bin/sh
#
# make sure that using codenames works https://bugs.debian.org/1003191
set -eu
export LC_ALL=C.UTF-8
trap "rm -f InRelease; rm -rf /tmp/debian-chroot.tar /tmp/expected" EXIT INT TERM
/usr/lib/apt/apt-helper download-file "{{ MIRROR }}/dists/{{ DIST }}/InRelease" InRelease
codename=$(awk '/^Codename: / { print $2; }' InRelease)
{{ CMD }} --mode={{ MODE }} --variant=apt "$codename" /tmp/debian-chroot.tar {{ MIRROR }}
echo "deb {{ MIRROR }} $codename main" >/tmp/expected
tar --to-stdout --extract --file /tmp/debian-chroot.tar ./etc/apt/sources.list \
| diff -u /tmp/expected -

14
tests/dpkgopt
View file

@ -1,14 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
trap "rm -rf /tmp/debian-chroot; rm -f /tmp/config" EXIT INT TERM
echo no-pager >/tmp/config
{{ CMD }} --mode=root --variant=apt --dpkgopt="path-exclude=/usr/share/doc/*" --dpkgopt=/tmp/config --dpkgopt="path-include=/usr/share/doc/dpkg/copyright" {{ DIST }} /tmp/debian-chroot {{ MIRROR }}
printf 'path-exclude=/usr/share/doc/*\nno-pager\npath-include=/usr/share/doc/dpkg/copyright\n' | cmp /tmp/debian-chroot/etc/dpkg/dpkg.cfg.d/99mmdebstrap -
rm /tmp/debian-chroot/etc/dpkg/dpkg.cfg.d/99mmdebstrap
tar -C /tmp/debian-chroot --one-file-system -c . | tar -t | sort >tar2.txt
{
grep -v '^./usr/share/doc/.' tar1.txt
echo ./usr/share/doc/dpkg/
echo ./usr/share/doc/dpkg/copyright
} | sort | diff -u - tar2.txt

48
tests/eatmydata-via-hook-dir
View file

@ -1,48 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
cat <<SCRIPT >/tmp/checkeatmydata.sh
#!/bin/sh
set -exu
cat << EOF | diff - "\$1"/usr/bin/dpkg
#!/bin/sh
exec /usr/bin/eatmydata /usr/bin/dpkg.distrib "\\\$@"
EOF
[ -e "\$1"/usr/bin/eatmydata ]
SCRIPT
chmod +x /tmp/checkeatmydata.sh
# first four bytes: magic
elfheader='\177ELF'
# fifth byte: bits
case "$(dpkg-architecture -qDEB_HOST_ARCH_BITS)" in
32) elfheader="$elfheader\\001" ;;
64) elfheader="$elfheader\\002" ;;
*)
echo "bits not supported"
exit 1
;;
esac
# sixth byte: endian
case "$(dpkg-architecture -qDEB_HOST_ARCH_ENDIAN)" in
little) elfheader="$elfheader\\001" ;;
big) elfheader="$elfheader\\002" ;;
*)
echo "endian not supported"
exit 1
;;
esac
# seventh and eigth byte: elf version (1) and abi (unset)
elfheader="$elfheader\\001\\000"
{{ CMD }} --mode=root --variant=apt \
--customize-hook=/tmp/checkeatmydata.sh \
--essential-hook=/tmp/checkeatmydata.sh \
--extract-hook='printf "'"$elfheader"'" | cmp --bytes=8 - "$1"/usr/bin/dpkg' \
--hook-dir=./hooks/eatmydata \
--customize-hook='printf "'"$elfheader"'" | cmp --bytes=8 - "$1"/usr/bin/dpkg' \
{{ DIST }} /tmp/debian-chroot {{ MIRROR }}
tar -C /tmp/debian-chroot --one-file-system -c . \
| tar -t \
| sort \
| diff -u tar1.txt -
rm /tmp/checkeatmydata.sh
rm -r /tmp/debian-chroot

8
tests/empty-sources.list
View file

@ -1,8 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
trap "rm -f /tmp/debian-chroot.tar" EXIT INT TERM
printf '' | {{ CMD }} --mode={{ MODE }} --variant=apt \
--setup-hook='test -e "$1"/etc/apt/sources.list || echo "deb {{ MIRROR }} {{ DIST }} main" > "$1"/etc/apt/sources.list' \
{{ DIST }} /tmp/debian-chroot.tar -
tar -tf /tmp/debian-chroot.tar | sort | diff -u tar1.txt -

13
tests/empty-suite
View file

@ -1,13 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
export SOURCE_DATE_EPOCH={{ SOURCE_DATE_EPOCH }}
trap "rm -f /tmp/debian-chroot.tar" EXIT INT TERM
{{ CMD }} --variant={{ VARIANT }} --verbose \
--setup-hook='echo deb {{ MIRROR }} {{ DIST }} main >> "$1"/etc/apt/sources.list' \
'' /tmp/debian-chroot.tar
cmp ./cache/mmdebstrap-{{ DIST }}-{{ VARIANT }}.tar /tmp/debian-chroot.tar \
|| diffoscope ./cache/mmdebstrap-{{ DIST }}-{{ VARIANT }}.tar /tmp/debian-chroot.tar

12
tests/error-if-stdout-is-tty
View file

@ -1,12 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
ret=0
script -qfec "{{ CMD }} --mode={{ MODE }} --variant=apt {{ DIST }} - {{ MIRROR }}" /dev/null || ret=$?
if [ "$ret" = 0 ]; then
echo expected failure but got exit $ret >&2
exit 1
fi

20
tests/essential-hook
View file

@ -1,20 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
trap "rm -rf /tmp/debian-chroot; rm -f /tmp/essential.sh" EXIT INT TERM
cat <<'SCRIPT' >/tmp/essential.sh
#!/bin/sh
echo tzdata tzdata/Zones/Europe select Berlin | chroot "$1" debconf-set-selections
SCRIPT
chmod +x /tmp/essential.sh
{{ CMD }} --mode=root --variant=apt --include=tzdata --essential-hook='echo tzdata tzdata/Areas select Europe | chroot "$1" debconf-set-selections' --essential-hook=/tmp/essential.sh {{ DIST }} /tmp/debian-chroot {{ MIRROR }}
[ "$(readlink /tmp/debian-chroot/etc/localtime)" = "/usr/share/zoneinfo/Europe/Berlin" ]
tar -C /tmp/debian-chroot --one-file-system -c . | tar -t | sort \
| grep -v '^./etc/localtime' \
| grep -v '^./etc/timezone' \
| grep -v '^./usr/sbin/tzconfig' \
| grep -v '^./usr/share/doc/tzdata' \
| grep -v '^./usr/share/lintian/overrides/tzdata' \
| grep -v '^./usr/share/zoneinfo' \
| grep -v '^./var/lib/dpkg/info/tzdata.' \
| diff -u tar1.txt -

9
tests/existing-directory-with-lost-found
View file

@ -1,9 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
trap "rm -rf /tmp/debian-chroot" EXIT INT TERM
mkdir /tmp/debian-chroot
mkdir /tmp/debian-chroot/lost+found
{{ CMD }} --mode=root --variant=apt {{ DIST }} /tmp/debian-chroot {{ MIRROR }}
rmdir /tmp/debian-chroot/lost+found
tar -C /tmp/debian-chroot --one-file-system -c . | tar -t | sort | diff -u tar1.txt -

7
tests/existing-empty-directory
View file

@ -1,7 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
trap "rm -rf /tmp/debian-chroot" EXIT INT TERM
mkdir /tmp/debian-chroot
{{ CMD }} --mode=root --variant=apt {{ DIST }} /tmp/debian-chroot {{ MIRROR }}
tar -C /tmp/debian-chroot --one-file-system -c . | tar -t | sort | diff -u tar1.txt -

13
tests/fail-installing-to-existing-file
View file

@ -1,13 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
trap "rm -f /tmp/exists" EXIT INT TERM
touch /tmp/exists
ret=0
{{ CMD }} --mode={{ MODE }} --variant=apt {{ DIST }} /tmp/exists {{ MIRROR }} || ret=$?
if [ "$ret" = 0 ]; then
echo expected failure but got exit $ret >&2
exit 1
fi

13
tests/fail-installing-to-non-empty-lost-found
View file

@ -1,13 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
trap "rm /tmp/debian-chroot/lost+found/exists; rmdir /tmp/debian-chroot/lost+found /tmp/debian-chroot" EXIT INT TERM
mkdir /tmp/debian-chroot
mkdir /tmp/debian-chroot/lost+found
touch /tmp/debian-chroot/lost+found/exists
ret=0
{{ CMD }} --mode={{ MODE }} --variant=apt {{ DIST }} /tmp/debian-chroot {{ MIRROR }} || ret=$?
if [ "$ret" = 0 ]; then
echo expected failure but got exit $ret >&2
exit 1
fi

13
tests/fail-installing-to-non-empty-target-directory
View file

@ -1,13 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
trap "rmdir /tmp/debian-chroot/lost+found; rm /tmp/debian-chroot/exists; rmdir /tmp/debian-chroot" EXIT INT TERM
mkdir /tmp/debian-chroot
mkdir /tmp/debian-chroot/lost+found
touch /tmp/debian-chroot/exists
ret=0
{{ CMD }} --mode={{ MODE }} --variant=apt {{ DIST }} /tmp/debian-chroot {{ MIRROR }} || ret=$?
if [ "$ret" = 0 ]; then
echo expected failure but got exit $ret >&2
exit 1
fi

9
tests/fail-installing-to-root
View file

@ -1,9 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
ret=0
{{ CMD }} --mode={{ MODE }} --variant=apt {{ DIST }} / {{ MIRROR }} || ret=$?
if [ "$ret" = 0 ]; then
echo expected failure but got exit $ret >&2
exit 1
fi

9
tests/fail-with-missing-lz4
View file

@ -1,9 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
ret=0
{{ CMD }} --mode={{ MODE }} --variant=apt {{ DIST }} /tmp/debian-chroot.tar.lz4 {{ MIRROR }} || ret=$?
if [ "$ret" = 0 ]; then
echo expected failure but got exit $ret >&2
exit 1
fi

12
tests/fail-with-path-with-quotes
View file

@ -1,12 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
trap 'rm -rf /tmp/quoted\"path' EXIT INT TERM
ret=0
{{ CMD }} --mode={{ MODE }} --variant=apt {{ DIST }} /tmp/quoted\"path {{ MIRROR }} || ret=$?
if [ "$ret" = 0 ]; then
echo expected failure but got exit $ret >&2
exit 1
fi

16
tests/fail-without-etc-subuid
View file

@ -1,16 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
if [ ! -e /mmdebstrap-testenv ]; then
echo "this test modifies the system and should only be run inside a container" >&2
exit 1
fi
useradd --home-dir /home/user --create-home user
rm /etc/subuid
ret=0
runuser -u user -- {{ CMD }} --mode=unshare --variant=apt {{ DIST }} /tmp/debian-chroot {{ MIRROR }} || ret=$?
if [ "$ret" = 0 ]; then
echo expected failure but got exit $ret >&2
exit 1
fi
[ ! -e /tmp/debian-chroot ]

17
tests/fail-without-username-in-etc-subuid
View file

@ -1,17 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
if [ ! -e /mmdebstrap-testenv ]; then
echo "this test modifies the system and should only be run inside a container" >&2
exit 1
fi
useradd --home-dir /home/user --create-home user
awk -F: '$1!="user"' /etc/subuid >/etc/subuid.tmp
mv /etc/subuid.tmp /etc/subuid
ret=0
runuser -u user -- {{ CMD }} --mode=unshare --variant=apt {{ DIST }} /tmp/debian-chroot {{ MIRROR }} || ret=$?
if [ "$ret" = 0 ]; then
echo expected failure but got exit $ret >&2
exit 1
fi
[ ! -e /tmp/debian-chroot ]

10
tests/failing-customize-hook
View file

@ -1,10 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
ret=0
{{ CMD }} --mode=root --variant=apt --customize-hook='chroot "$1" sh -c "exit 1"' {{ DIST }} /tmp/debian-chroot {{ MIRROR }} || ret=$?
rm -r /tmp/debian-chroot
if [ "$ret" = 0 ]; then
echo expected failure but got exit $ret >&2
exit 1
fi

13
tests/file-mirror
View file

@ -1,13 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
if [ ! -e /mmdebstrap-testenv ]; then
echo "this test requires the cache directory to be mounted on /mnt and should only be run inside a container" >&2
exit 1
fi
{{ CMD }} --mode={{ MODE }} --variant=apt \
--setup-hook='mkdir -p "$1"/mnt/cache/debian; mount -o ro,bind /mnt/cache/debian "$1"/mnt/cache/debian' \
--customize-hook='umount "$1"/mnt/cache/debian; rmdir "$1"/mnt/cache/debian "$1"/mnt/cache' \
{{ DIST }} /tmp/debian-chroot.tar "deb file:///mnt/cache/debian {{ DIST }} main"
tar -tf /tmp/debian-chroot.tar | sort | diff -u tar1.txt -
rm /tmp/debian-chroot.tar

20
tests/file-mirror-automount-hook
View file

@ -1,20 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
if [ ! -e /mmdebstrap-testenv ]; then
echo "this test requires the cache directory to be mounted on /mnt and should only be run inside a container" >&2
exit 1
fi
if [ "$(id -u)" -eq 0 ] && ! id -u user >/dev/null 2>&1; then
useradd --home-dir /home/user --create-home user
fi
prefix=
[ "$(id -u)" -eq 0 ] && [ "{{ MODE }}" != "root" ] && prefix="runuser -u user --"
[ "{{ MODE }}" = "fakechroot" ] && prefix="$prefix fakechroot fakeroot"
$prefix {{ CMD }} --mode={{ MODE }} --variant=apt \
--hook-dir=./hooks/file-mirror-automount \
--customize-hook='[ ! -e "$1"/mnt/cache/debian/ ] || rmdir "$1"/mnt/cache/debian/' \
--customize-hook='rmdir "$1"/mnt/cache' \
{{ DIST }} /tmp/debian-chroot.tar "deb file:///mnt/cache/debian {{ DIST }} main"
tar -tf /tmp/debian-chroot.tar | sort | diff -u tar1.txt -
rm /tmp/debian-chroot.tar

6
tests/help
View file

@ -1,6 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
# we redirect to /dev/null instead of using --quiet to not cause a broken pipe
# when grep exits before mmdebstrap was able to write all its output
{{ CMD }} --help | grep --fixed-strings 'mmdebstrap [OPTION...] [SUITE [TARGET [MIRROR...]]]' >/dev/null

49
tests/hook-directory
View file

@ -1,49 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
for h in hookA hookB; do
mkdir /tmp/$h
for s in setup extract essential customize; do
cat <<SCRIPT >/tmp/$h/${s}00.sh
#!/bin/sh
echo $h/${s}00 >> "\$1/$s"
SCRIPT
chmod +x /tmp/$h/${s}00.sh
cat <<SCRIPT >/tmp/$h/${s}01.sh
echo $h/${s}01 >> "\$1/$s"
SCRIPT
chmod +x /tmp/$h/${s}01.sh
done
done
{{ CMD }} --mode=root --variant=apt \
--setup-hook='echo cliA/setup >> "$1"/setup' \
--extract-hook='echo cliA/extract >> "$1"/extract' \
--essential-hook='echo cliA/essential >> "$1"/essential' \
--customize-hook='echo cliA/customize >> "$1"/customize' \
--hook-dir=/tmp/hookA \
--setup-hook='echo cliB/setup >> "$1"/setup' \
--extract-hook='echo cliB/extract >> "$1"/extract' \
--essential-hook='echo cliB/essential >> "$1"/essential' \
--customize-hook='echo cliB/customize >> "$1"/customize' \
--hook-dir=/tmp/hookB \
--setup-hook='echo cliC/setup >> "$1"/setup' \
--extract-hook='echo cliC/extract >> "$1"/extract' \
--essential-hook='echo cliC/essential >> "$1"/essential' \
--customize-hook='echo cliC/customize >> "$1"/customize' \
{{ DIST }} /tmp/debian-chroot {{ MIRROR }}
printf "cliA/setup\nhookA/setup00\nhookA/setup01\ncliB/setup\nhookB/setup00\nhookB/setup01\ncliC/setup\n" | diff -u - /tmp/debian-chroot/setup
printf "cliA/extract\nhookA/extract00\nhookA/extract01\ncliB/extract\nhookB/extract00\nhookB/extract01\ncliC/extract\n" | diff -u - /tmp/debian-chroot/extract
printf "cliA/essential\nhookA/essential00\nhookA/essential01\ncliB/essential\nhookB/essential00\nhookB/essential01\ncliC/essential\n" | diff -u - /tmp/debian-chroot/essential
printf "cliA/customize\nhookA/customize00\nhookA/customize01\ncliB/customize\nhookB/customize00\nhookB/customize01\ncliC/customize\n" | diff -u - /tmp/debian-chroot/customize
for s in setup extract essential customize; do
rm /tmp/debian-chroot/$s
done
tar -C /tmp/debian-chroot --one-file-system -c . | tar -t | sort | diff -u tar1.txt -
for h in hookA hookB; do
for s in setup extract essential customize; do
rm /tmp/$h/${s}00.sh
rm /tmp/$h/${s}01.sh
done
rmdir /tmp/$h
done
rm -r /tmp/debian-chroot

42
tests/i386-which-can-be-executed-without-qemu
View file

@ -1,42 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
if [ ! -e /mmdebstrap-testenv ]; then
echo "this test modifies the system and should only be run inside a container" >&2
exit 1
fi
# remove qemu just to be sure
apt-get remove --yes qemu-user-binfmt binfmt-support qemu-user
{{ CMD }} --mode={{ MODE }} --variant=apt --architectures=i386 {{ DIST }} /tmp/debian-chroot.tar {{ MIRROR }}
# we ignore differences between architectures by ignoring some files
# and renaming others
{
tar -tf /tmp/debian-chroot.tar \
| grep -v '^\./usr/bin/i386$' \
| grep -v '^\./usr/lib/ld-linux\.so\.2$' \
| grep -v '^\./usr/lib/i386-linux-gnu/ld-linux\.so\.2$' \
| grep -v '^\./usr/lib/gcc/i686-linux-gnu/$' \
| grep -v '^\./usr/lib/gcc/i686-linux-gnu/[0-9]\+/$' \
| grep -v '^\./usr/share/man/man8/i386\.8\.gz$' \
| grep -v '^\./usr/share/doc/[^/]\+/changelog\(\.Debian\)\?\.i386\.gz$' \
| sed 's/i386-linux-gnu/x86_64-linux-gnu/' \
| sed 's/i386/amd64/' \
| sed 's/\/stubs-32.ph$/\/stubs-64.ph/'
} | sort >tar2.txt
{
grep <tar1.txt -v '^\./usr/bin/i386$' \
| grep -v '^\./usr/bin/x86_64$' \
| grep -v '^\./usr/lib32/$' \
| grep -v '^\./lib32$' \
| grep -v '^\./lib64$' \
| grep -v '^\./usr/lib64/$' \
| grep -v '^\./usr/lib64/ld-linux-x86-64\.so\.2$' \
| grep -v '^\./usr/lib/gcc/x86_64-linux-gnu/$' \
| grep -v '^\./usr/lib/gcc/x86_64-linux-gnu/[0-9]\+/$' \
| grep -v '^\./usr/lib/x86_64-linux-gnu/ld-linux-x86-64\.so\.2$' \
| grep -v '^\./usr/lib/x86_64-linux-gnu/libmvec\.so\.1$' \
| grep -v '^\./usr/share/doc/[^/]\+/changelog\(\.Debian\)\?\.amd64\.gz$' \
| grep -v '^\./usr/share/man/man8/i386\.8\.gz$' \
| grep -v '^\./usr/share/man/man8/x86_64\.8\.gz$'
} | sort | diff -u - tar2.txt >&2
rm /tmp/debian-chroot.tar

11
tests/include
View file

@ -1,11 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
trap "rm -rf /tmp/debian-chroot" EXIT INT TERM
{{ CMD }} --mode=root --variant=apt --include=doc-debian {{ DIST }} /tmp/debian-chroot {{ MIRROR }}
rm /tmp/debian-chroot/usr/share/doc-base/doc-debian.debian-*
rm -r /tmp/debian-chroot/usr/share/doc/debian
rm -r /tmp/debian-chroot/usr/share/doc/doc-debian
rm /tmp/debian-chroot/var/lib/dpkg/info/doc-debian.list
rm /tmp/debian-chroot/var/lib/dpkg/info/doc-debian.md5sums
tar -C /tmp/debian-chroot --one-file-system -c . | tar -t | sort | diff -u tar1.txt -

40
tests/include-deb-file
View file

@ -1,40 +0,0 @@
#!/bin/sh
set -eu
export LC_ALL=C.UTF-8
trap "rm -rf /tmp/dummypkg.deb /tmp/dummypkg" EXIT INT TERM
prefix=
if [ "$(id -u)" -eq 0 ] && [ "{{ MODE }}" != "root" ] && [ "{{ MODE }}" != "auto" ]; then
if ! id "${SUDO_USER:-user}" >/dev/null 2>&1; then
if [ ! -e /mmdebstrap-testenv ]; then
echo "this test modifies the system and should only be run inside a container" >&2
exit 1
fi
useradd --home-dir "/home/${SUDO_USER:-user}" --create-home "${SUDO_USER:-user}"
fi
prefix="runuser -u ${SUDO_USER:-user} --"
fi
# instead of obtaining a .deb from our cache, we create a new package because
# otherwise apt might decide to download the package with the same name and
# version from the cache instead of using the local .deb
mkdir -p /tmp/dummypkg/DEBIAN
cat <<END >"/tmp/dummypkg/DEBIAN/control"
Package: dummypkg
Priority: optional
Section: oldlibs
Maintainer: Johannes Schauer Marin Rodrigues <josch@debian.org>
Architecture: all
Multi-Arch: foreign
Source: dummypkg
Version: 1
Description: dummypkg
END
dpkg-deb --build "/tmp/dummypkg" "/tmp/dummypkg.deb"
$prefix {{ CMD }} --mode={{ MODE }} --variant=apt --include="/tmp/dummypkg.deb" \
--hook-dir=./hooks/file-mirror-automount \
--customize-hook='chroot "$1" dpkg-query -W -f="\${Status}\n" dummypkg | grep "^install ok installed$"' \
{{ DIST }} /dev/null {{ MIRROR }}

Some files were not shown because too many files have changed in this diff Show more