Local package cache?

darkpenguin commented

2024-04-22 10:51:46 +00:00

Hi!
debootstrap has an option to cache downloaded packages:
--cache-dir=/var/cache/apt/archives
This caches downloaded packages on the host (not in the "guest"), which speeds up consecutive bootstrap runs.
Does mmdebstrap has some way to achieve a similar effect?

Hi! `debootstrap` has an option to cache downloaded packages: `--cache-dir=/var/cache/apt/archives` This caches downloaded packages on the host (not in the "guest"), which speeds up consecutive bootstrap runs. Does `mmdebstrap` has some way to achieve a similar effect?

josch commented

2024-04-22 12:50:56 +00:00

Owner

What the debootstrap --cache-dir option does is a layer violation. You have at least two options to do something similar in mmdebstrap:

add a similar layer violation in the form of hook scripts which copy-in your debs before apt starts the download and which copy out the debs at the end of the run
use a http cacher like apt-cacher or apt-cacher-ng

If you go the first route, feel free to share your scripts. If they are general-purpose enough I can include them in the hooks directory so that others can make use of them as well.

What the debootstrap `--cache-dir` option does is a layer violation. You have at least two options to do something similar in mmdebstrap: 1. add a similar layer violation in the form of hook scripts which copy-in your debs before apt starts the download and which copy out the debs at the end of the run 2. use a http cacher like apt-cacher or apt-cacher-ng If you go the first route, feel free to share your scripts. If they are general-purpose enough I can include them in the `hooks` directory so that others can make use of them as well.

darkpenguin commented

2024-04-22 14:02:55 +00:00

Author

Hm, instead of copy-in and copy-out, I would just bind-mount the directory. But then it has to be unmounted if the process is interrupted due to an error or a SIGINT. Oh, I see the docs; if at some point it installs everything in that directory, then this wouldn't work...

OK, that's fine, I was just wondering if there is already a solution I couldn't find. (Maybe the --cache-dir option should be added to the list of "debootstrap options that don't exist" in the readme file?)

Hm, instead of copy-in and copy-out, I would just bind-mount the directory. But then it has to be unmounted if the process is interrupted due to an error or a SIGINT. Oh, I see the docs; if at some point it installs everything in that directory, then this wouldn't work... OK, that's fine, I was just wondering if there is already a solution I couldn't find. (Maybe the `--cache-dir` option should be added to the list of "debootstrap options that don't exist" in the readme file?)

👍 1

josch commented

2024-04-22 15:25:50 +00:00

Owner

No, mmdebstrap creates its own mount namespace, so everything gets unmounted automatically even if you should forget to unmount or if the process dies early. Have a look in /usr/share/mmdebstrap/hooks/file-mirror-automount for inspiration how an external file:// repository can be auto-mounted.

Also, you really should not mess with /var/lib/apt/archives. That directory is an implementation detail of apt and only apt should operate on it.

The best solution is really to set up a local apt proxy. There are various solutions to that problem so mmdebstrap does not have to add its own thing.

No, mmdebstrap creates its own mount namespace, so everything gets unmounted automatically even if you should forget to unmount or if the process dies early. Have a look in `/usr/share/mmdebstrap/hooks/file-mirror-automount` for inspiration how an external `file://` repository can be auto-mounted. Also, you *really* should not mess with `/var/lib/apt/archives`. That directory is an implementation detail of apt and only apt should operate on it. The best solution is really to set up a local apt proxy. There are various solutions to that problem so mmdebstrap does not have to add its own thing.

petervuto referenced this issue

2024-09-20 09:35:23 +00:00

Unknown option: cache-dir #45

darkpenguin commented

2026-05-06 03:35:42 +00:00

Author

I'm fighting with the same problem... oh wait, it was me who opened this in the first place. 😄

I gave up before, but now I want ti try again. The goal is to cache packages so that they are not downloaded on consequent invocations.

I use a --setup-hook to bind-mount a directory into /var/lib/apt/archives.
To prevent mmdebstrap from deleting all packages in that directory, I add --skip=cleanup/apt to its arguments.
I also add --aptopt='APT::Keep-Downloaded-Packages true' and --aptopt='Binary::apt::APT::Keep-Downloaded-Packages true' for good measure, so that apt definitely, absolutely, never removes packages.

What behaviour do I observe:

I: downloading packages with apt... - downloads ~25M worth of packages into /var/lib/apt/archives
I: extracting archives...
I: installing essential packages...
After this step, those downloaded packages are deleted. Only those. So, the next time they have to be downloaded again.
I: installing remaining packages inside the chroot... - no, it's not "installing", it's "downloading" ~25M more of different packages. And these are not deleted! And the next time, they are not re-downloading - they are correctly picked up.

So, the question is: how can I prevent deleting and re-downloading those first files every time? I guess they are downloaded "differently" in some way?

UPDATE: I've read the docs before, but only now that I'm reading them again in more detail, I see the --skip=essential/unlink flag. 🥲

I'm fighting with the same problem... oh wait, it was me who opened this in the first place. 😄 I gave up before, but now I want ti try again. The goal is to cache packages so that they are not downloaded on consequent invocations. - I use a `--setup-hook` to bind-mount a directory into `/var/lib/apt/archives`. - To prevent mmdebstrap from deleting all packages in that directory, I add `--skip=cleanup/apt` to its arguments. - I also add `--aptopt='APT::Keep-Downloaded-Packages true'` and `--aptopt='Binary::apt::APT::Keep-Downloaded-Packages true'` for good measure, so that apt definitely, absolutely, never removes packages. What behaviour do I observe: - `I: downloading packages with apt...` - downloads ~25M worth of packages into `/var/lib/apt/archives` - `I: extracting archives...` - `I: installing essential packages...` - After this step, those downloaded packages are deleted. Only those. So, the next time they have to be downloaded again. - `I: installing remaining packages inside the chroot...` - no, it's not "installing", it's "downloading" ~25M more of different packages. And these are not deleted! And the next time, they are not re-downloading - they are correctly picked up. So, the question is: how can I prevent deleting and re-downloading those first files every time? I guess they are downloaded "differently" in some way? UPDATE: I've read the [docs](https://gitlab.mister-muffin.de/josch/mmdebstrap/wiki#operation) before, but only now that I'm reading them again in more detail, I see the `--skip=essential/unlink` flag. 🥲

Rows
Columns

Local package cache? #42