josch/multistrap
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

* Add --simulate mode for cascading configuration testing. * Split out the POD - not needed in the runtime script.

Browse source 
git-svn-id: http://emdebian.org/svn/current@7107 563faec7-e20c-0410-992a-a66f704d0ccd
This commit is contained in:
codehelp 2010-04-17 21:24:53 +00:00
parent 70e000ad29
commit 84094ca16c
4 changed files with 377 additions and 292 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
debian/changelog vendored
View file

@ -1,3 +1,10 @@
multistrap (2.1.1) experimental; urgency=low
* Add --simulate mode for cascading configuration testing.
* Split out the POD - not needed in the runtime script.
-- Neil Williams <codehelp@debian.org> Sat, 17 Apr 2010 22:24:43 +0100
multistrap (2.1.0) experimental; urgency=low
* Experimental branch to replace pbuilder support in Crush.

2
debian/multistrap.install vendored
View file

@ -10,5 +10,5 @@ examples/config.sh ./usr/share/doc/multistrap/examples/
check-deps.sh ./usr/share/multistrap/
device-table.pl ./usr/share/multistrap/
update-rc.d ./usr/share/multistrap/
crosschroot.conf ./usr/share/multistrap/
bash/multistrap ./etc/bash_completion.d/
cross/*.conf ./usr/share/multistrap/

337
multistrap
View file

@ -30,7 +30,8 @@ use vars qw/ $progname $ourversion $dstrap $extra @aptsources $mirror
@e $sourcesname $libdir $dpkgdir @debootstrap %suites %components $chk
$component $repo @dirs @touch %sources $section %keys $host $key $value
$type $file $config $tidy $noauth $keyring %keyrings $deflist $cfgdir
@extrapkgs @includes %source $setupsh $configsh $omitrequired /;
@extrapkgs @includes %source $setupsh $configsh $omitrequired $dryrun
/;
setlocale(LC_MESSAGES, "");
textdomain("multistrap");
@ -70,6 +71,9 @@ while( @ARGV ) {
elsif (/^(--no-auth)$/) {
$noauth++;
}
elsif (/^(--dry-run|--simulate)$/) {
$dryrun++;
}
else {
die "$progname: "._g("Unknown option")." $_.\n";
}
@ -80,15 +84,26 @@ die ($msg)
$cfgdir=dirname($file);
cascade($file);
foreach my $inc (@includes)
{
$chk = `realpath $cfgdir/$inc 2>/dev/null`;
next if ($chk =~ /^\n?$/);
chomp ($chk);
# look for the full filepath or try same directory as current conf.
if (not -f $inc )
{
$chk = `realpath $cfgdir/$inc 2>/dev/null`;
next if ($chk =~ /^\n?$/);
chomp ($chk);
}
else
{
$chk = $inc;
}
printf (_g("%s %s using %s\n"), $progname, $ourversion, $chk);
cascade($chk);
}
&dump_config if (defined $dryrun);
# Translators: fields are: programname, versionstring, configfile.
printf (_g("%s %s using %s\n"), $progname, $ourversion, $file);
$host = `dpkg-architecture -qDEB_BUILD_ARCH`;
@ -316,8 +331,9 @@ print "apt-get -y $config_str install $str\n";
$retval = system ("apt-get -y $config_str install $str");
die (sprintf (_g("apt download failed. Exit value: %d\n"),($retval/256)))
if ($retval != 0);
system ("$setupsh") if (defined $setupsh);
&force_unpack if ($unpack eq "true");
system ("cp $configsh $dir/") if (defined $configsh);
system ("touch ${dir}${libdir}lists/lock");
&native if (not defined ($foreign));
&add_extra_packages;
@ -717,6 +733,8 @@ Options:
-d|--dir PATH: override the configuration file directory.
--no-auth: do not use Secure Apt for any repositories
--tidy-up: remove apt cache data and downloaded archives.
--dry-run: output the configuration and exit
--simulate: output the configuration and exit
-?|-h|--help: print this usage message and exit
--version: print this usage message and exit
@ -799,10 +817,10 @@ sub cascade
and (not defined $unpack));
$configsh = lc($keys{$section}{'configscript'})
if (defined $keys{$section}{'configscript'}
and (not defined $configsh));
and (not defined $configsh) and (-x $configsh));
$setupsh = lc($keys{$section}{'setupscript'})
if (defined $keys{$section}{'setupscript'}
and (not defined $setupsh));
and (not defined $setupsh) and (-x $setupsh));
$omitrequired = lc($keys{$section}{'omitrequired'})
if (defined $keys{$section}{'omitrequired'}
and (not defined $omitrequired));
@ -847,287 +865,24 @@ sub _g {
return gettext(shift);
}
=pod
=head1 Name
multistrap - extends debootstrap for multiple repository support
=head1 Synopsis
multistrap [-a ARCH] [-d DIR] -f CONFIG_FILE
multistrap -?|-h|--help|--version
=head1 Options
(These options can also be set in the configuration file.)
--tidy-up - remove apt cache data, downloaded Packages files and
the apt package cache. Same as cleanup=true.
--no-auth - allow the use of unauthenticated repositories. Same
as noauth=true
=head1 Description
multistrap provides a debootstrap-like method based on apt and
extended to provide support for multiple repositories, using a
configuration file to specify the relevant suites, architecture,
extra packages and the mirror to use for each debootstrap.
The aim is to create a complete debootstrap with all packages
installed and configured, instead of just the base system.
Example configuration:
[General]
arch=armel
directory=/opt/multistrap/
# same as --tidy-up option if set to true
cleanup=true
# same as --no-auth option if set to true
# keyring packages listed in each debootstrap will
# still be installed.
noauth=false
# extract all downloaded archives (default is true)
unpack=true
# aptsources is a list of sections to be used for downloading packages
# and lists and placed in the /etc/apt/sources.list.d/multistrap.sources.list
# of the target. Order is not important
aptsources=Grip Updates
# the order of sections is not important.
# the debootstrap option determines which repository
# is used to calculate the list of Priority: required packages.
debootstrap=Debian
[Debian]
packages=
source=http://ftp.uk.debian.org/debian
keyring=debian-archive-keyring
suite=lenny
This will result in a completely normal debootstrap of Debian lenny from
the specified mirror, for armel in '/opt/multistrap/'.
Specify a package to extend the multistrap to include that package and
all dependencies.
Specify more debootstraps by adding new sections. Section names are used
in the debootstrap general option.
Section names are case-insensitive.
e.g. change
debootstrap=Debian
to
debootstrap=Grip
then add the new section for Grip:
[Grip]
packages=locales
keyring=emdebian-archive-keyring
source=http://www.emdebian.org/grip
suite=lenny
Setting Grip instead of Debian in the debootstrap option, as above,
will provide a base system from Emdebian Grip 1.0 and locate any
missing dependencies in Debian 5.0 Lenny, allowing you to add any
package(s) you need from Debian that are not yet in Emdebian Grip.
All dependencies are resolved only by apt, using all configured
repositories, to use only the most recent and most suitable
dependencies. Note that multistrap turns off Install-Recommends
so if the multistrap needs a package that is only a Recommended
dependency, the recommended package needs to be specified in the
packages line explicitly.
'Architecture' and 'directory' can be overridden on the command line.
Other general options have command line options, except debootstrap
itself.
=head1 General settings:
'directory' specifies the top level directory where the debootstrap
will be created - it is not packed into a .tgz once complete.
As with debootstrap, multistrap will continue after errors.
multistrap does not currently implement the machine:variant support
used in Emdebian but the build directory is not packed up at the
end of the run so other scripts can be used to implement customisations.
=head1 Secure Apt
To use authenticated apt repositories, multistrap either needs to be
able to install an appropriate keyring package from the existing apt
sources *outside the multistrap environment* or have the relevant keys
already configured using apt-key *on the host system*.
If relevant packages exist, specify them in the 'keyring' option for
each repository. multistrap will then check that apt has already
installed this package so that the repository can be authenticated
before any packages are downloaded from it.
Note that *all* repositories to be used with multistrap must be
authenticated or apt will fail. Similarly, secure apt can only be
disabled for all repositories (by using the --no-auth command line
option or setting the general noauth option in the configuration
file), even if only one repository does not have a suitable keyring
available. Not all packages need keyring packages, if you configure
apt-key appropriately.
The keyring package(s) will also be installed inside the multistrap
environment to match the installed apt sources for the multistrap.
All configuration of apt-key needs to be done for the machine
running multistrap itself.
=head1 State
multistrap is stateless - if the directory exists, it will simply
proceed as normal and apt will try to pick up where it left off.
=head1 Configuration
multistrap unpacks the downloaded packages but other stages of
system configuration are not attempted. Examples include:
/etc/inittab
/etc/fstab
/etc/hosts
/etc/securetty
/etc/modules
/etc/hostname
/etc/network/interfaces
/etc/init.d
/etc/dhcp3
Any device-specific device nodes will also need to be created
using MAKEDEV.
As an alternative, multistrap includes a device-table.pl helper
script that can work around some of the issues with MAKEDEV.
device-table.pl requires a device table file along the lines of
the one in the mtd-utils source package.
Once multistrap has successfully created the basic file and
directory layout, other device-specific scripts are needed before
the filesystem can be packaged up and installed onto the
target device.
Once installed, the packages themselves need to be configured
using the package maintainer scripts and C<dpkg --configure -a>,
unless this is a native multistrap.
For C<dpkg> to work, F</proc> and F</sysfs> must be mounted (or
mountable), F</dev/pts> is also recommended.
See also: http://wiki.debian.org/Multistrap
=head1 Environment
To configure the unpacked packages (whether in native or cross mode),
certain environment variables are needed:
Debconf needs to be told to accept that user interaction is not
desired:
DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true
Perl needs to be told to accept that no locales are available inside
the chroot and not to complain:
LC_ALL=C LANGUAGE=C LANG=C
Then, dpkg can configure the packages:
chroot method (PATH = top directory of chroot):
DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true \
LC_ALL=C LANGUAGE=C LANG=C chroot /PATH/ dpkg --configure -a
at a login shell:
# export DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true
# export LC_ALL=C LANGUAGE=C LANG=C
# dpkg --configure -a
(As above, dpkg needs F</proc> and F</sysfs> mounted first.)
=head1 Native mode - multistrap
multistrap was not intended for native support, it was developed for
cross architecture support. In order for multiple repositories to be
used, multistrap only unpacks the packages selected by apt.
In native mode, various post-multistrap operations are likely to be
needed that debootstrap would do for you:
1. copy /etc/hosts into the chroot
2. clean the environment to unset LANGUAGE, LC_ALL and LANG
to silence nuisance perl warnings that obscure other errors
(An alternative to unset the localisation variables is to add
locales to your multistrap configuration file in the 'packages'
option.
A native multistrap can be used directly with chroot, so
C<multistrap> runs C<dpkg --configure -a> at the end of the
multistrap process.
=head1 Cascading configuration
=head1 Machine:variant support
The old packages.conf variables from emsandbox can all be converted
into C<multistrap> configuration variables. The machine:variant
support in C<multistrap> concentrates on the scripts,
F<config.sh> and F<setup.sh>
Once C<multistrap> has unpacked the downloaded packages, the
C<setup.sh> can be called, passing the location and architecture of
the root filesystem, so that other fine tuning can take place. At
this stage, any operations inside the rootfs must not try to execute
any binaries within the rootfs. As the final stage of the multistrap
process, C<config.sh> is copied into the F</machine/$MACHINE/default/>
directory of the rootfs.
One advantage of using machine:variant support is that the entire
rootfilesystem can be managed by a single call to multistrap - this
is useful when building root filesystems in userspace.
=head1 Restricting package selection
C<multistrap> includes Required packages by default, the current list
of packages can be seen using:
grep-available -FPriority 'required' -sPackage
If the OmitRequired option is set to true, these packages will not be
added - whilst useful, this option can easily lead to a useless
rootfs. Only the packages specified manually in the configuration
files will be used in the calculations - dependencies of those packages
will be added but no others.
=head1 Collecting packages from specific codenames/suites.
Packages specified explicitly in the configuration sections will be
passed to apt as package/codename so that the configuration controls
which version of a package is installed should the package exist in
two sources with different suites.
When using this support in Lenny, ensure that each section uses the
codename (etch, lenny, squeeze, sid) instead of the suite (oldstable,
stable, testing, sid) for the C<suite> configuration item as the version
of apt in Lenny and previous can only use the codename.
=head1 Recommends TOIMPLEMENT:
Default recommends OFF
option to set it as on.
=cut
sub dump_config {
print "Debootstrap: ".join (", ", sort @debootstrap)."\n";
print "Apt sources: ".join (", ", sort @aptsources)."\n";
print "Includes: ".join (", ", sort @includes)."\n";
print "Sources: ".join (", ", sort values %sources)."\n";
print "Packages: ".join (", ", sort values %packages)."\n";
print "Suites: ".join (", ", sort values %suites)."\n";
print "Components: ".join (", ", sort values %components)."\n";
print "Extra Packages: ".join (", ", sort @extrapkgs)."\n"
if (scalar @extrapkgs > 0);
print "arch: $arch\n";
print "dir: $dir\n";
print "unpack: $unpack\n" if (defined $unpack);
print "configscript: $configsh\n" if (defined $configsh);
print "setupscript: $setupsh\n" if (defined $setupsh);
print "omitrequired: $omitrequired\n" if (defined $omitrequired);
print "tidy_apt: $tidy\n" if (defined $tidy);
print "no_authentication: $noauth\n" if (defined $noauth);
print "source_dir: $sourcedir\n" if (defined $sourcedir);
exit 0;
}

323
multistrap.pod Normal file
View file

@ -0,0 +1,323 @@
=pod
=head1 Name
multistrap - extends debootstrap for multiple repository support
=head1 Synopsis
multistrap [-a ARCH] [-d DIR] -f CONFIG_FILE
multistrap -?|-h|--help|--version
=head1 Options
(These options can also be set in the configuration file.)
--tidy-up - remove apt cache data, downloaded Packages files and
the apt package cache. Same as cleanup=true.
--no-auth - allow the use of unauthenticated repositories. Same
as noauth=true
=head1 Description
multistrap provides a debootstrap-like method based on apt and
extended to provide support for multiple repositories, using a
configuration file to specify the relevant suites, architecture,
extra packages and the mirror to use for each debootstrap.
The aim is to create a complete debootstrap with all packages
installed and configured, instead of just the base system.
Example configuration:
[General]
arch=armel
directory=/opt/multistrap/
# same as --tidy-up option if set to true
cleanup=true
# same as --no-auth option if set to true
# keyring packages listed in each debootstrap will
# still be installed.
noauth=false
# extract all downloaded archives (default is true)
unpack=true
# aptsources is a list of sections to be used for downloading packages
# and lists and placed in the /etc/apt/sources.list.d/multistrap.sources.list
# of the target. Order is not important
aptsources=Grip Updates
# the order of sections is not important.
# the debootstrap option determines which repository
# is used to calculate the list of Priority: required packages.
debootstrap=Debian
[Debian]
packages=
source=http://ftp.uk.debian.org/debian
keyring=debian-archive-keyring
suite=lenny
This will result in a completely normal debootstrap of Debian lenny from
the specified mirror, for armel in '/opt/multistrap/'.
Specify a package to extend the multistrap to include that package and
all dependencies.
Specify more debootstraps by adding new sections. Section names are used
in the debootstrap general option.
Section names are case-insensitive.
e.g. change
debootstrap=Debian
to
debootstrap=Grip
then add the new section for Grip:
[Grip]
packages=locales
keyring=emdebian-archive-keyring
source=http://www.emdebian.org/grip
suite=lenny
Setting Grip instead of Debian in the debootstrap option, as above,
will provide a base system from Emdebian Grip 1.0 and locate any
missing dependencies in Debian 5.0 Lenny, allowing you to add any
package(s) you need from Debian that are not yet in Emdebian Grip.
All dependencies are resolved only by apt, using all configured
repositories, to use only the most recent and most suitable
dependencies. Note that multistrap turns off Install-Recommends
so if the multistrap needs a package that is only a Recommended
dependency, the recommended package needs to be specified in the
packages line explicitly.
'Architecture' and 'directory' can be overridden on the command line.
Other general options have command line options, except debootstrap
itself.
=head1 General settings:
'directory' specifies the top level directory where the debootstrap
will be created - it is not packed into a .tgz once complete.
As with debootstrap, multistrap will continue after errors.
multistrap also implements the machine:variant support originally
used in Emdebian Crush, although in a different implementation. Using
the cascading configuration support, particular machine:variant
combinations can be supported by simple changes on the command line.
Setting C<compress> to true also packs up the final filesystem into
a tarball.
=head1 Secure Apt
To use authenticated apt repositories, multistrap either needs to be
able to install an appropriate keyring package from the existing apt
sources *outside the multistrap environment* or have the relevant keys
already configured using apt-key *on the host system*.
If relevant packages exist, specify them in the 'keyring' option for
each repository. multistrap will then check that apt has already
installed this package so that the repository can be authenticated
before any packages are downloaded from it.
Note that *all* repositories to be used with multistrap must be
authenticated or apt will fail. Similarly, secure apt can only be
disabled for all repositories (by using the --no-auth command line
option or setting the general noauth option in the configuration
file), even if only one repository does not have a suitable keyring
available. Not all packages need keyring packages, if you configure
apt-key appropriately.
The keyring package(s) will also be installed inside the multistrap
environment to match the installed apt sources for the multistrap.
All configuration of apt-key needs to be done for the machine
running multistrap itself.
=head1 State
multistrap is stateless - if the directory exists, it will simply
proceed as normal and apt will try to pick up where it left off.
=head1 Configuration
multistrap unpacks the downloaded packages but other stages of
system configuration are not attempted. Examples include:
/etc/inittab
/etc/fstab
/etc/hosts
/etc/securetty
/etc/modules
/etc/hostname
/etc/network/interfaces
/etc/init.d
/etc/dhcp3
Any device-specific device nodes will also need to be created
using MAKEDEV.
As an alternative, multistrap includes a device-table.pl helper
script that can work around some of the issues with MAKEDEV.
device-table.pl requires a device table file along the lines of
the one in the mtd-utils source package.
Once multistrap has successfully created the basic file and
directory layout, other device-specific scripts are needed before
the filesystem can be packaged up and installed onto the
target device.
Once installed, the packages themselves need to be configured
using the package maintainer scripts and C<dpkg --configure -a>,
unless this is a native multistrap.
For C<dpkg> to work, F</proc> and F</sysfs> must be mounted (or
mountable), F</dev/pts> is also recommended.
See also: http://wiki.debian.org/Multistrap
=head1 Environment
To configure the unpacked packages (whether in native or cross mode),
certain environment variables are needed:
Debconf needs to be told to accept that user interaction is not
desired:
DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true
Perl needs to be told to accept that no locales are available inside
the chroot and not to complain:
LC_ALL=C LANGUAGE=C LANG=C
Then, dpkg can configure the packages:
chroot method (PATH = top directory of chroot):
DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true \
LC_ALL=C LANGUAGE=C LANG=C chroot /PATH/ dpkg --configure -a
at a login shell:
# export DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true
# export LC_ALL=C LANGUAGE=C LANG=C
# dpkg --configure -a
(As above, dpkg needs F</proc> and F</sysfs> mounted first.)
=head1 Native mode - multistrap
multistrap was not intended for native support, it was developed for
cross architecture support. In order for multiple repositories to be
used, multistrap only unpacks the packages selected by apt.
In native mode, various post-multistrap operations are likely to be
needed that debootstrap would do for you:
1. copy /etc/hosts into the chroot
2. clean the environment to unset LANGUAGE, LC_ALL and LANG
to silence nuisance perl warnings that obscure other errors
(An alternative to unset the localisation variables is to add
locales to your multistrap configuration file in the 'packages'
option.
A native multistrap can be used directly with chroot, so
C<multistrap> runs C<dpkg --configure -a> at the end of the
multistrap process.
=head1 Cascading configuration
To support multiple variants of a basic (common) configuration,
C<multistrap> allows configuration files to include other (more general)
configuration files. i.e. the most detailed / specific configuration
file is specified on the command line and that file includes another
file which is shared by other configurations.
Base file:
/usr/share/multistrap/crosschroot.conf
Variations:
/usr/share/multistrap/armel.conf
Specifying just the armel.conf file will get the rest of the settings
from crosschroot.conf so that common changes only need to be made in a
single file.
It is B<strongly> recommended that any changes to the configuration files
involved in any particular cascade are tested using the C<--simulate>
option to multistrap which will output a summary of the options that
have been set once the cascade is complete. Note that multistrap does
B<not warn you> if a configuration file contains an unrecognised
option (for future compatibility with backported configurations), so a
simple typo can result in an option not being set.
=head1 Machine:variant support
The old packages.conf variables from emsandbox can all be converted
into C<multistrap> configuration variables. The machine:variant
support in C<multistrap> concentrates on the scripts,
F<config.sh> and F<setup.sh>
Once C<multistrap> has unpacked the downloaded packages, the
C<setup.sh> can be called, passing the location and architecture of
the root filesystem, so that other fine tuning can take place. At
this stage, any operations inside the rootfs must not try to execute
any binaries within the rootfs. As the final stage of the multistrap
process, C<config.sh> is copied into the F</machine/$MACHINE/default/>
directory of the rootfs.
One advantage of using machine:variant support is that the entire
rootfilesystem can be managed by a single call to multistrap - this
is useful when building root filesystems in userspace.
To enable machine:variant support, specify the path to the scripts to
be called in the variant configuration file (General section):
[General]
include=/path/to/general.conf
setup_script=/path/to/setup.sh
config_script=/path/to/config.sh
=head1 Restricting package selection
C<multistrap> includes Required packages by default, the current list
of packages can be seen using:
grep-available -FPriority 'required' -sPackage
If the OmitRequired option is set to true, these packages will not be
added - whilst useful, this option can easily lead to a useless
rootfs. Only the packages specified manually in the configuration
files will be used in the calculations - dependencies of those packages
will be added but no others.
=head1 Collecting packages from specific codenames/suites.
Packages specified explicitly in the configuration sections will be
passed to apt as package/codename so that the configuration controls
which version of a package is installed should the package exist in
two sources with different suites.
When using this support in Lenny, ensure that each section uses the
codename (etch, lenny, squeeze, sid) instead of the suite (oldstable,
stable, testing, sid) for the C<suite> configuration item as the version
of apt in Lenny and previous can only use the codename.
=head1 Recommends TOIMPLEMENT:
Default recommends OFF
option to set it as on.
=cut