This is a long long post because I’m providing a lot of configuration files and explanation. I could have split this into multiple posts, but I felt it made sense to put it all in one entry
In my homelab I use virsh
to manage QEMU/KVM virtual machines. I might
want to spin up a test VM (oh, say Debian 12) to do some playing around
and then destroy it again. And, naturally, I don’t want to do an interactive
install each time; I just want to run a script and 5 minutes later have a
shiny new VM to play with. And I don’t want a GUI; these are “server”
builds, and so should have a serial console.
Fortunately both Rocky Linux (via RedHat kickstart) and Debian Linux (via preseed) has the ability to perform hands-off installations.
Pre-requisites.
Pretty much all you need is a local webserver that can server out the kickstart/preseed file, and any other file you want.
For historical reasons (I started doing this with RedHat, then later CentOS)
the base of my tree is at http://10.0.0.137/CentOS/kickstart
. I still use this
base for my Debian builds, even though it isn’t needed.
A fast internet connection is useful. You could
mirror upstream installer trees (I used to do this),
and then install locally, but now I have gigabit internet access the
speed of the download isn’t so much the limiting factor. As part of
my migration to Debian, I installed apt-cacher-ng
on a machine so
repeated installs won’t hit the upstream infrastructure so much (packages
are served locally instead) and this ended up only saving 4 or 5 seconds.
(Enterprise’s probably have a local mirror already because you shouldn’t
be allowing direct access to external sites!)
Scripting
Now I’ve scripted everything I’m going to describe below so I don’t actually run any of these commands. But I’m going to explain the low level commands so you can see what to do and can write your own wrappers. My scripts aren’t that useful because they are very dependent on other parts of my highly unique setup (e.g. it looks at the inventory file, if it sees the hostname I want to build in there then it extracts the MAC address and allocates this to the VM; in that way the VM will get a consistent value from DHCP ‘cos the DHCP server config is also built from the same file!)
Allocating disk space
KVM can use files on the filesystem or volumes. For example, to create a 10G logical volume I could run
sudo lvcreate --wipesignatures n -L 10G -n vm.debian13 SSD
This will create /dev/SSD/vm.debian13
(I have a naming scheme; all VM logical volumes are called vm.
followed
by the name of the VM. That way I can do an lvs
and know exactly what
each volume is for).
Now when running virt-install
to create the VM you would pass this paramter
--disk path=/dev/SSD/vm.debian13`
Alternatively if you want to use a QCOW image on the filesystem would would specify the path to the image and the size you want; the paramter then becomes something like
--disk path=/var/lib/libvirt/images/debian13.img,size=10G
Networking
I have 3 VLANs that are visible on
the host as br-lan
, br-guest
, and br-iot
. Normally I build my VMs on
the main LAN, but I could build on any of the VLANs.
This is specified with the virt-install
network paramter;
--network=bridge=br-lan,target=v-debian13
We can also specify a MAC address here if we wanted to:
--network=bridge=br-lan,target=v-debian13,mac=01:02:03:04:05:06
The target
value will name the virtual interface so, once again, I can
quickly see what running VMs are on what bridges:
% brctl show
bridge name bridge id STP enabled interfaces
br-guest 8000.b0416f0e52ab no enp1s0.11
v-pinky9-guest
br-lan 8000.b0416f0e52ab no enp1s0.10
v-debian13
v-pinky9
v-test-debian
br-iot 8000.b0416f0e52ab no enp1s0.12
v-pinky9-iot
Or by looking at the ip a
output we can also see it; eg
19: v-debian13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UNKNOWN group default qlen 1000
link/ether fe:54:00:92:38:7f brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc54:ff:fe92:387f/64 scope link
valid_lft forever preferred_lft forever
Other common parameters
To name the VM, -n debian13
To prevent virt-install
from allocating a graphics console we specify `--nographics
To allocate RAM, we specify --ram 1024
(although the amount allocated may
vary across the different OS releases; we’ll get to that, below).
To specify the number of virtual CPUs, --vcpus=1
Installing different OSes
Now we’ve got the basics out of the way we can start to look at the differences
between different OS builds. For ease of reading when it comes to the
network and disk allocations, I’ll just ...
the value since I’ve described
them in detail above.
Rocky Linux
I have configurations for Rocky 8 and Rocky 9. Most of the parameters
to virt-install
are the same. The differences are:
Rocky 8 variables
ks=rocky8
repo=https://dl.rockylinux.org/pub/rocky/8/BaseOS/x86_64/kickstart/
ram=3073
os=rocky8
Rocky 9 variables
ks=rocky9
repo=https://dl.rockylinux.org/pub/rocky/9/BaseOS/x86_64/kickstart/
ram=4096
os=rocky9
Kicking off the Rocky install
We can then run the install process
virt-install \
--noreboot \
--vcpus=1 \
--nographics \
--machine pc-i440fx-rhel7.6.0 \
--accelerate \
-v \
-n $name \
--os-variant=$os \
-r $ram \
--network=... \
--disk ... \
-l $repo \
-x "inst.ks=http://10.0.0.137/CentOS/kickstart/$ks.cfg
ksdevice=ens2 ip=dhcp console=ttyS0,9600"
(that last -x
line entry is really on one line; I’ve split it here for readability)
The ksdevice
entry has changed over time (eg on CentOS 7 it was eth0
) but
it’s been consistent between Rocky 9 and Rocky 9.
The --machine
entry is maybe not needed any more and might be a holdover
from older host OSes (I’m now using Rocky 8 and Rocky 9 as the hypervisor
OS). Similarly the --accelerate
option may no longer be needed!
The -l
flag is where virt-install
will try to download the installer
kernel/ramdisk from. We’re using the files in the kickstart
tree.
The -x
flag specifies what is passed to the kernel as parameters. In this
case we specify where to find the kickstart configuration, as well as the
network device, how to get an IP address and to use a serial console.
Now in this installer I specify --noreboot
so that after the VM has been
built it’s shutdown. This is so I can do some post-install VM tuning;
specifically reduce the memory configured. Rocky Linux will run in a lot
less memory than it needs to install (eg 512), so after the install has
completed I reset the memory requirements and restart the VM
virsh setmaxmem $machine 512M --config
virsh setmem $machine 512M --config
virsh start $machine
The real magic starts in the kickstart file!
Rocky 8 kickstart config
We’ll break this down into parts
install
url --url https://dl.rockylinux.org/pub/rocky/8/BaseOS/x86_64/os
poweroff
lang en_US.UTF-8
keyboard us
network --onboot yes --device eth0 --bootproto dhcp --ipv6 auto
rootpw --iscrypted $6$...
authconfig --enableshadow --passalgo=sha512
firewall --disabled
selinux --disabled
timezone --utc America/New_York
So far this is pretty understandable. Note the rootpw
entry is
pre-encrypted (you could just that this from an existing /etc/shadow
entry, for example) so there’s no plain text secrets.
I also disable firewalld and SELinux (yeah yeah, security guy disabling security…)
Now we get to creating the partitions; I create a 500M /boot
, a small
swap partition and set the rest to be an ext4
. You can define whatever
layout you want here; it’s pretty simple!
zerombr
clearpart --all --initlabel
part /boot --fstype=ext4 --asprimary --size=500
part swap --asprimary --size=512
part / --fstype=ext4 --asprimary --grow --size=1
bootloader --location=mbr --driveorder=vda --append=" crashkernel=auto quiet" --timeout=0
Next we define what packages we want. I typically want to install as small as possible, with some specific extras. I also remove unnecessary firmware
repo --name=AppStream --mirrorlist https://mirrors.rockylinux.org/mirrorlist?repo=AppStream-8&arch=x86_64
%packages
@^minimal-environment
python36
wget
ksh
dos2unix
logwatch
tar
postfix
bind-utils
bc
-kdump
-iwl100-firmware
-iwl1000-firmware
-iwl105-firmware
-iwl135-firmware
-iwl2000-firmware
-iwl2030-firmware
-iwl3160-firmware
-iwl3945-firmware
-iwl4965-firmware
-iwl5000-firmware
-iwl5150-firmware
-iwl6000-firmware
-iwl6000g2a-firmware
-iwl6050-firmware
-iwl7260-firmware
%end
Since I removed kdump
, we can also tell this to be disabled
%addon com_redhat_kdump --disable --reserve-mb='auto'
%end
And then some post-install steps. I’ll get to what the “post.tar” file is later in this post because I use a consistent method for Rocky and Debian
%post
exec 1>/root/ks-post.log 2>&1
tail -f /root/ks-post.log > /dev/console &
alternatives --set python /usr/bin/python3
mkdir /tmp/post
cd /tmp/post
wget -q http://10.0.0.137/CentOS/kickstart/post_r8.tar
tar xf post_r8.tar
sh runme
cd /tmp
rm -rf post
%end
And that’s it! The complete file is available here
Rocky 9 kickstart config
This is kinda the same deal, but with some minor differences. Rather than detail the whole file again, my config is available here
Mostly the differences are in url
and repo
entries, as well as allocating
more space to boot
and swap.
Debian Linux
The Debian installer works in a different way to Redhat’s. It exposes things in a more “raw” level; essentially each part of the installer has a tag for the questions it asks and you can “preseed” answers by defining the tag type and value. It’s also a little inconsistent around when it reads the file, so some of the values need to be passed to the kernel for them to be read.
There are fewer variables that need to be considered for virt-install
;
basically just the OS version and codename
Debian 12 variables
version=12
codename=bookworm
Debian 13 variables
version=13
codename=trixie
Kicking off the Debian install
This looks very similar to the Rocky one, but the “-x” options are a lot different
virt-install \
--noreboot \
--vcpus=1 \
--nographics \
--accelerate \
-v \
-n $name \
--os-variant=debian12 \
--ram 1024 \
--network=... \
--disk ... \
-l http://ftp.us.debian.org/debian/dists/$codename/main/installer-amd64/ \
-x "console=tty0 console=ttyS0 auto
url=http://10.0.0.137/CentOS/kickstart/debian$version.cfg
language=en country=US locale=en_US.UTF-8 keymap=us
netcfg/get_hostname=debian netcfg/get_domain=spuddy.org"
(that last -x
line entry is really on one line; I’ve split it here for readability)
The “preseed” file is specified by the auto url=...
option. But we also
need to specify other stuff (language
, country
, locale
, keymap
)
because the installer tries to use those values before the preseed file is
read. Oddly enough we also need to specify the hostname and domain name
here as well, even though we get real values from DHCP.
Debian 12 preseed config
This is a much more hard-to-build file than the kickstart one.
Lines beginning d-i
mean “debian-installer”. There are also tasksel
and popularity-contest
lines. And because I’m installing postfix
we
also need some postfix
lines. This took lots of trial and error and
reading docs and examples (and help from r/debian
).
#_preseed_V1
# These values all based on
# https://d-i.debian.org/manual/en.amd64/apbs04.html
# https://michael.kjorling.se/pages/debian-12-bookworm-preseed/preseed.cfg
d-i debian-installer/language string en
d-i debian-installer/country string US
d-i debian-installer/locale string en_US.UTF-8
d-i keyboard-configuration/xkb-keymap select us
Those lines duplicate the options we passed on the kernel parameters; they may not be needed!
# Where to find the installer
d-i mirror/protocol string http
d-i mirror/country string manual
d-i mirror/http/hostname string http.us.debian.org
d-i mirror/http/directory string /debian
d-i mirror/http/proxy string http://10.0.0.2:3142
# Suite to install.
d-i mirror/suite string bookworm
Here we’re defining where to install from; the proxy
line is telling the
installer to use the apt-cacher-ng
server. We’re installing Debian 12
aka “bookworm”.
Aside: I really dislike using codenames in configs. It’s so much easier to remember “11” or “12” or “13” rather than … well, I don’t remember what 11 is; 12 is bookworm, 13 is trixie. It’s cute, but it’s bad. And, yes, I’m staring very hard at Apple…
The config continues and is pretty easy to understand
d-i passwd/root-login boolean true
d-i passwd/root-password-crypted password $6$...
# To create a normal user account.
d-i passwd/user-fullname string Stephen Harris
d-i passwd/username string sweh
d-i passwd/user-password-crypted password $6$...
d-i passwd/user-uid string 500
d-i time/zone string US/Eastern
And now we get to partitioning. Debian comes with some default templates, but they tend to put the OS at the beginning of the disk and the swap at the end. I wanted swap at the beginning so I could extend the disk and simply resize/extend the partition if I needed more space. So this will create a 1024M swap file and then the rest is a single ext4 root partition
d-i partman-auto/method string regular
d-i partman-auto/expert_recipe string \
boot-root :: \
1024 1024 1024 linux-swap \
$primary{ } \
method{ swap } format{ } \
. \
1 1 -1 ext4 \
$primary{ } $bootable{ } \
method{ format } format{ } \
use_filesystem{ } filesystem{ ext4 } \
mountpoint{ / } \
.
d-i partman/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
We can then configure what additional repositories are in use:
d-i apt-setup/cdrom/set-first boolean false
d-i apt-setup/non-free-firmware boolean true
d-i apt-setup/non-free boolean true
d-i apt-setup/contrib boolean true
d-i apt-setup/disable-cdrom-entries boolean true
Now we define the OS base (“standard” with “ssh-server”) along with any
additional packages; note that I chose postfix
as my mail server; that
will also need configuring later on.
tasksel tasksel/first multiselect standard, ssh-server
d-i pkgsel/include string ksh sudo curl postfix rsyslog binutils dump bsd-mailx apt-file rsync collectd-core
d-i pkgsel/upgrade select full-upgrade
popularity-contest popularity-contest/participate boolean false
Grub information:
d-i grub-installer/only_debian boolean true
d-i grub-installer/bootdev string /dev/vda
d-i finish-install/reboot_in_progress note
Now comes the postfix
config
postfix postfix/mailname string spuddy.org
postfix postfix/main_mailer_type string 'Internet with smarthost'
postfix postfix/relayhost string mailhost.spuddy.org
And then a post-install command that does something similar to the %post
section of kickstart. This command is run at the end of the installer.
d-i preseed/late_command string chroot /target sh -c "cd /tmp && /usr/bin/curl -o post.tar http://10.0.0.137/CentOS/kickstart/post_d12.tar && /usr/bin/tar xf post.tar && /bin/sh -x /tmp/debian-postinstall"
The complete file is available here
Debian 13 preseed config.
This is almost identical, except the mirror/suite
refers to trixie
and the late_command
refers to post_d13.tar
. Just for completeness,
it’s here.
The post-install steps
For all 4 build types there’s a post-install step. This primarily involves downloading a tarball, extracting it, and running the script that’s inside it.
What this does is deploy some standard configurations to each machine;
e.g. it ensures my account has a standard $HOME
(with my ssh public
keys), might add an additional repos (eg EPEL for Rocky), configures
postfix for my smarthost, creates email aliases so mail to my account
and to root is forwarded properly and so on. Because it’s a full script
you can do almost anything; it runs inside the build chroot
environment.
An example of this might be my Debian 12 script:
# Extract all the files I want
cd /
/usr/bin/tar xvfp /tmp/files.tar
/bin/rm /tmp/files.tar
# Remove "quiet" from kernel boot params
sed -i 's/"quiet"/""/' /etc/default/grub
# Ensure grub is updated with any changes that might have made
/sbin/update-grub
# Fix my account
/usr/bin/chsh -s /bin/ksh sweh
/sbin/usermod -a -G sudo sweh
/sbin/usermod -a -G root sweh
# Setup postfix correctly
echo 'root: root@mailhost' >> /etc/aliases
/usr/bin/newaliases
/usr/sbin/postconf append_dot_mydomain=yes
/usr/sbin/postconf mydomain=spuddy.org
/usr/sbin/postconf 'mydestination=$myhostname, localhost.$mydomain, localhost'
/usr/sbin/postconf 'myorigin=$myhostname'
/usr/sbin/postconf -X myhostname
/usr/bin/systemctl enable postfix
# Remove static hostname
echo "" > /etc/hostname
# Remove this annoying entry from hosts file
sed -i '/127.0.1.1/d' /etc/hosts
# Fix PAM SSH
sed -i 's/user_readenv=1//' /etc/pam.d/sshd
echo `date`: Post script completed
The files.tar
it refers to is embedded into the post_d12.tar
script
and contains things such as
./etc/apt/apt.conf.d/00aptproxy
./etc/sudoers.d/sweh
./etc/sysctl.d/00-dmesg.conf
./etc/default/grub.d/console.cfg
./etc/default/grub.d/apparmor.cfg
./etc/logwatch/conf/services/zz-disk_space.conf
./etc/ntp.conf
./etc/cron.daily/need_patches
./home/sweh/.profile
./home/sweh/.envfile
./home/sweh/.forward
./home/sweh/.ssh/authorized_keys
./root/.ssh/authorized_keys
(that’s not the complete list, just a sample)
Basically, any “standard” changes you want deployed to all my VMs is performed in this script, and it can be customised to each OS variant.
Conclusion
What I’ve described here is suitable for a homelab. But some of the same
concepts can be applied to enterprises as well. For example, that kickstart
file could be modified to be regional specific to pick a local replica to
get the repos. The post-install script could call out to something like
ansbible
or puppet
to do proper configuration management (even auto
install and configure something like Apache web server if the machine is
defined as “webserver” in an inventory system). The automated install
options provided by Rocky (RedHat) and Debian are very flexible.
In my case, I can just run deb_install -13 new_host
and it will create
the LV for disk space, and run virt_install
with all the required parameters.
5 minutes later I have a new fully patched VM to test and play with, with
sane default values.