Hands off Rocky/Debian installs

For my homelab

This is a long long post because I’m providing a lot of configuration files and explanation. I could have split this into multiple posts, but I felt it made sense to put it all in one entry

In my homelab I use virsh to manage QEMU/KVM virtual machines. I might want to spin up a test VM (oh, say Debian 12) to do some playing around and then destroy it again. And, naturally, I don’t want to do an interactive install each time; I just want to run a script and 5 minutes later have a shiny new VM to play with. And I don’t want a GUI; these are “server” builds, and so should have a serial console.

Fortunately both Rocky Linux (via RedHat kickstart) and Debian Linux (via preseed) has the ability to perform hands-off installations.

Pre-requisites.

Pretty much all you need is a local webserver that can server out the kickstart/preseed file, and any other file you want.

For historical reasons (I started doing this with RedHat, then later CentOS) the base of my tree is at http://10.0.0.137/CentOS/kickstart. I still use this base for my Debian builds, even though it isn’t needed.

A fast internet connection is useful. You could mirror upstream installer trees (I used to do this), and then install locally, but now I have gigabit internet access the speed of the download isn’t so much the limiting factor. As part of my migration to Debian, I installed apt-cacher-ng on a machine so repeated installs won’t hit the upstream infrastructure so much (packages are served locally instead) and this ended up only saving 4 or 5 seconds. (Enterprise’s probably have a local mirror already because you shouldn’t be allowing direct access to external sites!)

Scripting

Now I’ve scripted everything I’m going to describe below so I don’t actually run any of these commands. But I’m going to explain the low level commands so you can see what to do and can write your own wrappers. My scripts aren’t that useful because they are very dependent on other parts of my highly unique setup (e.g. it looks at the inventory file, if it sees the hostname I want to build in there then it extracts the MAC address and allocates this to the VM; in that way the VM will get a consistent value from DHCP ‘cos the DHCP server config is also built from the same file!)

Allocating disk space

KVM can use files on the filesystem or volumes. For example, to create a 10G logical volume I could run

   sudo lvcreate --wipesignatures n -L 10G -n vm.debian13 SSD

This will create /dev/SSD/vm.debian13

(I have a naming scheme; all VM logical volumes are called vm. followed by the name of the VM. That way I can do an lvs and know exactly what each volume is for).

Now when running virt-install to create the VM you would pass this paramter

   --disk path=/dev/SSD/vm.debian13`

Alternatively if you want to use a QCOW image on the filesystem would would specify the path to the image and the size you want; the paramter then becomes something like

   --disk path=/var/lib/libvirt/images/debian13.img,size=10G

Networking

I have 3 VLANs that are visible on the host as br-lan, br-guest, and br-iot. Normally I build my VMs on the main LAN, but I could build on any of the VLANs.

This is specified with the virt-install network paramter;

   --network=bridge=br-lan,target=v-debian13

We can also specify a MAC address here if we wanted to:

   --network=bridge=br-lan,target=v-debian13,mac=01:02:03:04:05:06

The target value will name the virtual interface so, once again, I can quickly see what running VMs are on what bridges:

   % brctl show
   bridge name     bridge id               STP enabled     interfaces
   br-guest        8000.b0416f0e52ab       no              enp1s0.11
                                                           v-pinky9-guest
   br-lan          8000.b0416f0e52ab       no              enp1s0.10
                                                           v-debian13
                                                           v-pinky9
                                                           v-test-debian
   br-iot          8000.b0416f0e52ab       no              enp1s0.12
                                                           v-pinky9-iot

Or by looking at the ip a output we can also see it; eg

   19: v-debian13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UNKNOWN group default qlen 1000
       link/ether fe:54:00:92:38:7f brd ff:ff:ff:ff:ff:ff
       inet6 fe80::fc54:ff:fe92:387f/64 scope link
          valid_lft forever preferred_lft forever

Other common parameters

To name the VM, -n debian13

To prevent virt-install from allocating a graphics console we specify `--nographics

To allocate RAM, we specify --ram 1024 (although the amount allocated may vary across the different OS releases; we’ll get to that, below).

To specify the number of virtual CPUs, --vcpus=1

Installing different OSes

Now we’ve got the basics out of the way we can start to look at the differences between different OS builds. For ease of reading when it comes to the network and disk allocations, I’ll just ... the value since I’ve described them in detail above.

Rocky Linux

I have configurations for Rocky 8 and Rocky 9. Most of the parameters to virt-install are the same. The differences are:

Rocky 8 variables

   ks=rocky8
   repo=https://dl.rockylinux.org/pub/rocky/8/BaseOS/x86_64/kickstart/ 
   ram=3073 
   os=rocky8

Rocky 9 variables

   ks=rocky9 
   repo=https://dl.rockylinux.org/pub/rocky/9/BaseOS/x86_64/kickstart/ 
   ram=4096 
   os=rocky9

Kicking off the Rocky install

We can then run the install process

   virt-install \
       --noreboot \
       --vcpus=1 \
       --nographics \
       --machine pc-i440fx-rhel7.6.0 \
       --accelerate \
       -v \
       -n $name \
       --os-variant=$os \
       -r $ram \
       --network=... \
       --disk ... \
       -l $repo \
       -x "inst.ks=http://10.0.0.137/CentOS/kickstart/$ks.cfg
       ksdevice=ens2 ip=dhcp console=ttyS0,9600"

(that last -x line entry is really on one line; I’ve split it here for readability)

The ksdevice entry has changed over time (eg on CentOS 7 it was eth0) but it’s been consistent between Rocky 9 and Rocky 9.

The --machine entry is maybe not needed any more and might be a holdover from older host OSes (I’m now using Rocky 8 and Rocky 9 as the hypervisor OS). Similarly the --accelerate option may no longer be needed!

The -l flag is where virt-install will try to download the installer kernel/ramdisk from. We’re using the files in the kickstart tree.

The -x flag specifies what is passed to the kernel as parameters. In this case we specify where to find the kickstart configuration, as well as the network device, how to get an IP address and to use a serial console.

Now in this installer I specify --noreboot so that after the VM has been built it’s shutdown. This is so I can do some post-install VM tuning; specifically reduce the memory configured. Rocky Linux will run in a lot less memory than it needs to install (eg 512), so after the install has completed I reset the memory requirements and restart the VM

   virsh setmaxmem $machine 512M --config
   virsh setmem $machine 512M --config
   virsh start $machine

The real magic starts in the kickstart file!

Rocky 8 kickstart config

We’ll break this down into parts

   install
   url --url https://dl.rockylinux.org/pub/rocky/8/BaseOS/x86_64/os
   poweroff

   lang en_US.UTF-8
   keyboard us

   network --onboot yes --device eth0 --bootproto dhcp --ipv6 auto

   rootpw  --iscrypted $6$...
   authconfig --enableshadow --passalgo=sha512

   firewall --disabled
   selinux --disabled

   timezone --utc America/New_York

So far this is pretty understandable. Note the rootpw entry is pre-encrypted (you could just that this from an existing /etc/shadow entry, for example) so there’s no plain text secrets.

I also disable firewalld and SELinux (yeah yeah, security guy disabling security…)

Now we get to creating the partitions; I create a 500M /boot, a small swap partition and set the rest to be an ext4. You can define whatever layout you want here; it’s pretty simple!

   zerombr
   clearpart --all --initlabel
   part /boot --fstype=ext4 --asprimary --size=500
   part swap --asprimary --size=512
   part / --fstype=ext4 --asprimary --grow --size=1
   bootloader --location=mbr --driveorder=vda --append=" crashkernel=auto quiet" --timeout=0

Next we define what packages we want. I typically want to install as small as possible, with some specific extras. I also remove unnecessary firmware

   repo --name=AppStream --mirrorlist https://mirrors.rockylinux.org/mirrorlist?repo=AppStream-8&arch=x86_64

   %packages
   @^minimal-environment
   python36
   wget
   ksh
   dos2unix
   logwatch
   tar
   postfix
   bind-utils
   bc
   -kdump
   -iwl100-firmware
   -iwl1000-firmware
   -iwl105-firmware
   -iwl135-firmware
   -iwl2000-firmware
   -iwl2030-firmware
   -iwl3160-firmware
   -iwl3945-firmware
   -iwl4965-firmware
   -iwl5000-firmware
   -iwl5150-firmware
   -iwl6000-firmware
   -iwl6000g2a-firmware
   -iwl6050-firmware
   -iwl7260-firmware
   %end

Since I removed kdump, we can also tell this to be disabled

   %addon com_redhat_kdump --disable --reserve-mb='auto'
   %end

And then some post-install steps. I’ll get to what the “post.tar” file is later in this post because I use a consistent method for Rocky and Debian

   %post
   exec 1>/root/ks-post.log 2>&1
   tail -f /root/ks-post.log > /dev/console &

   alternatives --set python /usr/bin/python3

   mkdir /tmp/post
   cd /tmp/post
   wget -q http://10.0.0.137/CentOS/kickstart/post_r8.tar
   tar xf post_r8.tar
   sh runme
   cd /tmp
   rm -rf post
   %end

And that’s it! The complete file is available here

Rocky 9 kickstart config

This is kinda the same deal, but with some minor differences. Rather than detail the whole file again, my config is available here

Mostly the differences are in url and repo entries, as well as allocating more space to boot and swap.

Debian Linux

The Debian installer works in a different way to Redhat’s. It exposes things in a more “raw” level; essentially each part of the installer has a tag for the questions it asks and you can “preseed” answers by defining the tag type and value. It’s also a little inconsistent around when it reads the file, so some of the values need to be passed to the kernel for them to be read.

There are fewer variables that need to be considered for virt-install; basically just the OS version and codename

Debian 12 variables

   version=12
   codename=bookworm

Debian 13 variables

   version=13
   codename=trixie

Kicking off the Debian install

This looks very similar to the Rocky one, but the “-x” options are a lot different

   virt-install \
     --noreboot \
     --vcpus=1 \
     --nographics \
     --accelerate \
     -v \
     -n $name \
     --os-variant=debian12 \
     --ram 1024 \
     --network=... \
     --disk ... \
     -l http://ftp.us.debian.org/debian/dists/$codename/main/installer-amd64/ \
     -x "console=tty0 console=ttyS0 auto
     url=http://10.0.0.137/CentOS/kickstart/debian$version.cfg
     language=en country=US locale=en_US.UTF-8 keymap=us
     netcfg/get_hostname=debian netcfg/get_domain=spuddy.org"

(that last -x line entry is really on one line; I’ve split it here for readability)

The “preseed” file is specified by the auto url=... option. But we also need to specify other stuff (language, country, locale, keymap) because the installer tries to use those values before the preseed file is read. Oddly enough we also need to specify the hostname and domain name here as well, even though we get real values from DHCP.

Debian 12 preseed config

This is a much more hard-to-build file than the kickstart one.

Lines beginning d-i mean “debian-installer”. There are also tasksel and popularity-contest lines. And because I’m installing postfix we also need some postfix lines. This took lots of trial and error and reading docs and examples (and help from r/debian).

   #_preseed_V1

   # These values all based on
   #   https://d-i.debian.org/manual/en.amd64/apbs04.html
   #   https://michael.kjorling.se/pages/debian-12-bookworm-preseed/preseed.cfg

   d-i debian-installer/language string en
   d-i debian-installer/country string US
   d-i debian-installer/locale string en_US.UTF-8
   d-i keyboard-configuration/xkb-keymap select us

Those lines duplicate the options we passed on the kernel parameters; they may not be needed!

   # Where to find the installer
   d-i mirror/protocol string http
   d-i mirror/country string manual
   d-i mirror/http/hostname string http.us.debian.org
   d-i mirror/http/directory string /debian
   d-i mirror/http/proxy string http://10.0.0.2:3142

   # Suite to install.
   d-i mirror/suite string bookworm

Here we’re defining where to install from; the proxy line is telling the installer to use the apt-cacher-ng server. We’re installing Debian 12 aka “bookworm”.

Aside: I really dislike using codenames in configs. It’s so much easier to remember “11” or “12” or “13” rather than … well, I don’t remember what 11 is; 12 is bookworm, 13 is trixie. It’s cute, but it’s bad. And, yes, I’m staring very hard at Apple…

The config continues and is pretty easy to understand

   d-i passwd/root-login boolean true
   d-i passwd/root-password-crypted password $6$...

   # To create a normal user account.
   d-i passwd/user-fullname string Stephen Harris
   d-i passwd/username string sweh
   d-i passwd/user-password-crypted password $6$...
   d-i passwd/user-uid string 500

   d-i time/zone string US/Eastern

And now we get to partitioning. Debian comes with some default templates, but they tend to put the OS at the beginning of the disk and the swap at the end. I wanted swap at the beginning so I could extend the disk and simply resize/extend the partition if I needed more space. So this will create a 1024M swap file and then the rest is a single ext4 root partition

   d-i partman-auto/method string regular
   d-i partman-auto/expert_recipe string                         \
         boot-root ::                                            \
                 1024 1024 1024 linux-swap                       \
                         $primary{ }                             \
                         method{ swap } format{ }                \
                 .                                               \
                 1 1 -1 ext4                                     \
                         $primary{ } $bootable{ }                \
                         method{ format } format{ }              \
                         use_filesystem{ } filesystem{ ext4 }    \
                         mountpoint{ / }                         \
                 .
   d-i partman/confirm_write_new_label boolean true
   d-i partman/choose_partition select finish
   d-i partman/confirm boolean true
   d-i partman/confirm_nooverwrite boolean true

We can then configure what additional repositories are in use:

   d-i apt-setup/cdrom/set-first boolean false
   d-i apt-setup/non-free-firmware boolean true
   d-i apt-setup/non-free boolean true
   d-i apt-setup/contrib boolean true
   d-i apt-setup/disable-cdrom-entries boolean true

Now we define the OS base (“standard” with “ssh-server”) along with any additional packages; note that I chose postfix as my mail server; that will also need configuring later on.

   tasksel tasksel/first multiselect standard, ssh-server

   d-i pkgsel/include string ksh sudo curl postfix rsyslog binutils dump bsd-mailx apt-file rsync collectd-core
   d-i pkgsel/upgrade select full-upgrade

   popularity-contest popularity-contest/participate boolean false

Grub information:

   d-i grub-installer/only_debian boolean true
   d-i grub-installer/bootdev  string /dev/vda

   d-i finish-install/reboot_in_progress note

Now comes the postfix config

   postfix postfix/mailname string spuddy.org
   postfix postfix/main_mailer_type string 'Internet with smarthost'
   postfix postfix/relayhost string mailhost.spuddy.org

And then a post-install command that does something similar to the %post section of kickstart. This command is run at the end of the installer.

   d-i preseed/late_command string chroot /target sh -c "cd /tmp && /usr/bin/curl -o post.tar http://10.0.0.137/CentOS/kickstart/post_d12.tar && /usr/bin/tar xf post.tar && /bin/sh -x /tmp/debian-postinstall"

The complete file is available here

Debian 13 preseed config.

This is almost identical, except the mirror/suite refers to trixie and the late_command refers to post_d13.tar. Just for completeness, it’s here.

The post-install steps

For all 4 build types there’s a post-install step. This primarily involves downloading a tarball, extracting it, and running the script that’s inside it.

What this does is deploy some standard configurations to each machine; e.g. it ensures my account has a standard $HOME (with my ssh public keys), might add an additional repos (eg EPEL for Rocky), configures postfix for my smarthost, creates email aliases so mail to my account and to root is forwarded properly and so on. Because it’s a full script you can do almost anything; it runs inside the build chroot environment.

An example of this might be my Debian 12 script:

   # Extract all the files I want
   cd /
   /usr/bin/tar xvfp /tmp/files.tar
   /bin/rm /tmp/files.tar
   
   # Remove "quiet" from kernel boot params
   sed -i 's/"quiet"/""/' /etc/default/grub
   
   # Ensure grub is updated with any changes that might have made
   /sbin/update-grub
   
   # Fix my account
   /usr/bin/chsh -s /bin/ksh sweh
   /sbin/usermod -a -G sudo sweh
   /sbin/usermod -a -G root sweh
   
   # Setup postfix correctly
   echo 'root: root@mailhost' >> /etc/aliases
   /usr/bin/newaliases
   
   /usr/sbin/postconf append_dot_mydomain=yes
   /usr/sbin/postconf mydomain=spuddy.org
   /usr/sbin/postconf 'mydestination=$myhostname, localhost.$mydomain, localhost'
   /usr/sbin/postconf 'myorigin=$myhostname'
   /usr/sbin/postconf -X myhostname
   /usr/bin/systemctl enable postfix
   
   # Remove static hostname
   echo "" > /etc/hostname
   
   # Remove this annoying entry from hosts file
   sed -i '/127.0.1.1/d' /etc/hosts
   
   # Fix PAM SSH
   sed -i 's/user_readenv=1//' /etc/pam.d/sshd
   
   echo `date`: Post script completed

The files.tar it refers to is embedded into the post_d12.tar script and contains things such as

   ./etc/apt/apt.conf.d/00aptproxy
   ./etc/sudoers.d/sweh
   ./etc/sysctl.d/00-dmesg.conf
   ./etc/default/grub.d/console.cfg
   ./etc/default/grub.d/apparmor.cfg
   ./etc/logwatch/conf/services/zz-disk_space.conf
   ./etc/ntp.conf
   ./etc/cron.daily/need_patches
   ./home/sweh/.profile
   ./home/sweh/.envfile
   ./home/sweh/.forward
   ./home/sweh/.ssh/authorized_keys
   ./root/.ssh/authorized_keys

(that’s not the complete list, just a sample)

Basically, any “standard” changes you want deployed to all my VMs is performed in this script, and it can be customised to each OS variant.

Conclusion

What I’ve described here is suitable for a homelab. But some of the same concepts can be applied to enterprises as well. For example, that kickstart file could be modified to be regional specific to pick a local replica to get the repos. The post-install script could call out to something like ansbible or puppet to do proper configuration management (even auto install and configure something like Apache web server if the machine is defined as “webserver” in an inventory system). The automated install options provided by Rocky (RedHat) and Debian are very flexible.

In my case, I can just run deb_install -13 new_host and it will create the LV for disk space, and run virt_install with all the required parameters. 5 minutes later I have a new fully patched VM to test and play with, with sane default values.