Considerations

  • Ubuntu Server 24.04.2 (x64) is used as a guest system
  • Host system: Linux (x64). At the moment of writing the article I used Fedora Workstation 42
  • At some point we will need Docker to compile a binary to handle one issue

The roadmap

  1. Compile the kernel
  2. Build initial RAM disk
    • Compile Busybox
    • Make an initial RAM disk file a.k.a. initrd
  3. Install the OS in virtual machine
  4. Boot the virtual machine with compiled kernel + initial RAM disk
  5. Attach GDB to running machine

Compilation of Linux Kernel

Prerequisites

You will need:

  • C compiler (gcc), of course
  • flex and bison binaries. They are most likely installed to your system, at least I didn’t have to install them. I accidentally discovered that these applications are needed when tried to compile the kernel in Docker
  • Development version of ncurses library to render the configuration menu when running make menuconfig. In Fedora in can be installed via dnf install ncurses-devel. If you prefer to edit the config file manually instead of uring the menu, this step might be skipped.
  • GNU make utility

How to obtain the code

The code can be pulled from Github mirror

git clone --depth=1 https://github.com/torvalds/linux.git # or
git clone --depth=1 git@github.com:torvalds/linux.git

…or from kernel.org, if you prefer the original source:

git clone --depth=1 git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

You can omit the depth=1 flag if you would like to pull the whole history, but it will take longer time. This would be the Kernel source directory.

Configuration

We will need to compile a kernel which is compatible with QEMU and which has debug information for GDB. Navigate to Kernel source directory and run:

make x86_64_defconfig
make kvm_guest.config

These commands will create a basic .config file which is configured for x64 system and which has components for KVM virtualization that we will use in QEMU. We only need to enable the debug information in the kernel:

make menuconfig

This will display a TUI menu where you need to navigate to Kernel hacking (it must be the last item). Verify if Kernel debugging is enabled. Then go to Compile-time checks and compiler options > Debug information sub-level and set the Generate DWARF Version 5 debuginfo option. Now you can save the config and exit from configuration utility.

Rename the kernel (optional)

How cool it will be to have your name in the kernel info? In addition, it will be a way to ensure that the loaded kernel is your custom one. To set a custom version name, you can edit the Makefile and assign some suffix to EXTRAVERSION parameter in beginning of the file. The initial value is something like -rc4, so you can make it -john-doe-edition. It will be observable via uname -a in the future.

Building the kernel

make -j$(nproc)

The -j$(nproc) argument will resolve to -j<number_of_your_cpu_cores> which will significantly improve the time of compilation. Compilation might be time consuming, especially if your CPU doesn’t have manu cores. It might take even hours if you skip the CPU number parameter. But in my case (16 cores) it took probably 10 minutes.

When the compilation completes, there should be output like

Kernel: arch/x86/boot/bzImage is ready

This is your kernel. Just to clarify - there is also arch/x86_64/boot/bzImage file, but it is just a symlink to arch/x86/boot/bzImage.

Making initrd

Linux kernel needs an initial filesystem to bootstrap, so let’s build it.

Compilation of Busybox

Busybox combines lots of basic utilities in one binary file, that’s why it’s so useful to bootstrap the Linux system. Our goal is to include Busybox into initial ramdisk.

Note: you can actually borrow the ready-to-use Busybox binary from your host system. Use ‘which busybox’ to check it in your system. If it’s not there, you can look it up from your package manager. It’s important that it was statically linked. If you go this way, you can just copy it to the source directory (see below). But I still suggest to pull the project and to run “make install” - after copying binary to the project, because it will create important symlinks around Busybox. If you decided to go hard way and compile Busybox, ignore this note.

The source code of Busybox could be pulled from its official repository:

git clone --depth=1 git://busybox.net/busybox.git

Let’s call the result directory as Busybox source directory.

In Busybox source directory, execute

make menuconfig

Navigate to settings and set the Build static binary (no shared libs) flag, it must be approximately in the middle of menu. Now Busybox is ready for compilation.

Unfortunalely, there are some issues with compilation of Busybox on desktop system, at least on my Fedora setup. At least it drops

networking/tc.c: In function ‘cbq_print_opt’:
networking/tc.c:236:27: error: ‘TCA_CBQ_MAX’ undeclared (first use in this function); did you mean ‘TCA_CBS_MAX’?
  236 |         struct rtattr *tb[TCA_CBQ_MAX+1];

Even with having TC setting disabled, it drops tons of errors:

/usr/bin/ld: cannot find -lm: No such file or directory
/usr/bin/ld: have you installed the static version of the m library ?
/usr/bin/ld: cannot find -lresolv: No such file or directory
/usr/bin/ld: have you installed the static version of the resolv library ?

As a workaround, I would suggest to build the application inside Docker container. Execute this command in Busybox source directory:

docker run --rm -it \
  -u "$(id -u):$(id -g)" \
  -v $PWD:/build/busybox \
  -w /build/busybox \
  gcc:15.1.0-bookworm \
  sh -c "make clean && make -j$(nproc) && make install"

This command will mount the Busybox source directory into /build/busybox inside Docker container and will build Busybox there using official GCC Docker image. The expected output would be like

# more text here
  ./_install//usr/sbin/ubiupdatevol -> ../../bin/busybox
  ./_install//usr/sbin/udhcpd -> ../../bin/busybox


--------------------------------------------------
You will probably need to make your busybox binary
setuid root to ensure all configured applets will
work properly.
--------------------------------------------------

The outcome will be:

  • busybox binary
  • _install directory which we will need later

Now let’s make initrd

Create a new directory (we will refer it as Initrd source directory). Then create an init file with the following contents:

#!/bin/sh

mount -t proc none /proc
mount -t sysfs none /sys
mount -t devtmpfs none /dev

# # Note: this might be useful for troubleshooting in case if the guest Linux OS will not load
# # Use it instead of the code below to load basic system in RAM
# cat <<!
# Welcome!
# Boot took $(cut -d' ' -f1 /proc/uptime) seconds
# !
# exec /bin/sh

# This code mounts the Ubuntu partition into /mnt and switches to it
mkdir /mnt
mount /dev/vda2 /mnt
exec switch_root /mnt /sbin/init

This file will be loaded on system boot.

Now create a directory with the structure of RAM filesystem; replace the !busybox_src! with path to Busybox source directory:

# Create a rootfs directory with contents
mkdir -pv rootfs/{bin,sbin,aetc,proc,sys,usr/{bin,sbin},dev}
# Copy the binaries into rootfs directory
cp -av !busybox_src!/_install/* rootfs/
# Copy the init file into rootfs directory and make it executable
cp init rootfs/init
chmod +x rootfs/init

Here is expected structure of subdirectories:

# Sub-dirs
$ tree rootfs -L1
rootfs
├── bin
├── dev
├── etc
├── init
├── linuxrc -> bin/busybox
├── proc
├── sbin
├── sys
└── usr

# non-exhaustive list of files in bin directory
tree rootfs | head -n 7
rootfs
├── bin
│   ├── arch -> busybox
│   ├── ash -> busybox
│   ├── base32 -> busybox
│   ├── base64 -> busybox
│   ├── busybox

Finally, create a rootfs.cpio.gz file with the contents of rootfs:

# Navigate to rootfs directory
cd rootfs
find . -print0 | cpio --null -ov --format=newc | gzip -9 > ../rootfs.cpio.gz

Setting up the virtual machine

Let’s create yet another new directory which is Guest OS directory. It will contain files which are necessary for installation of Ubuntu Server into QEMU virtual machine. Go to this directory and download an ISO image of Ubuntu Server:

wget https://releases.ubuntu.com/24.04.2/ubuntu-24.04.2-live-server-amd64.iso

Now let’s create a virtual hard drive where we will install Ubuntu to:

qemu-img create ubuntu_hdd.qcow2 10G -f qcow2

This will create a ubuntu_hdd.qcow2 file which will be a 10 gigabyte disk in qcow format.

Finally, start a virtual machine with ISO image + virtual disk mounted. !guest_dir! should be replaced with path to Guest OS directory.

qemu-system-x86_64 \
    -enable-kvm \
    -m 2048 \
    -cpu host \
    -smp 2 \
    -drive file=!guest_dir!/ubuntu_hdd.qcow2,format=qcow2,if=virtio \
    -cdrom !guest_dir!/ubuntu-24.04.2-live-server-amd64.iso \
    -net nic \
    -net user \
    -vga std \
    -boot d

This command will open a window with virtual machine which uses 2 CPU cores (-smp 2), 2GB RAM (-m 2048) and network of the host machine. It also boots from ISO image (-boot d flag). Proceed with installation of Ubuntu.

NB! When configuring disk partitions, do NOT enable the disk encryption (the Set up this disk as LVM group option), because you will not be able to mount it later. Also this tutorial implies using default settings for disk partitioning, so I would recommend to use the default settings apart from notice about encryption above.

When installer requests to reboot the system, you can just close the window. To verify the installation, run the previous command without -cdrom and -boot flags:

qemu-system-x86_64 \
    -enable-kvm \
    -m 2048 \
    -cpu host \
    -smp 2 \
    -drive file=!guest_dir!/ubuntu_hdd.qcow2,format=qcow2,if=virtio \
    -net nic \
    -net user \
    -vga std

Now the system must boot from virtual disk.

To verify if virtual disk had been created correctly, check the list of disks (df -h) after login to the guest system. There must be /dev/vda2 disk mounted to /. If there are paths like /dev/mapper/.. then you most likely missed the previous “NB” block and created the encrypted disk.

Booting the compiled kernel

Now let’s adjust the previous command to boot your kernel. Please don’t forget to replace:

  • !guest_dir! > Guest OS directory
  • !initrd_dir! > Initrd source directory
  • !kernel_dir! > Kernel source directory
qemu-system-x86_64 \
    -enable-kvm \
    -m 2048 \
    -cpu host \
    -smp 2 \
    -drive file=!guest_dir!/ubuntu_hdd.qcow2,format=qcow2,if=virtio \
    -append "root=/dev/vda2 console=ttyS0 earlyprintk=vga debug initcall_debug nokaslr net.ifnames=0 biosdevname=0" \
    -initrd !initrd_dir!/rootfs.cpio.gz \
    -kernel !kernel_dir!/arch/x86_64/boot/bzImage \
    -vga std

If everything went well, it should start as usual. Now, if you type:

uname -a

it should display the version of compiled kernel. If you added some your custom input, as suggested in beginning of article, it should be printed too.

Debugging the kernel

The last step is to enable debugging in the running machine:

qemu-system-x86_64 \
    -enable-kvm \
    -m 2048 \
    -cpu host \
    -smp 2 \
    -drive file=!guest_dir!/ubuntu_hdd.qcow2,format=qcow2,if=virtio \
    -append "root=/dev/vda2 console=ttyS0 earlyprintk=vga debug initcall_debug nokaslr net.ifnames=0 biosdevname=0" \
    -initrd !initrd_dir!/rootfs.cpio.gz \
    -kernel !kernel_dir!/arch/x86_64/boot/bzImage \
    -vga std \
    -s -S

-s -S flags start a GDB server at port 1234 and halt the machine on startup. After running command, you will see a message:

Guest has not initialized the display (yet)

So execution had stopped even before display was initialized. Now in Kernel source directory run this:

gdb -tui vmlinux

Inside GDB session, let’s connect to GDB server which we had started on port 1234 and add a hardware-assisted breakpoint on start_kernel function:

target remote :1234
hb start_kernel

After setting a breakpoint, you can resume the execution by sending c command to GDB. Some text will be printed in QEMU window and GDB will freeze in beginning of start_kernel function. Send a n command a couple of times to execute few lines of the code. Your GDB and QEMU windows will look like this:

QEMU and GDB windows

If you have approximately the same result, then congratulations, you can debug the kernel now!

P.S.

Before publishing the article, I tried these commands once again in a temporary directory (I just did it in Downloads). This is a structure of directories which was sufficient for me to compile the whole thing:

$ tree -L1
.
├── 00_linux_kernel
├── 01_busybox
├── 02_initrd
└── 03_ubuntu

Kudos