Recently I had to track down a minor bug in the OpenBSD kernel. I tapped QEMU and GDB as debugging tools for the task, running on Ubuntu 12.04 as the host OS. This combination worked extremely well, so for the record here’s how I set it all up.

OpenBSD comes equipped with two kernel debugging mechanisms: ddb and kgdb. ddb(4) is an in-kernel debugger, enabled by default in the GENERIC kernel, and can be invoked either explicitly from the console or automatically in the event of a panic. It is analogous to the Linux debugger kdb in that it can be used to set breakpoints and examine the stack or register state, but (like kdb) it is not a source-level debugger.

For source debugging there is kgdb(7), which offers the ability to remotely debug the kernel by way of a GDB stub running over a serial port; this is similar to the Linux debugger kgdboc. However, kgdb it is not available in the GENERIC kernel, and it imposes an additional set of configurations and debugger latencies on the user. If your debugging task is amenable to running OpenBSD within a virtual machine, as mine was, then there is an easier and better way…

The VM monitor QEMU provides its own GDB stub useful for debugging virtualized systems. For source-level debugging this still requires obtaining a kernel compiled with the appropriate debug info, but the stub makes no additional requirements of the guest OS and performs very well. The necessary QEMU tools can be installed on Ubuntu 12.04 as follows:

$ sudo apt-get install qemu-kvm qemu-utils

To start with, you’ll need an OpenBSD installation in a QEMU virtual machine. Prepare your virtual disk image (20GB will be more than enough):

$ qemu-img create -f qcow2 ~/obsd52.qcow2 20G

Next, download the OpenBSD i386 install CD image from a mirror, such as:

ftp://ftp3.usa.openbsd.org/pub/OpenBSD/5.2/i386/install52.iso

Now boot a VM from the OpenBSD installation CD, with the QCOW2 image that you just created as your virtual disk image. Aside from setting a system RAM size of 256 MB, the options specified here will provide a userspace NAT virtual network interface for the VM:

$ qemu-system-i386 -m 256M -net nic -net user -cdrom ~/install52.iso \
      ~/obsd52.qcow2

Wait for the CD image to boot and then install OpenBSD in the usual manner. When prompted, specify that the SSH server should be started by default, as this will be needed shortly. Reboot into the new guest OS when the installer has finished.

OpenBSD isn’t yet ready for debugging, though – although the system boots, its kernel does not provide the debug symbols needed by GDB. So on the guest system now, download and install the system kernel source code from an FTP mirror:

obsd# cd /tmp
obsd# ftp ftp://ftp3.usa.openbsd.org/pub/OpenBSD/5.2/sys.tar.gz
obsd# cd /usr/src
obsd# tar xzf /tmp/sys.tar.gz

Make a new kernel configuration based on the i386 GENERIC config, then edit this config to turn on debug symbol generation by adding the line makeoptions DEBUG="-g" under the option lines:

obsd# cd /usr/src/sys/arch/i386/conf
obsd# cp GENERIC DEBUG
obsd# vi DEBUG

Now you can build and install the new kernel. Specifying the COPTS environment variable here overrides the default optimization level -O2, which will make control flow much easier to follow while single-stepping in the debugger:

obsd# config DEBUG
obsd# cd ../compile/DEBUG
obsd# make depend
obsd# COPTS="-O0" make
obsd# make install

Halt the OpenBSD guest and exit QEMU, and then restart the VM with additional options -s and -redir, to enable the GDB stub and access to the VM’s SSH service respectively:

$ qemu-system-i386 -m 256M -net nic -net user -s \
      -redir tcp:2022::22 ~/obsd52.qcow2

When the guest is booted, SCP from the host to copy the kernel source tree and build output.

$ scp -pr -P2022 localhost:/usr/src/sys ~/obsd52-sys

With debug output enabled the OpenBSD kernel build produces an unstripped ELF kernel image named bsd.gdb, which is suitable for use by GDB. Load this into GDB and then attach to QEMU’s debug stub (which defaults to TCP port 1234):

$ cd ~/obsd52-sys/arch/i386/compile/DEBUG
$ gdb bsd.gdb
(gdb) target remote :1234

The VM will halt when you connect the debugger, leaving you free to set system call breakpoints or inspect state before continuing.