Tricks for Debugging QEMU — savevm snapshots

For the next entry in this occasional series of posts about tricks for debugging QEMU I want to talk about savevm snapshots.

QEMU’s savevm snapshot feature is designed as a user feature, but it’s surprisingly handy as a developer tool too. Suppose you have a guest image which misbehaves when you run a particular userspace program inside the guest. This can be very awkward to debug because it takes so long to get to the point of failure, especially if it requires user interaction along the way. If you take a snapshot of the VM state just before the bug manifests itself, you can create a simpler and shorter test case by making QEMU start execution from the snapshot point. It’s then often practical to use debug techniques like turning on QEMU’s slow and voluminous tracing of all execution, now that you’re only dealing with a short run of execution.

To use savevm snapshots you’ll need to be using a disk image format which supports them, like QCOW2. If you have a different format like a raw disk, you can convert it with qemu-img:

qemu-img convert -f raw -O qcow2 your-disk.img your-disk.qcow2

and then change your command line to use the qcow2 file rather than the old raw image. (As a bonus it should be faster and take less disk space too!)

If the QEMU system you’re trying to debug doesn’t have a disk image at all, you can create a dummy disk which will be used for nothing but snapshots like this:

qemu-img create -f qcow2 dummy.qcow2 32M

and then add this option to your command line:

-drive if=none,format=qcow2,file=dummy.qcow2

(QEMU may warn that the drive is “orphaned” because it’s not connected to anything, but that’s fine.)

To create a snapshot, you use this QEMU monitor command:

savevm some-name

This will save the VM state, and usually takes a second or two. Once it’s done you can type quit at the monitor to exit QEMU. You can make multiple snapshots with different names.

Then to make QEMU start automatically from the snapshot add the option:

-loadvm some-name

to your QEMU command line. (You still need to specify all the same device and configuration options you did when you saved the snapshot.)

Before you dive into debugging your reduced test case, do check that the bug you’re reproducing is still present in the shortened test case. Some bugs don’t reproduce in a snapshot — for instance if the problem is that QEMU has stale information cached in its TLB or translated code cache, then the bug will probably not manifest when the snapshot is loaded, because these caches will be empty. (Not reproducing in a snapshot is interesting diagnostic information in itself, in fact.)

You should also be aware that snapshotting requires support from all the devices in the system QEMU is modelling. This works fine for the x86 PC models, and also for most of the major ARM boards (including ‘virt’, ‘vexpress’ and ‘versatilepb’), but if you’re trying this on a more obscure guest CPU architecture or board you might run into trouble. Missing snapshotting support will manifest as the reloaded system misbehaving (eg device stops working, or perhaps there are no interrupts so nothing responds). I think this debugging technique is valuable enough that it’s worth stopping to fix up missing snapshot support in devices just so you can use it. If you don’t feel up to that, feel free to report the bugs on qemu-devel…

You can automate the process of taking the initial snapshot using the ‘expect’ utility. Here are some command line options that create a monitor session on TCP port 4444 and make QEMU start up in a ‘stopped’ state, so the VM doesn’t run until we ask it to:

-chardev socket,id=monitor,host=127.0.0.1,port=4444,server,nowait,telnet -mon chardev=monitor,mode=readline -S

And here’s an expect script that connects to the monitor, tells QEMU to start, and then takes a snapshot 0.6 seconds into the run:

#!/usr/bin/expect -f

set timeout -1
spawn telnet localhost 4444
expect "(qemu)"
send "c\r"
sleep 0.6
send "savevm foo\r"
expect "(qemu)"
send "quit\r"
expect eof

I used this recently to debug a problem in early boot that was causing a hang — by adjusting the timeout I was able to get a snapshot very close to the point where the trouble occured. Even a second of execution can generate enough execution trace to be unmanageable…

Snapshots won’t solve your debugging problem all on their own, but they can cut the problem down to a size where you can apply some of the other tools in your toolbox.