translatedcode

QEMU for and on ARM cores

Installing Debian on QEMU’s 32-bit ARM “virt” board

with 25 comments

In this post I’m going to describe how to set up Debian on QEMU emulating a 32-bit ARM “virt” board. There are a lot of older tutorials out there which suggest using boards like “versatilepb” or “vexpress-a9”, but these days “virt” is a far better choice for most people, so some documentation of how to use it seems overdue. (I may do a followup post for 64-bit ARM later.)

Update 2017-07-24: I have now written that post about installing a 64-bit ARM guest.

Why the “virt” board?

QEMU has models of nearly 50 different ARM boards, which makes it difficult for new users to pick one which is right for their purposes. This wild profusion reflects a similar diversity in the real hardware world: ARM systems come in many different flavours with very different hardware components and capabilities. A kernel which is expecting to run on one system will likely not run on another. Many of QEMU’s models are annoyingly limited because the real hardware was also limited — there’s no PCI bus on most mobile devices, after all, and a fifteen year old development board wouldn’t have had a gigabyte of RAM on it.

My recommendation is that if you don’t know for certain that you want a model of a specific device, you should choose the “virt” board. This is a purely virtual platform designed for use in virtual machines, and it supports PCI, virtio, a recent ARM CPU and large amounts of RAM. The only thing it doesn’t have out of the box is graphics, but graphical programs on a fully emulated system run very slowly anyway so are best avoided.

Why Debian?

Debian has had good support for ARM for a long time, and with the Debian Jessie release it has a “multiplatform” kernel, so there’s no need to build a custom kernel. Because we’re installing a full distribution rather than a cut-down embedded environment, any development tools you need inside the VM will be easy to install later.

Prerequisites and assumptions

I’m going to assume you have a Linux host, and a recent version of QEMU (at least QEMU 2.6). I also use libguestfs to extract files from a QEMU disk image, but you could use a different tool for that step if you prefer.

Getting the installer files

I suggest creating a subdirectory for these and the other files we’re going to create.

To install on QEMU we will want the multiplatform “armmp” kernel and initrd from the Debian website:

wget -O installer-vmlinuz http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/vmlinuz
wget -O installer-initrd.gz http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/initrd.gz

Saving them locally as installer-vmlinuz and installer-initrd.gz means they won’t be confused with the final kernel and initrd that the installation process produces.

(If we were installing on real hardware we would also need a “device tree” file to tell the kernel the details of the exact hardware it’s running on. QEMU’s “virt” board automatically creates a device tree internally and passes it to the kernel, so we don’t need to provide one.)

Installing

First we need to create an empty disk drive to install onto. I picked a 5GB disk but you can make it larger if you like.

qemu-img create -f qcow hda.qcow2 5G

Now we can run the installer:

qemu-system-arm -M virt -m 1024 \
  -kernel installer-vmlinuz \
  -initrd installer-initrd.gz \
  -drive if=none,file=hda.qcow2,format=qcow,id=hd \
  -device virtio-blk-device,drive=hd \
  -netdev user,id=mynet \
  -device virtio-net-device,netdev=mynet \
  -nographic -no-reboot

(I would have preferred to use QEMU’s PCI virtio devices, but unfortunately the Debian kernel doesn’t support them; a future Debian release very likely will, which would allow you to use virtio-blk-pci and virtio-net-pci instead of virtio-blk-device and virtio-net-device.)

The installer will display its messages on the text console (via an emulated serial port). Follow its instructions to install Debian to the virtual disk; it’s straightforward, but if you have any difficulty the Debian release manual may help.
(Don’t worry about all the warnings the installer kernel produces about GPIOs when it first boots.)

The actual install process will take a few hours as it downloads packages over the network and writes them to disk. It will occasionally stop to ask you questions.

Late in the process, the installer will print the following warning dialog:

   +-----------------| [!] Continue without boot loader |------------------+
   |                                                                       |
   |                       No boot loader installed                        |
   | No boot loader has been installed, either because you chose not to or |
   | because your specific architecture doesn't support a boot loader yet. |
   |                                                                       |
   | You will need to boot manually with the /vmlinuz kernel on partition  |
   | /dev/vda1 and root=/dev/vda2 passed as a kernel argument.             |
   |                                                                       |
   |                              <Continue>                               |
   |                                                                       |
   +-----------------------------------------------------------------------+  

Press continue for now, and we’ll sort this out later.

Eventually the installer will finish by rebooting — this should cause QEMU to exit (since we used the -no-reboot option).

At this point you might like to make a copy of the hard disk image file, to save the tedium of repeating the install later.

Extracting the kernel

The installer warned us that it didn’t know how to arrange to automatically boot the right kernel, so we need to do it manually. For QEMU that means we need to extract the kernel the installer put into the disk image so that we can pass it to QEMU on the command line.

There are various tools you can use for this, but I’m going to recommend libguestfs, because it’s the simplest to use. To check that it works, let’s look at the partitions in our virtual disk image:

$ virt-filesystems -a hda.qcow2 
/dev/sda1
/dev/sda2

If this doesn’t work, then you should sort that out first. A couple of common reasons I’ve seen:

  • if you’re on Ubuntu then your kernels in /boot are installed not-world-readable; you can fix this with sudo chmod 644 /boot/vmlinuz*
  • if you’re running Virtualbox on the same host it will interfere with libguestfs’s attempt to run KVM; you can fix that by exiting Virtualbox

Looking at what’s in our disk we can see the kernel and initrd in /boot:

$ virt-ls -a hda.qcow2 /boot/
System.map-3.16.0-4-armmp-lpae
config-3.16.0-4-armmp-lpae
initrd.img
initrd.img-3.16.0-4-armmp-lpae
lost+found
vmlinuz
vmlinuz-3.16.0-4-armmp-lpae

and we can copy them out to the host filesystem:

$ virt-copy-out -a hda.qcow2 /boot/vmlinuz-3.16.0-4-armmp-lpae /boot/initrd.img-3.16.0-4-armmp-lpae .

(We want the longer filenames, because vmlinuz and initrd.img are just symlinks and virt-copy-out won’t copy them.)

An important warning about libguestfs, or any other tools for accessing disk images from the host system: do not try to use them while QEMU is running, or you will get disk corruption when both the guest OS inside QEMU and libguestfs try to update the same image.

Running

To run the installed system we need a different command line which boots the installed kernel and initrd, and passes the kernel the command line arguments the installer told us we’d need:

qemu-system-arm -M virt -m 1024 \
  -kernel vmlinuz-3.16.0-4-armmp-lpae \
  -initrd initrd.img-3.16.0-4-armmp-lpae \
  -append 'root=/dev/vda2' \
  -drive if=none,file=hda.qcow2,format=qcow,id=hd \
  -device virtio-blk-device,drive=hd \
  -netdev user,id=mynet \
  -device virtio-net-device,netdev=mynet \
  -nographic

This should boot to a login prompt, where you can log in with the user and password you set up during the install.

The installation has an SSH client, so one easy way to get files in and out is to use “scp” from inside the VM to talk to an SSH server outside it. Or you can use libguestfs to write files directly into the disk image (for instance using virt-copy-in) — but make sure you only use libguestfs when the VM is not running, or you will get disk corruption.

Advertisements

Written by pm215

November 3, 2016 at 10:33 pm

Posted in linaro, qemu

25 Responses

Subscribe to comments with RSS.

  1. Hi thanks so much for this tutorial. I am really interested in the virt board’s possibilities but not finding much info (as the main qemu page just points to this article for now). For versatilepb we have the following: http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0224i/index.html but would you know if there anything for virt — or is it known as another name? As I’m starting out I was really hoping to have a reference doc. Thanks!

    Ben

    November 15, 2016 at 12:15 am

    • There isn’t any specific documentation on exactly what the virt board has in it, I’m afraid. In general the idea is that the virt board will autogenerate a device tree (or ACPI tables) which tell the guest OS what components it has in it, and the guest OS will read that and do the right thing. Since (unlike versatilepb) it isn’t a model of a specific piece of real-world hardware there’s no third-party documentation the way there is for versatilepb.

      pm215

      November 27, 2016 at 5:56 pm

  2. I was trying to run qemu for raspberry PI 3 (which apprears to not be supported). I am following a youtube video and when I try to edit a file in /arm, it does not exist. Any suggestions, etc? This is the command I am trying to run from youtube.

    sudo nano /home/pi/raspidev/qemu/tcg/arm/tcg-targe­t.c

    youtube vid

    Really just trying to run WindowsXP or some other windows emulated OS.

    Steve Klise

    December 29, 2016 at 4:48 pm

    • The file has moved in the QEMU source tree and is now at tcg/arm/tcg-target.inc.c. Things do move around in the QEMU sources so if you’re following instructions on how to edit things you need to adjust for changes that have happened since the instructions were written.

      In this case the function the instructions want you to edit has been removed entirely. I would suggest just building QEMU without trying to edit any of its source files. If the compilation fails with an error message then let me know, because we can fix that bug in QEMU (but it may have been fixed already).

      PS: as those instructions note, it’s likely that the resulting setup will be too slow to run Windows usefully anyway.

      pm215

      December 30, 2016 at 11:43 am

      • AM now getting this. I was able to edit the files………..
        ————————————————————————————————————————————–

        In file included from /usr/include/string.h:640:0,
        from /home/pi/xpstuff/qemu/slirp/slirp.h:66,
        from slirp/misc.c:8:
        In function ‘memset’,
        inlined from ‘slirp_connection_info’ at slirp/misc.c:401:9:
        /usr/include/arm-linux-gnueabihf/bits/string3.h:81:7: error: call to ‘__warn_memset_zero_len’ declared with attribute warning: memset used with constant zero length parameter; this could be due to transposed parameters [-Werror]
        __warn_memset_zero_len ();
        ^
        cc1: all warnings being treated as errors
        /home/pi/xpstuff/qemu/rules.mak:18: recipe for target ‘slirp/misc.o’ failed
        make: *** [slirp/misc.o] Error 1
        pi@raspberrypi:~/xpstuff/qemu $ sudo make install
        CC slirp/misc.o
        In file included from /usr/include/string.h:640:0,
        from /home/pi/xpstuff/qemu/slirp/slirp.h:66,
        from slirp/misc.c:8:
        In function ‘memset’,
        inlined from ‘slirp_connection_info’ at slirp/misc.c:401:9:
        /usr/include/arm-linux-gnueabihf/bits/string3.h:81:7: error: call to ‘__warn_memset_zero_len’ declared with attribute warning: memset used with constant zero length parameter; this could be due to transposed parameters [-Werror]
        __warn_memset_zero_len ();
        ^
        cc1: all warnings being treated as errors
        /home/pi/xpstuff/qemu/rules.mak:18: recipe for target ‘slirp/misc.o’ failed
        make: *** [slirp/misc.o] Error 1
        pi@raspberrypi:~/xpstuff/qemu $

        Steve Klise

        December 30, 2016 at 5:05 pm

        • You seem to be trying to build a really ancient version of QEMU — that compile issue was fixed in QEMU upstream four years ago, and looking at the tarball the video instructions reference it is a five year old random development snapshot. Trying to build something that old you’re likely to run into other problems that have already been fixed. I suggest you try a newer release instead.

          pm215

          December 30, 2016 at 6:29 pm

  3. For my convenience I added a new VM for the ARM image to the Virtual Machine Manager. At step 1 select “Import existing disk image”, then in “Architectural Options” select “Arm”. “Machine Type will be “virt”. Click “Forward”. In step 2, enter the location of the image and its name in the first box. Add the paths and names of the initrd and kernel in the appropriate boxes and add “root=/dev/vda2” in the Kernel args box. DTB path can be left empty. Click “Forward”. In step 3 set the RAM and CPUs and click “Forward”. In step 4 check the box “Customize configuration before install” and click “Finish”. In the dialog that pops up select “SCSI Disk 1”. Open the Advanced options and click the drop down where it says “Disk bus”. Select “VirtIO”, then click “Apply”. Click “Begin Installation” at the top and the Virtual Machine Manager will create an entry for your machine and boot it. It all seems to work after that but some tweaking may be required.

    Cliff Pratt

    January 15, 2017 at 10:40 pm

  4. When I run the installer command, I get no output at all, as if the console was not properly redirected

    You need to add -append “console=ttyAMA0”

    pock

    February 14, 2017 at 9:37 am

    • Hmm, pretty sure it worked without for me, but certainly adding that append line won’t hurt.

      pm215

      February 14, 2017 at 9:58 am

  5. You mention that you would prefer to use virtio-pci devices, but that this is currently not supported by the debian kernel. Could you explain why this is the case?
    From my own experiments I also found that the debian kernel fails to recognize any PCI-devices of the qemu’s “virt” machine, however I was only able to find your blog post mentioning the problem.

    (There was also a comment in qemu’s 2.5 release notes that some kernels without CONFIG_LPAE=y will fail to detect the PCI controller if the flag “-machine highmem=off” was not used. However, neither setting the flag nor using a LPAE-kernel seems to help.)

    Best,
    Johannes

    Johannes Krupp

    February 22, 2017 at 12:12 pm

    • It’s just that the changes needed to get PCI supported went into the kernel relatively recently, and Debian’s release cycles mean that their stable kernel is just older than the necessary support/config changes. I expect that when the next Debian release happens (very shortly) PCI should just work, though I haven’t tested. (You could also try using Debian ‘testing’ for the moment.)

      pm215

      February 22, 2017 at 12:31 pm

  6. I’am very interesting for the 64 version of this tutorial. ; )
    Coming soon ?

    snak3xe

    July 20, 2017 at 12:24 pm

    • Heh, I’d forgotten I’d said I would do the 64-bit version. I’ll try to get to that this week or next…

      pm215

      July 20, 2017 at 12:56 pm

      • Thanks a lot. ; )

        snak3xe

        July 20, 2017 at 1:42 pm

        • I’ve now published the 64-bit install instructions post.

          pm215

          July 24, 2017 at 10:32 am

          • Do you have a quickly solution to bridge a network interface directly to the guest and not system dependant? I try the bridge feature of qemu but not work.

            On the host:

            # ip link add br0 type bridge

            # ip addr flush dev eth0

            # ip link set eth0 master br0

            and I add this to the qemu command line

            -netdev bridge,id=hn0
            -device virtio-net-device,netdev=hn0,id=nic1

            # ip link

            3: eth0: mtu 1500 qdisc fq_codel master br0 state UP mode DEFAULT group default qlen 1000
            link/ether 20:47:47:bb:51:ca brd ff:ff:ff:ff:ff:ff
            34: br0: mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
            link/ether 20:47:47:bb:51:ca brd ff:ff:ff:ff:ff:ff
            35: tap0: mtu 1500 qdisc fq_codel master br0 state UNKNOWN mode DEFAULT group default qlen 1000
            link/ether fe:23:18:f4:c6:31 brd ff:ff:ff:ff:ff:ff

            Next in the guest, I have an eth0 network interface, but dhcpclient fail.

            At boot time:

            [* ] A start job is running for LSB: Raise network interf…38s / no limit

            [FAILED] Failed to start LSB: IPv4 DHCP client with IPv4LL support.
            See ‘systemctl status dhcpcd.service’ for details.

            # systemctl status dhcpcd.service -l

            Jul 25 20:29:17 solid dhcpcd[276]: Not running dhcpcd because /etc/network/interfaces … failed!
            Jul 25 20:29:17 solid dhcpcd[276]: defines some interfaces that will use a DHCP client … failed!
            Jul 25 20:29:17 solid systemd[1]: dhcpcd.service: control process exited, code=exited status=6
            Jul 25 20:29:17 solid systemd[1]: Failed to start LSB: IPv4 DHCP client with IPv4LL support.
            Jul 25 20:29:17 solid systemd[1]: Unit dhcpcd.service entered failed state.

            # cat /etc/network/interfaces

            source /etc/network/interfaces.d/*

            auto lo
            iface lo inet loopback

            allow-hotplug eth0
            iface eth0 inet dhcp

            With dhcpcd:

            # dhcpcd eth0
            dhcpcd[560]: version 6.0.5 starting
            dhcpcd[560]: eth0: soliciting an IPv6 router
            dhcpcd[560]: eth0: using IPv4LL address 169.254.41.139
            dhcpcd[560]: eth0: adding host route to 169.254.41.139 via 127.0.0.1
            dhcpcd[560]: eth0: adding route to 169.254.0.0/16
            dhcpcd[560]: forked to background, child pid 583

            snak3xe

            July 25, 2017 at 7:40 pm

            • Afraid not — I’ve never used anything except ‘user’ networking. There should be no difference on ARM to getting this working with an x86 VM, though, so there should be plenty of tutorials out there.

              pm215

              July 25, 2017 at 8:32 pm

            • UPDATE

              I’am on Archlinux and I have fix the problem with the bridge interface with the following iptables rules:

              # iptables -A INPUT -i tap0 -j ACCEPT
              # iptables -A INPUT -i br0 -j ACCEPT
              # iptables -A FORWARD -i br0 -j ACCEPT

              snak3xe

              July 27, 2017 at 8:28 pm

  7. yeh ok

    snak3xe

    July 25, 2017 at 10:22 pm

  8. Hi, and thank you very much for this tutorial!
    Just as a reference for any other newbie (like me) in the Linux development, I’ve faced some issues with extracting kernel and initrd from virtual image file to host filesystem.
    Finally I solved using these 2 commands:

    $ virt-copy-out -a hda.qcow2 /boot/vmlinuz-3.16.0-4-armmp-lpae
    $ virt-copy-out -a hda.qcow2 /boot/initrd.img-3.16.0-4-armmp-lpae

    where is he folder where I’ve created the virtual image file hda.qcow2.

    mattia781

    August 31, 2017 at 10:06 am

    • Commands are:

      $ virt-copy-out -a hda.qcow2 /boot/vmlinuz-3.16.0-4-armmp-lpae local-host-folder
      $ virt-copy-out -a hda.qcow2 /boot/initrd.img-3.16.0-4-armmp-lpae local-host-folder

      where local-host-folder is the folder where I’ve created the virtual image file hda.qcow2

      mattia781

      August 31, 2017 at 10:08 am

  9. A question: is it possible to open a second terminal of the virtual machine?
    Using keys ctrl+shift+t a terminal of the host system is opened; using xdotool to emulate this same key sequence, I obtain the error “Can’t open diplay: (null) Failed creating new xdo instance”.

    mattia781

    September 1, 2017 at 1:16 pm

    • It’s a serial terminal, not a graphical window system, so you can’t just open new terminals like that. Probably the best approach is to run an ssh daemon inside the VM, configure QEMU’s networking to forward from (say) host port 2222 to VM port 22 (which is ssh), and then ssh to the host port 2222 to get another login.

      pm215

      September 1, 2017 at 1:34 pm

      • It works: thanks!

        mattia781

        September 1, 2017 at 3:05 pm


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s