Friday, May 31, 2013

Running Debian MIPS Linux in QEMU

Sometimes I need a MIPS Linux system that I can use for development and testing.  Maybe I need to test some shellcode or debug a binary I'm analyzing.  What I wish existed was a Raspberry Pi-like MIPS device.  I'd love to have a bunch of small, sub-$50 devices that I could network together as a sort of desktop exploit lab.  Unfortunately I don't know of such a device.  There is MIPS hardware you can get and install Linux on.  I have a Cobalt RAQ, and a lot of people like to get a hackable WiFi router and image it with OpenWRT.  But there's nothing as small, cheap and convenient as the RPi that is MIPS.

What some of us at Tactical Network Solutions use instead is the QEMU emulator running Debian MIPS Linux.  It works really well for most things, and isn't too bad to get set up.  You can apt-get install whatever utilities you want, including gcc, gdb, and other things.  This is a tutorial on getting Debian MIPS up and running in QEMU hosted on Ubuntu Linux.

Here are some details of my setup:

  • Ubuntu Linux 13.04 64-bit
  • Running in VMWare Fusion on OS X 10.8
  • Ubuntu's networked via VMWare's NAT interface
  • I export my workspace from my Mac to VMs via NFS

If your setup is different, hopefully this tutorial will still get you going.  I think as long as you're using Ubuntu underneath QEMU, you should be good.

Download, Patch, and Build QEMU


First we download and build QEMU from source.  There are a number of advantages for building from source vs. installing from Ubuntu's repositories.  Ubuntu's prebuilt QEMU is pretty dated.  Plus, we can ensure we get statically built QEMU binaries which are useful if you want to chroot into a firmware's filesystem and emulate a binary (I won't cover that in this tutorial).  There also is a bug in QEMU that we can easily patch before building.

Download the latest QEMU source tarball from http://wiki.qemu.org/Download.  I downloaded version 1.5.0.  I don't recommend checking out from the git repository since the development tree is likely to be unstable.

In order to build QEMU you need to install several dependencies:


# apt-get install autoconf automake libtool \
   zlib1g-dev libglib2.0-dev

Unpack the QEMU tarball and edit its Makefile.  We need to change the build for libpixman to ensure 'aclocal' gets run before libpixman is built (this wasn't neccesary for 1.4.0, but it is for 1.4.2 and 1.5.0.).  This step is necessary on Ubuntu but I'm not sure about other systems.

Look for the following part of QEMU's top-level Makefile:


pixman/Makefile: $(SRC_PATH)/pixman/configure
        (cd pixman; CFLAGS="$(CFLAGS) -fPIC $(extra_cflags) $(extra_ldflags)" $(SRC_PATH)/pixman/configure $(AUTOCONF_HOST) --disable-gtk --disable-shared --enable-static)

Change it to look like this:


pixman/Makefile: $(SRC_PATH)/pixman/configure
        (cd pixman; aclocal; CFLAGS="$(CFLAGS) -fPIC $(extra_cflags) $(extra_ldflags)" $(SRC_PATH)/pixman/configure $(AUTOCONF_HOST) --disable-gtk --disable-shared --enable-static)

Notice the addition of 'aclocal' after cd'ing into pixman/.  I wish I could explain what this fixes, but autoconf is voodoo, so that's not possible.  Anyway, I couldn't get QEMU to build without hacking the Makefile.

While we're hacking things up, let's edit the source file linux-user/elfload.c.  There's a bug in QEMU where the user-mode emulators will choke on binaries that have their section headers stripped.  Many firmware binaries do this to save space.  Since section headers don't get used by the elf loader, there's no need for QEMU to check them.  Thanks to Craig Heffner for this tip.  Find the following function and comment out the line shown:


static bool elf_check_ehdr(struct elfhdr *ehdr)
{
    return (elf_check_arch(ehdr->e_machine)
            && ehdr->e_ehsize == sizeof(struct elfhdr)
            && ehdr->e_phentsize == sizeof(struct elf_phdr)
            //&& ehdr->e_shentsize == sizeof(struct elf_shdr)
            && (ehdr->e_type == ET_EXEC || ehdr->e_type == ET_DYN));
}

With these two modifications in place, we can now build QEMU:

$ configure --static
$ make
# make install #as root


Download Debian QEMU Images


This is going to take a while.  Drink a few beers while it builds.  If you're building on a multiprocessor machine, you may need to drink a couple beers at once.  When it's all done, you'll have system and user-mode emulators for a ton of architectures including MIPS.

While the above is compiling and you're drinking your beers, you can also be downloading your Debian QEMU images.  You can get them from http://people.debian.org/~aurel32/qemu/.  The 'mips' directory is big endian MIPS and the 'mipsel' directory is little endian.  I downloaded both, and this tutorial assumes that's what you're doing as well.

The 2.6.32 kernel goes with the Debian Squeeze qcow image.  There's also an older kernel and a Debian Lenny image if you want that.  Sadly 2.6.32 and Squeeze are the newest AurĂ©lien has up.

I put my images in ~/debian-qemu/be and ~/debian-qemu/le.   Put them where you like, but be sure to adjust the scripts I provide accordingly.


Set up Bridged Networking


In order to have your QEMU systems bridged to Ubuntu's network, you'll need to do some additional setup.  There are some utilities you need to install:


# apt-get install uml-utilities bridge-utils

You need to add a group that will own the tap that will be used to bridge each QEMU system to Ubuntu's network.  Then add your user to that group.

# groupadd -r tuntap
# usermod -a -G tuntap zach

Log out and back in for group membership to take effect.

Here's what my /etc/network/interfaces file looks like:

# interfaces(5) file used by ifup(8) and ifdown(8)
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp

#auto br0
iface br0 inet dhcp
 pre-up tunctl -t tap0 -g tuntap
 pre-up tunctl -t tap1 -g tuntap
 pre-up ip link set tap0 up
 pre-up ip link set tap1 up
 bridge_ports eth0 tap0 tap1
 bridge_stp off
 bridge_maxwait 0
 bridge_fd 0
 post-down ip link set tap0 down
 post-down ip link set tap1 down
 post-down tunctl -d tap0
 post-down tunctl -d tap1

iface eth1 inet dhcp

Here's the script for starting network bridge.  It gets called by the next script.

#!/bin/sh

sudo ifdown eth0
sudo ifup br0


Here's the script for starting QEMU systems:

#!/bin/sh

quit()
{
 ret=0
 if [ $# -gt 0 ];
 then
  ret=$1
  shift
 fi
 if [ $# -gt 0 ];
 then
  printf "$1\n"
  shift
 fi
 exit $ret
}

image=""
kernel=""
if [ "x$1" = "xle" ];
then
 qemu=qemu-system-mipsel
 image="$HOME/debian-qemu/le/debian_squeeze_mipsel_standard.qcow2"
 kernel="$HOME/debian-qemu/le/vmlinux-2.6.32-5-4kc-malta"
 iface=tap1
 # reset /etc/udev/rules.d/70-persistent-net.rules in qemu guest if you change mac addrs
 macaddr="52:54:00:12:34:XX"
elif [ "x$1" = "xbe" ];
then
 qemu=qemu-system-mips
 image="$HOME/debian-qemu/be/debian_squeeze_mips_standard.qcow2"
 kernel="$HOME/debian-qemu/be/vmlinux-2.6.32-5-4kc-malta"
 iface=tap0
 # reset /etc/udev/rules.d/70-persistent-net.rules in qemu guest if you change mac addrs
 macaddr="52:54:00:12:34:XX"
else
 quit 1 "Specify le or be endianness."
fi

echo "Stopping eth0, starting br0."

bridge.sh || quit 1 "Failed to start br0."

echo "Starting Debian system QEMU session."


$qemu -net nic,macaddr=$macaddr  -net tap,ifname=$iface,script=no,downscript=no -nographic -M malta -kernel $kernel -hda $image -append "root=/dev/sda1 console=tty0"


Booting and Configuring the QEMU Systems


Be sure to edit the MAC addresses in the script above.  They need to be different from one another if you want to run both the little endian and big endian systems at once.

If everything went right, you should be able to start up your QEMU systems:

$ debqemu.sh le

The 'le' argument starts the little endian system and the 'be' argument starts the big endian system.  Be sure to repeat the following customization for both systems.

Once the QEMU system has booted up, you should get a login prompt.  Log in as root with password 'root'.

Since your MAC addresses have changed, edit /etc/udev/rules.d/70-persistent-net.rules and remove entries for any ethernet devices.  Reboot now so they will be regenerated.  This will ensure networking is working properly, in case it isn't currently.

QEMU's console is very limiting, so you'll want to SSH in to do most of your work. You can install OpenSSH in your Debian system using apt:

# apt-get update
# apt-get install openssh-server

From a separate terminal, you should be able to SSH into the QEMU system as root.  You may want to use ssh-copy-id to copy your SSH key in.

At this point I like to set up NFS so I can mount my workspace inside the QEMU system.  That's pretty straightforward so I won't cover it here.

To shutdown your system you can  issue 'halt' or 'init 0', but it doesn't actually power off.  Instead it reboots.   When the kernel's boot messages are scrolling down the console is a good time to cut the power.  Interacting with QEMU's console is similar to using screen or minicom.  You exit using ctrl+a, x, and QEMU terminates.

I hope this is helpful.  Be sure to let me know if anything doesn't make sense or if you get hung up anywhere.