When running a Raspberry Pi cluster, sometimes there’s just not enough power to build native armhf Docker images in a reasonable amount of time.
While the recent announcement of Docker Hub Official Multi-platform support makes it easier to run official multi-arch images, building a multi-arch image still requires a cross-build environment to speed up builds on lower powered hardware or when you don’t have the native platform.
Cross-building is defined as building a foreign architecture image different from the hosts architecture, such as building a armhf image on a x86 machine.
There are three methods of cross-building and running multi-arch Docker images each with different considerations,
Since the original beta, Docker for Mac has had binfmt_misc support built-in, which allows it to execute arm containers on x86 without any additional configuration.
If you have Docker for Mac you can test this out by running any multi-arch Docker image from Docker Hub such as the official arm32v7/debian image,
🐳 uname -a
Darwin sheik.local 16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64 x86_64 i386 MacBookPro11,3 Darwin
🐳 docker run -it --rm arm32v7/debian /bin/bash
Unable to find image 'arm32v7/debian:latest' locally
latest: Pulling from arm32v7/debian
d0e027c48353: Pull complete
Digest: sha256:d74cc69431f03bbfbbf9fd52c1eabd6ca491280a03da267acb63b65b81e30c8a
/ # uname -a
Linux 4372bf9a3462 4.9.41-moby #1 SMP Wed Sep 6 00:05:16 UTC 2017 armv7l GNU/Linux
Building an image is as easy as taking an existing Dockerfile and changing it’s FROM to point to a base multi-arch image. Depending on the image, it may fail if non-architecture binaries run, such as the Dockerfile using wget to download and run a non-native binary.
For example to re-build the NodeJS Debian Image for arm32v7.
🐳 curl -sSL https://raw.githubusercontent.com/nodejs/docker-node/c37d5e87fa6d46c0e387f73161b056bbf90b83aa/8.6/stretch/Dockerfile | sed "s/buildpack-deps:stretch/arm32v7\/buildpack-deps:stretch/g" > Dockerfile.nodejs.armhf
🐳 docker build -f Dockerfile.nodejs.armhf -t nodejs:armhf .
This is the easiest and quickest way to build an run any 32-bit or 64-bit Docker images. Disadvantages are you must be running macOS and are limited by the hardware macOS will run on, this isn’t an option for large multi-cpu cloud instance types that run Linux.
Most major Linux distributions have a way of setting up binfmt_misc using qemu and other cross-architecture tools. These can be quite complicated to setup and rely on distribution specific knowledge. The Multiarch Project makes setting up binfmt_misc much easier by wrapping the whole process in a Docker image itself called qemu-user-static. This will install and setup the qemu-*-static configurations for all architectures excluding the native hardware.
Installing
qemu-user-staticusing a container requires privileged mode since it will register binaries in the host systems/procspace
To setup qemu-user-static, follow the README, which basically consists of,
🐳 docker run --rm --privileged multiarch/qemu-user-static:register
With qemu-user-static setup, test qemu-user-static support works with a Multiarch image,
🐳 uname -a
Linux jezebel 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26) x86_64 GNU/Linux
🐳 docker run -it --rm multiarch/alpine:aarch64-edge /bin/sh
Unable to find image 'multiarch/alpine:aarch64-edge' locally
aarch64-edge: Pulling from multiarch/alpine
ee62da588733: Pull complete
c782b02d60f2: Pull complete
Digest: sha256:17a50d7864c2e052d1c48892252356c1ce9eea26d0a61236072d6c900e5bd6a6
Status: Downloaded newer image for multiarch/alpine:aarch64-edge
/ # uname -a
Linux 62670af32738 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26) aarch64 Linux
Building an image is similiar to Docker For Mac, take an existing Dockerfile and replace it’s FROM with a corresponding Multiarch Docker Hub image.
For example to re-build the NodeJS Alpine Image for aarch64,
🐳 curl -sSL https://raw.githubusercontent.com/nodejs/docker-node/c37d5e87fa6d46c0e387f73161b056bbf90b83aa/8.6/alpine/Dockerfile | sed "s/alpine:3.6/multiarch\/alpine:aarch64-v3.6/g" > Dockerfile.nodejs.aarch64
🐳 docker build -f Dockerfile.nodejs.aarch64 -t nodejs:aarch64 .
While not as cookie cutter as Docker for Mac, using qemu-user-static makes setting up cross-build environments on Linux much easier than it used to be. This also allows you to use much more powerful x86 hardware (either bare metal or cloud) to quickly build Docker images for other architectures than x86.
The only disadvantage is this method relies on images from Docker Hub that have qemu-*-static binaries added, this makes re-building images more difficult since you’ll need to track down a qemu enabled image.
Known repos that have qemu enabled images,
Both Debian and Ubuntu include the qemu-user-static package that includes statically built emulation binaries for QEMU. Installing the package on a host x86 architecture and bind mounting a qemu-*-static binary will allow the image to run a foreign architecture.
To setup qemu-user-static using the apt on Debian or Ubuntu,
🐳 apt update
🐳 apt install -y qemu qemu-user-static qemu-user binfmt-support
Any architecture that is supported on Docker Hub and qemu-*-static should run by bind mounting the correct qemu binary and using the appropriate Docker image.
Run a arm32v7/debian image, bind mount the /usr/bin/qemu-arm-static binary into the container,
🐳 uname -a
Linux jezebel 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26) x86_64 GNU/Linux
🐳 docker run -it --rm -v /usr/bin/qemu-arm-static:/usr/bin/qemu-arm-static arm32v7/debian /bin/bash
Unable to find image 'arm32v7/debian:latest' locally
latest: Pulling from arm32v7/debian
d0e027c48353: Pull complete
Digest: sha256:d74cc69431f03bbfbbf9fd52c1eabd6ca491280a03da267acb63b65b81e30c8a
Status: Downloaded newer image for arm32v7/debian:latest
root@7d91bbe1e01b:/# uname -a
Linux 7d91bbe1e01b 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26) armv7l GNU/Linux
Run a s390x/debian image, bind mount the /usr/bin/qemu-s390x-static binary into the container,
🐳 uname -a
Linux jezebel 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26) x86_64 GNU/Linux
🐳 docker run -it --rm -v /usr/bin/qemu-s390x-static:/usr/bin/qemu-s390x-static s390x/debian /bin/bash
Unable to find image 's390x/debian:latest' locally
latest: Pulling from s390x/debian
2f25bc6ba506: Pull complete
Digest: sha256:b01d35a1891549568b1f5fb66b329dded1e9cd45d6cb74f0c02aeb4c72a1417f
Status: Downloaded newer image for s390x/debian:latest
root@ad7f1fd946fa:/# uname -a
Linux ad7f1fd946fa 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26) s390x GNU/Linux
Using the apt package for qemu-user-static is much more powerful and flexible than the Multiarch method, however it requires more knowledge and configuration when running a container since each architecture will require it’s qemu-*-static binary bind mounted and a corresponding architecture Docker image.
A disadvantage of this method is it’s only useful for running a container, not building a new image since that would require the FROM image to include the qemu-*-static binary and it’s not possible to bind mount the binary when using docker build. For detailed information on cross-building images see the Docker Libary Official Images documentation.
Known official Docker multi-arch images,
Cross-building Docker images is different than the recent Docker Hub Multi-arch support announcement in September 2017. That announcement is a feature of Docker Hub where a repository can have a single image name and include multiple architectures in a manifest. This means that when running Docker on any supported hardware with a multi-arch enabled image, Docker will know to use the proper image for the architecture
Before, repositories would either maintain seperate images (eg https://hub.docker.com/u/arm64v8/) or tag images with specific architectures (eg multiarch/debian-debootstrap:arm64-jessie). Now most of this is abstracted away and running docker run -it --rm debian /bin/bash on x86, armhf, or s390x will automatically know what to do without additional configuration.