Building Cross-Architecture Docker Images

Recently I wrote about building ARM Docker images on an x86 machine. However, my chosen method was a bit hackey where you didn’t end up with a single Docker image tag that could be used on any architecture. So I did some more research and found the Docker Buildx plugin which helped me get much more desirable results.

The following articles got me moving in the right direction:

Setup:

I’m going to make the following assumptions:

  • You’re going to be building your pipeline on an x86 64 bit Linux machine.
  • You already have Docker installed and running on said Linux machine.
  • You are running Docker version 19.03.

Note: I tried doing these steps on my Raspberry Pi 4 8GB model and the build process was just so slow (over 20 minutes) for x86 64 bit images that I decided it wasn’t worth it. Please let me know if there is some way to speed this up on the Pi 4.

Installing Buildx:

First, we need to enable experimental mode on the Docker daemon.

Create /etc/docker/daemon.json with the following contents:

{ 
    "experimental": true 
}

Then restart Docker:

systemctl restart docker.service

Now we need to install buildx. The official documentation is here but I’ll run through an example setup. Replace the references to 0.4.2 with whatever the latest version number is.

mkdir -p ~/.docker/cli-plugins/

curl -L https://github.com/docker/buildx/releases/download/v0.4.2/buildx-v0.4.2.linux-amd64 -o ~/.docker/cli-plugins/docker-buildx

chmod +x ~/.docker/cli-plugins/docker-buildx

Just to make sure it worked, try running docker buildx:

# docker buildx --help

Usage:  docker buildx [OPTIONS] COMMAND

Build with BuildKit

Options:
      --builder string   Override the configured builder instance

Management Commands:
  imagetools  Commands to work on images in registry

Commands:
  bake        Build from a file
  build       Start a build
  create      Create a new builder instance
  du          Disk usage
  inspect     Inspect current builder instance
  ls          List builder instances
  prune       Remove build cache 
  rm          Remove a builder instance
  stop        Stop builder instance
  use         Set the current builder instance
  version     Show buildx version information 

Run 'docker buildx COMMAND --help' for more information on a command.

Setup QEMU Binfmt:

In order for Docker to build across architectures, it can use QEMU images to emulate the other architectures.

You can use the following Docker command to set up the QEMU images:

docker run --privileged --rm tonistiigi/binfmt --install all

Setup the Builder:

Run the following command to set up a builder image/container for buildx to use:

docker buildx create --use --name builder

Then bootstrap the builder and view what architectures are available to build with:

# docker buildx inspect --bootstrap
Name:   builder
Driver: docker-container

Nodes:
Name:      builder0
Endpoint:  unix:///var/run/docker.sock
Status:    running
Platforms: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

Building Cross-Platform Containers:

Now, in order to build your images, you can use docker buildx build and specify the platforms you want to target, like so:

docker buildx build --no-cache --platform linux/amd64,linux/arm64,linux/arm -t heywoodlh/example .

If you want to push the images to Docker Hub automatically you can add the --push flag to your buildx command.

Some advice I have would be to use base Docker images that are cross-architecture. For a lot of my containers I used to use the official Arch Linux Docker image, but that only supports x86-64 and so it can’t be used on ARM. I now prefer the Alpine image or the Debian image.

Bonus: My Script/Cron for Building:

All of my Dockerfiles are on Github. I use the following script that runs via cron once a day:

#!/usr/bin/env bash

dockerDir="/opt/dockerfiles"

cd ${dockerDir}
git pull origin master 


buildContainer () {
        container=$1
        echo "Building ${container} container..."
        cd ${dockerDir}/${container}/
        docker buildx build --no-cache --platform linux/amd64,linux/arm64,linux/arm --push -t heywoodlh/${container} . &&\
                error="false"
        if [[ ${error} == "false" ]]
        then
                echo "Building ${container} container succeeded!"
        else
                echo "Building ${container} container failed!" | ssmtp user@example.com
        fi
}

if [[ $1 != '' ]]
then
        container=$1
        buildContainer ${container} 
        exit 0
fi

# tomnomnom-tools build
buildContainer tomnomnom-tools

# red build
buildContainer red

# telnet build
buildContainer telnet

# metasploit build
buildContainer metasploit

# links build
buildContainer links

# aerc build
buildContainer aerc

# vt-cli build
buildContainer vt-cli

# openssh build
buildContainer openssh

# evilginx2 build
buildContainer evilginx2

# jackit build
buildContainer jackit

docker system prune -af

Clearly this is specific to how I organize things and you could definitely have something a bit more complex than I have here, but hopefully it’s a good starting point for anyone wanting to build their own cross-architecture Docker build pipeline.

Written on August 18, 2020