Using Tilt with Balena

Continuing my adventures with Balena, an amazing IoT orchestration platform, I recently found myself wanting to test the stack of an IoT project locally. Tilt is a great platform for this, while it's generally oriented towards Kubernetes it also provides support for docker-compose. For those who aren't familiar with docker-compose, it's essentially schema that allows you to define a collection of docker containers and how to run them.

version: '2'

services:
  my-service:
    build:
      context: .
      dockerfile: ./Dockerfile

  prometheus:
    build:
      context: ./deployments/prometheus

  grafana:
    build:
      context: ./deployments/grafana

It's perfect for local development and is what Balena uses to deploy multiple containers onto a device. The downside to docker-compose is that it doesn't expose anything for automatically rebuilding a service. This means for each change in our application we need to rebuild it manually, and that's no fun. This is where Tilt comes in, using the docker_compose function in our Tiltfile we're able to have docker-compose just work.

# Load the docker compose file
docker_compose("./docker-compose.yml")

This is great, but we still don't have our automatic rebuilds we want. We're able to achieve this by using the docker_build function. Let's take a look at the changes.

# Load the docker compose file
docker_compose("./docker-compose.yml")

# Build the image for goatkit in docker-compose, syncing the built binary.
docker_build(
  # Image to build as and swap containers w/ this image, and then the build context directory
  'my-service', '.',
  # Use this Dockerfile
  dockerfile='./Dockerfile',
  # Only include changes in these directories (also trims the build context)
  only=['./internal', './pkg', './cmd'],
)

If we run tilt up we'll automatically get our docker image rebuilt on changes ?'

Using with Balena

For those of familiar with Balena you're probably aware of balena push, which is a great command but it's not all that useful for trying to work with locally. Which is why we had to go build the docker-compose setup system earlier. When I first got that system working I was frustrated with having to maintain two development systems. Our docker-compose setup is further optimized to build binaries on the host and sync only those in, and the same was done with some hacky combination of balena push sync + air automated rebuilds, and entr for automatic restarts without restarting the container. So, I set out to see how we could potentially hook this up to Balena.

Where's the Docker Host?

I started by looking at the balena-cli source code trying to reverse engineer the balena push command. It turns out that the docker runtime is exposed at <ip>:2375 (source). I gave this a shot by using DOCKER_HOST=tcp://<device-ip>:2375 docker ps

CONTAINER ID   IMAGE                                                            COMMAND                  CREATED          STATUS                          PORTS     NAMES
d96ab57fe037   d91873660990                                                     "/bin/prometheus --c…"   36 minutes ago   Up 36 minutes                             prometheus_4874803_2159222_f47abb6c669dd1f683fa440f17822a43
89b5eb8751a1   c3a2c27f36c9                                                     "/usr/local/bin/my…"   36 minutes ago   Restarting (2) 15 seconds ago             my-service_4874801_2159222_f47abb6c669dd1f683fa440f17822a43
8052870f678a   a1dc2ecec4c4                                                     "/run.sh"                36 minutes ago   Up 36 minutes                             grafana_4874804_2159222_f47abb6c669dd1f683fa440f17822a43
91d306f3860f   registry2.balena-cloud.com/v2/85d77236b4d0fe3fc0e64c7ea43fc645   "/usr/src/app/entry.…"   2 hours ago      Up 2 hours (healthy)                      balena_supervisor

Turns out, it's that easy. There's no authentication on the docker runtime on the development images that Balena provides, which makes some sense given their disclaimer about development images not being safe for production usage.

Plugging it into Tilt

Taking a quick look at the images we can see that there's some special labels added to the images generated by Balena.

$ DOCKER_HOST=tcp://<device-ip>:2375 docker inspect <a-service-container-id> | jq .'[0].Config.Labels'

{
  "io.balena.app-id": "numericID",
  "io.balena.app-uuid": "UUID",
  "io.balena.service-id": "numericID",
  "io.balena.service-name": "my-service",
  "io.balena.supervised": "true"
}

It turns out that the supervisor really likes these labels to be set, so I went ahead and created a docker-compose.tilt.yml to be used for Tilt and inserted these labels in there for each of our service.

  my-service:
    labels:
      io.balena.app-id: "numericID"
      io.balena.app-uuid: "UUID"
      io.balena.service-id: "numericID"
      io.balena.service-name: "my-service"
      io.balena.supervised: "true"

After that I noticed the supervisor was pretty angry about the container name format. This was easy to fix by using the container_name attributed in the docker-compose.tilt.yaml, so I went and added that to each of my services.

  my-service:
    ...
    container_name: my-service_0000000_0000000_local

Running Tilt

Now that we have our docker-compose.tilt.yml all we need to do is get Tilt to use it. Simply modify the docker_compose call to include it:

docker_compose("./docker-compose.generated.yml", "./docker-compose.tilt.yml")

I wrote a quick Go script to do this, but an easier way is just to use the following one-liner (if there's a demand for this I'm happy to make this more portable):

DOCKER_HOST=tcp://<device-ip>:2375 tilt up

That's it! Press space to open the Web UI and you should be able to see your services in action.