Using Tilt with Balena
![Using Tilt with Balena](/content/images/size/w1920/2024/06/5d309cef4a43a92540e30a310200d3b8.webp)
Continuing my adventures with Balena, an amazing IoT orchestration platform, I recently found myself wanting to test the stack of an IoT project locally. Tilt is a great platform for this, while it's generally oriented towards Kubernetes it also provides support for docker-compose. For those who aren't familiar with docker-compose, it's essentially schema that allows you to define a collection of docker containers and how to run them.
version: '2'
services:
my-service:
build:
context: .
dockerfile: ./Dockerfile
prometheus:
build:
context: ./deployments/prometheus
grafana:
build:
context: ./deployments/grafana
Example of a monitoring stack plus a service
It's perfect for local development and is what Balena uses to deploy multiple containers onto a device. The downside to docker-compose is that it doesn't expose anything for automatically rebuilding a service. This means for each change in our application we need to rebuild it manually, and that's no fun. This is where Tilt comes in, using the docker_compose
function in our Tiltfile
we're able to have docker-compose just work.
# Load the docker compose file
docker_compose("./docker-compose.yml")
A Tiltfile that just essentially handles docker-compose up and down for us based on the respective tilt up/down commnads.
This is great, but we still don't have our automatic rebuilds we want. We're able to achieve this by using the docker_build
function. Let's take a look at the changes.
# Load the docker compose file
docker_compose("./docker-compose.yml")
# Build the image for goatkit in docker-compose, syncing the built binary.
docker_build(
# Image to build as and swap containers w/ this image, and then the build context directory
'my-service', '.',
# Use this Dockerfile
dockerfile='./Dockerfile',
# Only include changes in these directories (also trims the build context)
only=['./internal', './pkg', './cmd'],
)
Tiltfile with automatic rebuilds to the image my-service
when there is changes to internal
pkg
or cmd
(Go specific directories)
If we run tilt up
we'll automatically get our docker image rebuilt on changes ?'
Using with Balena
For those of familiar with Balena you're probably aware of balena push
, which is a great command but it's not all that useful for trying to work with locally. Which is why we had to go build the docker-compose
setup system earlier. When I first got that system working I was frustrated with having to maintain two development systems. Our docker-compose
setup is further optimized to build binaries on the host and sync only those in, and the same was done with some hacky combination of balena push
sync + air automated rebuilds, and entr for automatic restarts without restarting the container. So, I set out to see how we could potentially hook this up to Balena.
Where's the Docker Host?
I started by looking at the balena-cli source code trying to reverse engineer the balena push
command. It turns out that the docker runtime is exposed at <ip>:2375
(source). I gave this a shot by using DOCKER_HOST=tcp://<device-ip>:2375 docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d96ab57fe037 d91873660990 "/bin/prometheus --c…" 36 minutes ago Up 36 minutes prometheus_4874803_2159222_f47abb6c669dd1f683fa440f17822a43
89b5eb8751a1 c3a2c27f36c9 "/usr/local/bin/my…" 36 minutes ago Restarting (2) 15 seconds ago my-service_4874801_2159222_f47abb6c669dd1f683fa440f17822a43
8052870f678a a1dc2ecec4c4 "/run.sh" 36 minutes ago Up 36 minutes grafana_4874804_2159222_f47abb6c669dd1f683fa440f17822a43
91d306f3860f registry2.balena-cloud.com/v2/85d77236b4d0fe3fc0e64c7ea43fc645 "/usr/src/app/entry.…" 2 hours ago Up 2 hours (healthy) balena_supervisor
Turns out, it's that easy. There's no authentication on the docker runtime on the development
images that Balena provides, which makes some sense given their disclaimer about development images not being safe for production usage.
Plugging it into Tilt
Taking a quick look at the images we can see that there's some special labels added to the images generated by Balena.
$ DOCKER_HOST=tcp://<device-ip>:2375 docker inspect <a-service-container-id> | jq .'[0].Config.Labels'
{
"io.balena.app-id": "numericID",
"io.balena.app-uuid": "UUID",
"io.balena.service-id": "numericID",
"io.balena.service-name": "my-service",
"io.balena.supervised": "true"
}
It turns out that the supervisor really likes these labels to be set, so I went ahead and created a docker-compose.tilt.yml
to be used for Tilt and inserted these labels in there for each of our service.
my-service:
labels:
io.balena.app-id: "numericID"
io.balena.app-uuid: "UUID"
io.balena.service-id: "numericID"
io.balena.service-name: "my-service"
io.balena.supervised: "true"
After that I noticed the supervisor was pretty angry about the container name format. This was easy to fix by using the container_name
attributed in the docker-compose.tilt.yaml
, so I went and added that to each of my services.
my-service:
...
container_name: my-service_0000000_0000000_local
Running Tilt
Now that we have our docker-compose.tilt.yml
all we need to do is get Tilt to use it. Simply modify the docker_compose
call to include it:
docker_compose("./docker-compose.generated.yml", "./docker-compose.tilt.yml")
Tilt supports loading multiple docker-compose files
I wrote a quick Go script to do this, but an easier way is just to use the following one-liner (if there's a demand for this I'm happy to make this more portable):
DOCKER_HOST=tcp://<device-ip>:2375 tilt up
Start's Tilt on the Balena device. Make sure to run tilt down
when done, but turning local mode off will also clean it up. Ensure that local mode is on before running!
That's it! Press space
to open the Web UI and you should be able to see your services in action.
No spam, no sharing to third party. Only you and me.
Member discussion