When restart: always isn’t enough: making Compose stacks recover at boot

My Mastodon instance runs on Docker Compose, in an LXC container on a Proxmox host at home. The host reboots most weeks to pick up kernel updates, and that has never been a problem: every service in my compose files has restart: always or restart: unless-stopped, so everything comes back on its own.

Until one morning, when it didn’t.

My monitoring did its job and alerted me. However, the reboot is scheduled in the small hours, so I was fast asleep, and the instance was down for four hours before I got round to looking at it.

When I did look, docker ps was empty. Not a list of crashed containers - nothing running at all.

The first thing I did was bring everything back up by hand:

docker compose up -d

That worked, first time, in every project directory. Which was actually kinda annoying, because it meant whatever had gone wrong had already gone away, and I now had to find it after the fact.

The problem

The compose files were fine - I checked the restart policies first, and every service had one. So if the containers weren’t running, the daemon was the place to look:

sudo journalctl -u docker.service -b

And there it was:

level=error msg="Failed to load default apparmor profile" error="AppArmor enabled on 
system but the docker-default profile could not be loaded: running 
'/usr/sbin/apparmor_parser -Kr' failed with output: apparmor_parser: Unable to 
replace "docker-default". apparmor_parser: Access denied. You need policy admin
privileges to manage profiles. error: exit status 243"

Followed by thirteen of these, one per container:

level=error msg="failed to start container" container=518ed98356409f... 
error="AppArmor enabled on system but the docker-default profile could not be 
loaded: ... error: exit status 243"

And then, immediately after:

level=info msg="Loading containers: done."
level=info msg="Daemon has completed initialization"

So Docker couldn’t load its default AppArmor profile, every container refused to start as a result, and the daemon then declared itself fully up and left me hanging.

Worth knowing: restart: always does not help here. The restart policy governs what happens when a running container exits. When the daemon restores containers at startup it makes exactly one attempt each, and if that attempt fails, the policy never comes into play. Nothing retries. The daemon reports itself healthy with zero containers running, and sits there indefinitely.

Chasing the cause

I spent a while trying to work out why the profile load was denied, and I should say up front that I haven’t yet got to the bottom of it. Here’s what I ruled out, in case it saves someone else the time.

It wasn’t specific to Docker. Two seconds before dockerd tried, rsyslogd had failed the same way:

reload-apparmor-profile[136]: apparmor_parser: Unable to replace "rsyslogd". 
apparmor_parser: Access denied. You need policy admin privileges to manage profiles.

So nothing inside the LXC container could load AppArmor policy at that moment.

My first theory was a startup race on the Proxmox host: the container starting before the host had finished setting up AppArmor. That turned out to be wrong. The host’s apparmor.service finished at 02:01:16 and the container started at 02:01:28, a full twelve seconds later.

Nesting is enabled on the container, so that wasn’t it either.

And it doesn’t reproduce. I ran pct reboot on the container against the already-running host, and all thirteen came back perfectly. It’s also intermittent across host reboots - the host reboots most weeks, and every previous one has been fine.

So: something denied AppArmor policy loading inside the container for a window after the host booted, then quietly stopped denying it. I don’t know what, and I can’t trigger it on demand.

One dead end worth flagging

While testing this I reached for an obvious-looking check - try loading an existing profile and see if it works:

sudo apparmor_parser -Kr /etc/apparmor.d/usr.sbin.rsyslogd

This fails with “Access denied” every single time, whether or not Docker is healthy. It looks exactly like a reproduction, and it isn’t one. That profile is loaded by the Proxmox host, and a nested container can’t replace a profile owned by the parent namespace. It’s a different permission failing for a different reason.

There’s also no /etc/apparmor.d/docker to test with, because dockerd doesn’t read its profile from a file - it generates docker-default at runtime and pipes it to apparmor_parser. So if you want to test the permission Docker actually needs, load a throwaway profile of your own:

printf 'profile probe {\n  file,\n}\n' | sudo apparmor_parser -a -; echo "rc=$?"
printf 'profile probe {\n  file,\n}\n' | sudo apparmor_parser -R -

The solution

Since I couldn’t find the trigger, and it clears itself, I stopped trying to prevent it and made the recovery automatic instead.

What’s needed is something that runs after Docker at boot, brings every stack up, and - crucially - keeps trying if it doesn’t work the first time. That last part is the whole point. A one-shot docker compose up -d at boot would have failed on that particular morning exactly like the daemon did.

So I wrote compose-autostart.sh: it finds every compose project in the BASE_DIR directory, brings each one up, drops the ones that succeeded, and retries the rest every 30 seconds for up to an hour. On a normal boot it does one round and exits.

Two things I should highlight here.

This script only runs at boot, not on a timer. My backup script deliberately stops each stack while it snapshots it, and a reconciler running on a schedule would happily start them again mid-backup. For the same reason, the script skips any round where the backup is holding its lock.

And it will bring up any containers that I may have stopped on purpose, since it doesn’t know what was and wasn’t running by the time the host rebooted.

Making the solution permanent

The unit itself is unremarkable:

[Unit]
Description=Bring all Docker Compose stacks up at boot, retrying until they start
Requires=docker.service
After=docker.service network-online.target
Wants=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/path/to/compose-autostart.sh
TimeoutStartSec=infinity

One systemd wrinkle: you can’t use Restart= with Type=oneshot, so the retry loop has to live in the script rather than in the unit. TimeoutStartSec=infinity is there because the script owns its own deadline.

Enable the service with

sudo systemctl daemon-reload
sudo systemctl enable compose-autostart.service

Wrapping up

I haven’t yet found out what denied AppArmor policy loading that morning. I’d love to know if you have any theories.

The more important part for now is that - hopefully - a failure like this should in future clear itself: retrying in a loop turns a middle-of-the-night outage into something that’s already fixed by the time I wake up.