Ownership, Kamal volumes and directories, Docker volumes and bind mounts

Sep 20, 2026

There is indeed not just one way for a Docker container to persist state! Let’s go under the hood to find out more.

Motivation

One of my personal projects is a rather over-engineered Ruby on Rails application. Its deployment/infrastructure is intentionally over-done for a personal project, to allow me to practise designing, maintaining and managing a “serious” infrastructure setup in production. I chose Kamal as a deployment tool, since it seemed like it would Just Work™ on a bare Linux server.

In addition to the main application, my project also uses Grafana and InfluxDB services, containerised using their respective Docker images. While deploying to production for the first time, I found that I apparently faced a permissions issue that prevented them from writing to their respective data directories. I was able to bypass the issue by chowning the relevant directories manually on the production host, but this required looking up/reverse-engineering the desired owner user IDs (UIDs) manually ¹. These two manual steps (looking up the UID used in the container, and chowning the directories manually with that UID) made me wonder if there was a better way.

Some time later I tried to replicate the Kamal deployment config with Docker Compose (easier than I expected - the two map quite nicely onto one another). I noticed that I was no longer facing the permissions issue (no manual tweaks required). I became curious, since in both cases Grafana and InfluxDB were running as Docker containers. What could be the difference between the behaviours? I set aside some time to look into the implementation.

In the end, the situation ended up being both more complex and less exciting than I expected. As far as I can tell, there is no bug in Kamal, but its documentation is rather lacking. The only thing I was missing was an understanding of the features of Kamal and Docker. Read on to find out more about the features Kamal and Docker give you, to enable containers to persist state, and how they are implemented!

Background

It’s not important for you to know how Kamal works. I will assume some familiarity with Docker.

Some concepts such as inodes are not specific to Linux, but for simplicity I will not distinguish between Unix and Linux. Further, some points I present as facts here are dependent on the filesystem implementation, so I will always assume a POSIX/Unix-like filesystem.

Let’s assume for the purposes of this post that I am running Docker as root. That is, I am not using rootless Docker.

Note that I also don’t care that much about what happens when a container is restarted, or when a Docker container/image associated with a service is upgraded. I would like persistent state precisely to enable me to tweak the Docker container definition/image version without losing service/application state. I’m assuming that if I can share data between a container and a host once, then I will naturally be able to share the same data between the host and a newer version of the container/service, in the same way.

File/directory permissions and ownership

The ownership information for a file in Linux is stored as a numeric UID and group ID (GID) in the file’s inode. Therefore it’s not a problem if the UID/GID do not refer to any existing user/group. When you are “sharing” (not a technical term) a file or directory between a host machine and a Docker container, it’s really the inode that is “shared”. Can you see where this is going? That’s all you need to know about inodes.

Docker containers

Let’s understand a little bit about how containers are defined.

For example, consider the InfluxDB Dockerfile. It’s relatively short. For our purposes, only the following instructions are interesting:

<snip>
RUN groupadd --gid 1500 influxdb3 && \
    useradd  --uid 1500 --gid influxdb3 --shell /bin/bash --create-home influxdb3 && \
    mkdir -p /var/lib/influxdb3 \
             /usr/lib/influxdb3 \
             /plugins
<snip>
RUN <snip> && \
    chown -R influxdb3:influxdb3 /var/lib/influxdb3 /plugins && <snip>
<snip>
USER influxdb3
<snip>

The commands to create a user, create some directories and set their owner are all run during Docker’s build step. After building a Dockerfile, you get a Docker image. You may think of a Docker image as a zipped-up Linux filesystem ², with some metadata attached (for example the entrypoint). I found that to be a pretty helpful mental model!

By default, in the build step, Docker runs RUN instructions and the entrypoint command as the container’s root user. The USER instruction ³ instructs Docker to execute every subsequent instruction in the build step as that user. If we don’t switch back to root, the entrypoint will also be executed as that user ⁴.

InfluxDB uses the /var/lib/influxdb3 directory by default to persist state. You might guess that if the running InfluxDB process can’t write to this directory, bad things will happen. Indeed, this was a symptom of the “permissions issue” I was experiencing!

Persisting state with Kamal

If you don’t care about Kamal and care more about Docker, you may skip this section.

By skimming Kamal’s source code, we may quickly understand that it is effectively a wrapper over some shell scripts (notably including ssh and docker). This means that its behaviour isn’t very difficult to figure out! Kamal runs all of its commands on the remote host as a single user; by default this is root, but you may change this.

Deploying auxiliary services with Kamal is done using accessories. These define additional services (Docker containers) that are deployed alongside your app. Unlike Docker Compose, the Kamal configuration can only point to container images; Kamal does not provide any functionality to build Docker images for accessories; instead you must do this in advance.

In this case, Grafana and InfluxDB are my accessories. These are not stateless services - they will need to persist some state on the host machine. For persisting and “sharing” state between the containers and the host, Kamal gives you a few options. For each one, I wanted to figure out what was happening under the hood.

Files

I am not very interested in the files declaration, since I would like to deal with entire directories.

Directories

The Kamal directories declaration promises to create directories on the host before mounting them in the accessory container.

Let’s consider an example. Suppose we specify that Kamal commands should use user alice (UID 123), and that I define the following accessory with a directory.

accessories:
  myaccessory:
    <snip>
    directories:
      - data:/var/lib/mysql

In that case, Kamal will run the following on the remote host, as the alice user.

  INFO [1c6b6f72] Running /usr/bin/env mkdir -p $PWD/store-myaccessory/data on <remote-host-IP>
  INFO [1c6b6f72] Finished in 0.942 seconds with exit status 0 (successful).
<snip>
  INFO [16d293e9] Running docker run <snip> --volume $PWD/store-myaccessory/data:/var/lib/mysql <snip> on <remote-host-IP>
  INFO [16d293e9] Finished in 2.076 seconds with exit status 0 (successful).

--volume? Yes indeed, Kamal directories are implemented using mkdir and the Docker CLI --volume flag (which doesn’t actually create a volume in this case - we’ll get to that later)! First Kamal creates the directories on the host (and assigns permissions and ownership, if specified explicitly), then instructs Docker, using the --volume flag, to use the directories as mounted storage.

What happens to the directory, from the perspective of the container? What does the directory look like? What happens if the Docker image (remember, it’s just a filesystem snapshot with some metadata) already defined a directory at the same mount path? Those are all the responsibilities of the Docker Engine, not Kamal.

Volumes

The Kamal volumes declaration does not have very descriptive documentation, but as we’ll see, it’s the most powerful declaration.

Following the example from the documentation, we may define the following accessory with a volume.

accessories:
  myaccessory:
    <snip>
    volumes:
      - /path/to/mysql-logs:/var/log/mysql

Then, Kamal will run the following on the remote host: docker run <snip> --volume /path/to/mysql-logs:/var/log/mysql <snip>.

In fact (and this is undocumented), we may also put a simple string in place of the host path.

accessories:
  myaccessory:
    <snip>
    volumes:
      - mysql:/var/log/mysql

Then, Kamal will run the following on the remote host: docker run <snip> --volume mysql:/var/log/mysql <snip>.

Now there is in fact a subtle but important difference between these two examples. Only the second one creates a Docker volume! The first is a bind mount, even though it uses the same --volume syntax. The documentation of Kamal is not helpful with this difference. They have different behaviours and use cases, but again, that is the responsibility of the Docker Engine, not Kamal.

Persisting state with Docker

Let’s go deeper and understand what features Docker itself provides for persistent storage, and how they are implemented.

I will make a number of simplifications in this section, since Docker is complex. Firstly, I will ignore the existence of the Dockerfile VOLUME instruction ⁵. I will also ignore any abstractions provided by Docker Compose and go straight to the Docker CLI; that’s because Docker Compose may be viewed as only a client; indeed the Docker Engine is not even aware of who called it.

Volumes

Docker volumes are the canonical mechanism to persist data used by containers. Their storage location is not intended to be managed by the user. If your use case doesn’t require the host to be aware of the path where the data is stored (or to manipulate that data), and instead all you want is for your containers to persist state between upgrades, then you should be using a volume.

When using a volume, you should probably use the preferred --mount syntax. Alternatively, use the docker run --volume <volume-name>:<mount-path> syntax, where <volume-name> is just a name, not a path.

Docker volumes are completely managed by the Docker Engine, and they are stored in a Docker-managed location on the host.

Now the behaviour of Docker volumes differs significantly depending on whether the <mount-path> (the path in the container) already exists in the container image (remember that a Docker container image is just a zipped-up Linux filesystem).

If the mount path does not exist in the container image, it ends up being owned by root, since it’s managed entirely by the Docker Engine, and the engine runs as root (recall that we are not using rootless Docker).

Now if the path exists in the container image, the behaviour is different, and partially documented.

If you mount an empty volume into a directory in the container in which files or directories exist, these files or directories are propagated (copied) into the volume by default. Similarly, if you start a container and specify a volume which does not already exist, an empty volume is created for you.

It turns out that aside from copying the Docker image’s file contents, the ownership is also copied (!!). Let’s see how that works.

1. When a container is created for the first time (recall that this was one of our assumptions), the volumes are populated in agreement with the above documentation.
2. Tracing further, we can see a more specifically-named method CopyImagePathContent that is responsible for copying files from the image to the volume. So far so good.
3. This ends up being a small wrapper. Let’s keep going.
4. Ah, we come to a docstring saying “copyExistingContents copies from the source to the destination and ensures the ownership is appropriately set”. What does it mean for the ownership to be “appropriately set”?
5. For this we’ll need to check the source code of the package github.com/containerd/continuity, a library used by the Docker Engine for some filesystem operations. In turn, the function called there attempts a “most efficient copy of the files”. Seems reasonable to me.
6. Finally, we come to the call to copyFileInfo, which at last calls os.Lchown from the Go standard library. This sets the UID and GID of the directory, according to the ownership from the Docker image.

The directory ownership data is copied to the host along with the contents! At least to me, this behaviour is a little bit unintuitive, since the ownership information ends up “propagating” up from the Docker image onto the host. It’s surely a convenient feature, since it avoids the “bug” I described at the beginning, without requiring any additional configuration from me - the configuration is “encoded” in the Docker image I’m using. For example, if I use a Docker volume with the InfluxDB image, its data directory (in the image) is owned by UID 1500; therefore this owner UID ends up visible on the host as well (recall that it doesn’t matter that no user maps to UID 1500 on the host). Still, this behaviour means that a container image can influence my host in a way that was a little unexpected to me.

Bind mounts

The typical use case for bind mounts is sharing artefacts between the host and the container.

When using a bind mount, you should probably use the preferred --mount syntax. Alternatively, use the docker run --volume <host-path>:<container-path> syntax, where <host-path> is a path to a directory on the host, not just a name.

If the directory already exists on the host, Docker does not touch it, and permissions remain the same. If the directory does not exist on the host at container start, Docker creates it (as root, since Docker is running as root). In neither case can a non-root user inside the container access the mounted directory, unless you create the directory on the host ahead of time, and the owner UID of the directory on the host happens to match the UID of the user running inside the container.

The use case seems to require some degree of coordination between the host and the container. This is in contrast to volumes, where Docker tries its best to perform any necessary setup automatically.

How does a bind mount handle the scenario where the Docker image already has some data under the mount path? Well, the existing data is shadowed.

If you bind mount file or directory into a directory in the container in which files or directories exist, the pre-existing files are obscured by the mount. This is similar to if you were to save files into /mnt on a Linux host, and then mounted a USB drive into /mnt. The contents of /mnt would be obscured by the contents of the USB drive until the USB drive was unmounted.

This seems pretty sensible. Docker has to make a choice about whether to respect the bind mount or the existing image data, and in this case, the bind mount takes precedence. I didn’t check the implementation of this feature.

Why does the Docker CLI use the same --volume syntax for bind mounts and volumes?

I don’t know. It implements this distinction by checking whether the --volume flag refers to a path on the host or not, as documented.

Conclusion

The intersections of these features of Kamal and Docker were not obvious to me. In particular, Kamal seems very poorly documented in this respect, and it doesn’t even point the user to the relevant Docker documentation. On the other hand, the Docker documentation seems pretty reasonable.

Setting the owner UID of a bind mount directory was what I was actually doing, as a workaround for the “bug” described at the beginning. I initially chose an absolute path for my Kamal directory, since I wanted to back up that data later. I wasn’t aware that Kamal directories were implemented using --volume, and that Docker would actually use bind mounts when using --volume syntax with an absolute path. Instead, when using Docker Compose, I was using a genuine Docker volume, for which the ownership was assigned automatically.

Investigating the reasons for this “strange” behaviour went more or less as expected! I was able to reproduce any scenario I wanted pretty easily and deterministically. Tracing through the source code was not too challenging, and it made me appreciate the complexity the Docker authors had to hide from the user to enable a smooth experience with volumes.

Going forward, I would advise you not to use Kamal without first understanding its behaviour in Docker terms. When using the Docker CLI, prefer --mount to --volume where possible.