Secrets management with Docker Swarm

06 Aug 2017, 17:34

container / docker / basics / dockerbasics

One of the big problems with a cloudy environment is in how to allow the application to get the username/password needed to reach a backend service (e.g. a MySQL database). With a normal application the application operate team can inject these credentials at install time, but a cloudy app needs to be able to start/stop/restart/scale without human intervention. This can get worse with containers because these may be started a lot more frequently.

It’s tempting to just put the information into a config file that’s part of the container build, but this has a number of issues, including:

The container can be pulled apart and the credentials exposed
The config file may be checked into source control
Passwords may change; you don’t want to rebuild and redeploy each time

The 12 Factor approach is to have credentials passed in via the environment (typically an environment variable). Docker secrets is the model that Docker Swarm Mode uses to achieve this.

Note: this doesn’t work with standalone Docker, nor with containers not started as part of a service. It will work with a single-host swarm and a scale 1 service, though.

What is a secret?

A secret, in Docker, is pretty much any string of text. This means you could put pretty much anything into it; a JSON file, an SSL certificate, API keys, a password… anything. The limit is around 500K, which should be large enough!

This is stored in the Docker Swarm raft log, which means it’s replicated across all the management nodes for resiliency. It’s also (since Docker 1.13) encrypted, which means the secrets are never stored in plain text on any disk.

Creating a secret

This is pretty straight forward. You can specify a file that has the secret, or use - to mean STDIN.

% echo hello | docker secret create my-secret
aqfet0saziaqzspvih6p6w72n

% docker secret ls
ID                          NAME                CREATED             UPDATED
aqfet0saziaqzspvih6p6w72n   my-secret           13 seconds ago      13 seconds ago

% docker secret inspect my-secret
[
    {
        "ID": "aqfet0saziaqzspvih6p6w72n",
        "Version": {
            "Index": 893
        },
        "CreatedAt": "2017-08-06T21:53:38.771368641Z",
        "UpdatedAt": "2017-08-06T21:53:38.771368641Z",
        "Spec": {
            "Name": "my-secret",
            "Labels": {}
        }
    }
]

That’s all there is!

Using secrets

We can create a service that refers to the secret. This will then appear in /run inside the container

% docker service create --detach=true --name secret-service \
         --secret my-secret --entrypoint /bin/sh --tty=true centos
iqdxqenn4hrmo6meobq95rinl

% docker service ps secret-service
ID                  NAME                IMAGE               NODE                   DESIRED STATE       CURRENT STATE                    ERROR               PORTS
kv3uo8qm95h3        secret-service.1    centos:latest       docker-ce.spuddy.org   Running             Running less than a second ago

% docker exec -it secret-service.1.kv3uo8qm95h3anox5q372pojh /bin/sh
sh-4.2# ls /run/secrets
my-secret
sh-4.2# cat /run/secrets/my-secret
hello

Note that /run is a tmpfs filesystem and so the secret still doesn’t appear on disk; it’s only stored in memory.

Using secrets with a stack

Because a stack can have private namespaces as well as using the global one we have to define this as an external resource. The model looks very similar to what we’ve seen for networks. That’s deliberate.

% cat secret_stack.yml
version: '3.1'

services:
  os:
    image: centos
    deploy:
      replicas: 1
    entrypoint: /bin/sh
    stdin_open: true
    tty: true
    secrets:
      - source: my-secret

secrets:
  my-secret:
    external: true

% docker stack deploy -c secret_stack.yml secret-stack
Creating network secret-stack_default
Creating service secret-stack_os

% docker service ps secret-stack
ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
eh1ta48g3gfy        secret-stack_os.1   centos:latest       test2.spuddy.org    Running             Running 17 seconds ago

% ssh test2 docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
2e17c5b3a50a        centos:latest       "/bin/sh"           31 seconds ago      Up 30 seconds                           secret-stack_os.1.eh1ta48g3gfyduvgxxx7f0uev

% ssh test2 docker exec secret-stack_os.1.eh1ta48g3gfyduvgxxx7f0uev cat /run/secrets/my-secret
hello

Updating secrets

This is where we start to hit limits. A secret, once in use, can not be changed or even removed. There’s not even a docker secret update option, and docker secret rm fails…

% docker secret

Usage:  docker secret COMMAND

Manage Docker secrets

Options:
      --help   Print usage

Commands:
  create      Create a secret from a file or STDIN as content
  inspect     Display detailed information on one or more secrets
  ls          List secrets
  rm          Remove one or more secrets

Run 'docker secret COMMAND --help' for more information on a command.

% docker secret rm my-secret
Error response from daemon: rpc error: code = 3 desc = secret 'my-secret' is in use by the following services: secret-service, secret-stack_os

This may be a bit of a problem; if you have placed an SSL cert into the secret store and it now needs to be updated then how can you do this?

We have to cheat a little. We can create a new secret and then update the service(s) to use this new secret but under the old name:

% echo A new secret | docker secret create my-secret.v2
kolmhlqxh8wsle75j4siv6dl8

% docker service update --secret-rm my-secret --detach=true \
        --secret-add source=my-secret.v2,target=my-secret secret-service
secret-service

% docker service ps secret-service
ID                  NAME                   IMAGE               NODE                   DESIRED STATE       CURRENT STATE            ERROR               PORTS
4hf3m0rwroeh        secret-service.1       centos:latest       docker-ce.spuddy.org   Ready               Ready 9 seconds ago
kv3uo8qm95h3         \_ secret-service.1   centos:latest       docker-ce.spuddy.org   Shutdown            Running 11 seconds ago

% docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
7ac269ffb300        centos:latest       "/bin/sh"           35 seconds ago      Up 21 seconds                           secret-service.1.4hf3m0rwroehf57evqdowmbsz

% docker exec -it secret-service.1.4hf3m0rwroehf57evqdowmbsz cat /run/secrets/my-secret
A new secret

That’s messy; the docker service update command causes the service to be restarted. In theory that shouldn’t cause downtime to your app, but it’s still a restart and any internal state will be lost.

We also need to update every service that uses a secret before we can remove the old secret

% docker service update --secret-rm my-secret --detach=true \
        --secret-add source=my-secret.v2,target=my-secret secret-stack_os
secret-stack_os

% docker secret rm my-secret
my-secret

% docker secret ls
ID                          NAME                CREATED             UPDATED
kolmhlqxh8wsle75j4siv6dl8   my-secret.v2        6 minutes ago       6 minutes ago

Using secrets as a “seed”

Docker secrets enable us to use an external password vault (e.g. Hashivault). The secret stored inside the Swarm would be sufficient to let your application log into the vault and then get the password it needs (e.g. for the database). It could be used to talk to an API to generate and retrieve an SSL certificate, with the management happening at the enterprise system (e.g. renewal; next time your container starts it will automatically get the new cert).

These “seed” credentials still need to be controlled because they provide access to the final system, but it allows for greater flexibility, especially where credentials may be short lived (Hashivault can generate short-lived passwords in the database; Lets Encrypt certs live 90 days

Since Docker 17.06 there is also docker config. These work conceptually in a similar way to secrets but are implemented slightly differently at the backend; in particular they’re not encrypted at rest and they’re mounted directly into the container.

Management of configs (and the limitations in rotation) are very similar to secrets, just using the docker config command.

Docker configs can work with secrets; e.g. you could create a web server where the config and the HTML index file come from config entries and the SSL certificate comes from the secret entry.

Summary

Docker secrets are currently very limited in what they can do. In particular rotation and updating of secrets is painful and requires your service to be redeployed.

However they do provide a way of handling the “how to get secrets into a container”, which is a hard problem to solve without technology like this.

Using them as a “seed” to access a fully fledged secrets solution (eg a Vault or a Cert management system) overcomes many of the limitations and allows enterprise scale secret management.

Note, however, that anyone in the docker group can get the plain text! This is yet another reason to treat all access to these servers as highly privileged.

Treating passwords as configuration items