Containerized applications running in Kubernetes almost always need access to external resources that usually require secrets, passwords, keys, or tokens to gain access. Kubernetes Secrets lets you securely store these items, removing the need to store them in Pod definitions or container images.
In this post, we’ll discuss the various ways to create and use secrets in Kubernetes, with an aim to help you select the best approach for your environment.
Creating Kubernetes Secrets
Kubernetes Secrets offers three methods to create and store secrets:
- Via the command line
- In a configuration file
- With a generator
Let’s take a look at creating some secrets with these methods.
Creating Kubernetes Secrets from the command line
You can create a secret via the Kubernetes administrator command line tool,
kubectl. This tool allows you to use files or pass in literal strings from your local machine, package them into secrets, and create objects on the cluster server using an API. It’s important to note that secret objects must be in the form of a DNS subdomain name.
For username and password secrets, use this command line pattern:
kubectl create secret generic <secret-object-name> <flags>
For secrets using TLS from a given public/private key pair, use this command line pattern:
kubectl create secret tls <secret-object-name> --cert=<cert-path> --key=<key-file-path>
You can also create a generic secret using a username and password combination for a database. This example applies the
literal flag to specify the username and password at the command prompt:
kubectl create secret generic sample-db-secret --from-literal=username=admin --from-literal=password=’7f3,F9D^LJz37]!W’
The command creates a new secret called sample-db-secret, with a username value of admin and a password value of
7f3,F9D^LJz37]!W. It is worth noting that strong, complex passwords often have characters that need to be escaped. To avoid this, you can put all your usernames and passwords in text files and use the following flags:
kubectl create secret generic sample-db-secret --from-file=username.txt --from-file=password.txt
This command drops the username and password keys as it provides the name of a file containing this information. You can add this back into the
--from-file switch the same way as the
--from-literal switch if the key is different than the file name.
kubectl create secret generic sample-db-secret --from-file=username=123.txt --from-file=password=xyz.txt
Setting Kubernetes Secrets in a configuration file
Another option is to create your secret using a JSON or YAML configuration file. A secret created in a configuration file has two data maps:
stringData. The former requires your values to be base64-encoded, whereas the latter allows you to provide values as unencoded strings.
The following template is used for secrets in YAML files:
apiVersion: v1 kind: Secret metadata: name: <secret name> type: Opaque data: <key>: <base64 Value> stringData: <key>: <string value>
You apply the template using the
kubectl apply -f ./<filename>.yaml command. As an example, here is a YAML file for an application that requires a number of secret values:
apiVersion: v1 kind: Secret metadata: name: my-example-app type: Opaque data: app-user: YWRtaW5pc3RyYXRvcg== app-password: cGFzc3dvcmQ= stringData: Dbconnection: Server=tcp:myserver.database.net,1433;Database=myDB;User ID=mylogin@myserver;Password=myPassword;Trusted_Connection=False;Encrypt=True; config.yaml: |- LogLevel: Warning API_TOKEN: NcNIMcMYMAMg.MGwjPnPfEBgqMl8Q API_URI: https://www.myapp.com/api
The above YAML file contains a number of values, including:
- The secret name (my-example-app)
config.yamlfile with data
This is a good way for packaging up many secrets and potentially sensitive configuration information into a single configuration file.
Creating Kubernetes Secrets with a generator
The third option for creating secrets is to use Kustomize, a standalone tool for customizing Kubernetes objects using a configuration file called
Kustomize allows you to generate secrets in the fashion similar to the command line by specifying secrets in files (with a key/value pair on each line), or as literals within the configuration file.
For secrets, the following structure is used within a kustomization.yaml file:
secretGenerator: name: <secret-name> files: <filename> literals: <key>=<value>
When you have created the kustomization.yaml file and included all the linked files in a directory, you can use the kubectl kustomize <directory> command, then apply the configuration using the kubectl apply -k <directory> command.
The following example
kustomization.yaml file creates a secret with two literal key/values (API_TOKEN and API_URI), as well as a config.yaml file:
secretGenerator: name: example-app-secrets files: passwords.txt literals: API_TOKEN: NcNIMcMYMAMg.MGwjPnPfEBgqMl8Q API_URI: https://www.myapp.com/api
config.yaml file referenced in the above example could be the config file for an application, for instance.
Which method for creating Kubernetes Secrets is the best?
Each of the methods we’ve discussed is “the best” under specific circumstances.
The command line is most useful when you have one or two secrets you want to add to a Pod—for instance a username and password—and you have them in a local file or want to pass them in as literals.
Configuration files are great when handling a handful of secrets that you want to include in a Pod all at one time.
The Kustomize configuration file is the preferred option when you have one or more configuration files and secrets you want to deploy to multiple Pods.
Once you have created your secrets, you can access them in two main ways:
- Via files (volume mounted)
- Via environment variables
The first option is similar to accessing configuration files as part of the application process. The second option loads the secrets as environment variables for the application to access. We are going to explore both methods.
Accessing volume mounted Kubernetes secrets
To access secrets loaded in a volume, first you need to add the secret to the Pod under
spec.volumes.secret.secretName. You then add a volume to each container
under spec.containers.volumeMounts, where the name of the volume is the same as that of the secret, and where readOnly is set to
There are also a number of additional options you can specify. Let’s have a look at an example Pod with a single container:
apiVersion: v1 kind: Pod metadata: name: mypod spec: containers: - name: myapp image: ubuntu volumeMounts: - name: secrets mountPath: "/etc/secrets” readOnly: true volumes: - name: secrets secret: secretName: mysecret defaultMode: 0400 items: - key: username path: my-username
This configuration file specifies a single Pod (
mypod) with a single container (
myapp). In the volumes section for the Pod, you have a volume named secrets, which is shared by all containers. This volume is of type secret, and it loads the secret called mysecret. It loads the volume using the Unix file permissions
0400, which gives read access to the owner (
root), and no access to other users.
The secret also contains the items list, which casts only specific secret keys (in this case,
username) and an appended path (
my-username). In your container (
myapp), you map the volume in volumeMounts (name is
secrets) to the mountPath as read only.
Because you’re casting the username to my-username, the directory
/etc/secrets in the container will look something like this:
lrwxrwxrwx 1 root root 2 September 20 19:18 my-username -> ..data/username
Because you’re casting a single value and changing the path, the key has changed. If you follow symlink, you will see that the permissions are set correctly:
-r-------- 1 root root 2 September 20 19:18 username
All the files that reside on the secret volume will contain the base64-decoded values of the secrets.
The advantage of loading your secrets into a volume is that, when the secrets are updated or modified, the volume is eventually updated as well, allowing your applications to re-read the secrets. It also makes parsing secret files (such as sensitive configuration files), as well as referencing multiple secrets, a lot easier.
Accessing Kubernetes Secrets as environment variables
You can project your secrets into a container using environment variables. To do this, you add an environment variable for every secret key you wish to add using
env.valueFrom.secretKeyRef. Let’s have a look at an example Pod specification:
apiVersion: v1 kind: Pod metadata: name: mypod spec: containers: - name: myapp image: ubuntu env: - name: USERNAME valueFrom: secretKeyRef: name: mysecret key: username - name: PASSWORD valueFrom: secretKeyRef: name: mysecret key: password
This configuration file passes two keys (
password) from the mysecret secret to the container as environment variables. These values are the base64-decoded values of the secrets.
If you logged into the container and ran the echo
$USERNAME command, you would get the decoded value of the
username secret (for example, “admin”).
One of the biggest advantages of casting secrets this way is that you can be very specific with the secret value. Besides, for some applications, reading environment variables is easier than parsing configuration files.
Alternatives to Kubernetes Secrets
While secret management in a Kubernetes cluster is relatively simple, fairly secure, and can meet most requirements, it does have some downsides. In particular, secrets use namespaces like Pods, so if secrets and Pods are in the same namespace, all Pods can read the secrets.
The other major downside is that keys are not rotated automatically. You need to manually rotate secrets.
To address these issues and provide a more centralized secret management, you can use an alternative configuration, such as:
- Integrating a cloud vendor secrets management tool, such as Hashicorp Vault or AWS Secrets Manager. These tools typically use Kubernetes Service Accounts to grant access to the vault for secrets and mutating webhooks to mount the secrets into the Pod.
- Integrate a cloud vendor Identity and Access Management (IAM) tool, such as AWS Identity and Access Manager. This type of integration uses a method similar to OpenID Connect for web applications, which allows Kubernetes to utilize tokens from a Secure Token Service.
- Run a third party secrets manager, such as Conjur loaded into Pods as a sidecar.
Secure storage of secrets is critical to running containers in Kubernetes because almost all applications require access to external resources—databases, services, and so on. Using Kubernetes Secrets allows you to manage sensitive application information across your cluster, minimizing the risks of maintaining secrets in a non-centralized fashion.
This article was originally published on New Relic’s blog.