Containerized applications running in Kubernetes almost always need access to external resources that usually require secrets, passwords, keys, or tokens to gain access. Kubernetes Secrets lets you securely store these items, removing the need to store them in Pod definitions or container images.

In this post, we’ll discuss the various ways to create and use secrets in Kubernetes, with an aim to help you select the best approach for your environment.

Creating Kubernetes Secrets

Kubernetes Secrets offers three methods to create and store secrets:

  • Via the command line
  • In a configuration file
  • With a generator

Let’s take a look at creating some secrets with these methods.

Creating Kubernetes Secrets from the command line

You can create a secret via the Kubernetes administrator command line tool,kubectl. This tool allows you to use files or pass in literal strings from your local machine, package them into secrets, and create objects on the cluster server using an API. It’s important to note that secret objects must be in the form of a DNS subdomain name.

For username and password secrets, use this command line pattern:

kubectl create secret generic <secret-object-name> <flags>

For secrets using TLS from a given public/private key pair, use this command line pattern:

kubectl create secret tls <secret-object-name> --cert=<cert-path> --key=<key-file-path>

You can also create a generic secret using a username and password combination for a database. This example applies the literal flag to specify the username and password at the command prompt:

kubectl create secret generic sample-db-secret --from-literal=username=admin --from-literal=password=’7f3,F9D^LJz37]!W’

The command creates a new secret called sample-db-secret, with a username value of admin and a password value of 7f3,F9D^LJz37]!W. It is worth noting that strong, complex passwords often have characters that need to be escaped. To avoid this, you can put all your usernames and passwords in text files and use the following flags:

kubectl create secret generic sample-db-secret --from-file=username.txt --from-file=password.txt

This command drops the username and password keys as it provides the name of a file containing this information. You can add this back into the --from-file switch the same way as the --from-literal switch if the key is different than the file name.

kubectl create secret generic sample-db-secret --from-file=username=123.txt --from-file=password=xyz.txt

Setting Kubernetes Secrets in a configuration file

Another option is to create your secret using a JSON or YAML configuration file. A secret created in a configuration file has two data maps: data and stringData. The former requires your values to be base64-encoded, whereas the latter allows you to provide values as unencoded strings.

The following template is used for secrets in YAML files:

apiVersion: v1
kind: Secret
    name: <secret name>
type: Opaque
    <key>: <base64 Value>
    <key>: <string value>

You apply the template using the kubectl apply -f ./<filename>.yaml command. As an example, here is a YAML file for an application that requires a number of secret values:

apiVersion: v1
kind: Secret
    name: my-example-app
type: Opaque
    app-user: YWRtaW5pc3RyYXRvcg==
    app-password: cGFzc3dvcmQ=
    Dbconnection:,1433;Database=myDB;User ID=mylogin@myserver;Password=myPassword;Trusted_Connection=False;Encrypt=True;
    config.yaml: |-
        LogLevel: Warning

The above YAML file contains a number of values, including:

  • The secret name (my-example-app)
  • An app-user (“administrator,” base64-encoded)
  • An app-pasword (“password,” base64-encoded)
  • A dbconnection string
  • The config.yaml file with data

This is a good way for packaging up many secrets and potentially sensitive configuration information into a single configuration file.

Creating Kubernetes Secrets with a generator

The third option for creating secrets is to use Kustomize, a standalone tool for customizing Kubernetes objects using a configuration file called kustomization.yaml.

Kustomize allows you to generate secrets in the fashion similar to the command line by specifying secrets in files (with a key/value pair on each line), or as literals within the configuration file.

For secrets, the following structure is used within a kustomization.yaml file:

    name: <secret-name>

When you have created the kustomization.yaml file and included all the linked files in a directory, you can use the kubectl kustomize <directory> command, then apply the configuration using the kubectl apply -k <directory> command.

The following example kustomization.yaml file creates a secret with two literal key/values (API_TOKEN and API_URI), as well as a config.yaml file:

    name: example-app-secrets

The config.yaml file referenced in the above example could be the config file for an application, for instance.

Which method for creating Kubernetes Secrets is the best?

Each of the methods we’ve discussed is “the best” under specific circumstances.

The command line is most useful when you have one or two secrets you want to add to a Pod—for instance a username and password—and you have them in a local file or want to pass them in as literals.

Configuration files are great when handling a handful of secrets that you want to include in a Pod all at one time.

The Kustomize configuration file is the preferred option when you have one or more configuration files and secrets you want to deploy to multiple Pods.

Once you have created your secrets, you can access them in two main ways:

  • Via files (volume mounted)
  • Via environment variables

The first option is similar to accessing configuration files as part of the application process. The second option loads the secrets as environment variables for the application to access. We are going to explore both methods.

Accessing volume mounted Kubernetes secrets

To access secrets loaded in a volume, first you need to add the secret to the Pod under spec[].[]volumes[].secret.secretName. You then add a volume to each container under spec[].containers[].volumeMounts, where the name of the volume is the same as that of the secret, and where readOnly is set to “true”.

There are also a number of additional options you can specify. Let’s have a look at an example Pod with a single container:

apiVersion: v1
kind: Pod
    name: mypod
        - name: myapp
        image: ubuntu
            - name: secrets
            mountPath: "/etc/secrets”
            readOnly: true
    - name: secrets
        secretName: mysecret
        defaultMode: 0400
            - key: username
            path: my-username

This configuration file specifies a single Pod (mypod) with a single container (myapp). In the volumes section for the Pod, you have a volume named secrets, which is shared by all containers. This volume is of type secret, and it loads the secret called mysecret. It loads the volume using the Unix file permissions 0400, which gives read access to the owner (root), and no access to other users.

The secret also contains the items list, which casts only specific secret keys (in this case, username) and an appended path (my-username). In your container (myapp), you map the volume in volumeMounts (name is secrets) to the mountPath as read only.

Because you’re casting the username to my-username, the directory /etc/secrets in the container will look something like this:

lrwxrwxrwx 1 root root 2 September 20 19:18 my-username ->

Because you’re casting a single value and changing the path, the key has changed. If you follow symlink, you will see that the permissions are set correctly:

-r-------- 1 root root  2 September 20 19:18 username

All the files that reside on the secret volume will contain the base64-decoded values of the secrets.

The advantage of loading your secrets into a volume is that, when the secrets are updated or modified, the volume is eventually updated as well, allowing your applications to re-read the secrets. It also makes parsing secret files (such as sensitive configuration files), as well as referencing multiple secrets, a lot easier.

Accessing Kubernetes Secrets as environment variables

You can project your secrets into a container using environment variables. To do this, you add an environment variable for every secret key you wish to add using env[].valueFrom.secretKeyRef. Let’s have a look at an example Pod specification:

apiVersion: v1
kind: Pod
    name: mypod
        - name: myapp
        image: ubuntu
           - name: USERNAME
                   name: mysecret
                   key: username
           - name: PASSWORD
                   name: mysecret
                   key: password

This configuration file passes two keys (username and password) from the mysecret secret to the container as environment variables. These values are the base64-decoded values of the secrets.

If you logged into the container and ran the echo $USERNAME command, you would get the decoded value of the username secret (for example, “admin”).

One of the biggest advantages of casting secrets this way is that you can be very specific with the secret value. Besides, for some applications, reading environment variables is easier than parsing configuration files.

Alternatives to Kubernetes Secrets

While secret management in a Kubernetes cluster is relatively simple, fairly secure, and can meet most requirements, it does have some downsides. In particular, secrets use namespaces like Pods, so if secrets and Pods are in the same namespace, all Pods can read the secrets.

The other major downside is that keys are not rotated automatically. You need to manually rotate secrets.

To address these issues and provide a more centralized secret management, you can use an alternative configuration, such as:

  • Integrating a cloud vendor secrets management tool, such as Hashicorp Vault or AWS Secrets Manager. These tools typically use Kubernetes Service Accounts to grant access to the vault for secrets and mutating webhooks to mount the secrets into the Pod.
  • Integrate a cloud vendor Identity and Access Management (IAM) tool, such as AWS Identity and Access Manager. This type of integration uses a method similar to OpenID Connect for web applications, which allows Kubernetes to utilize tokens from a Secure Token Service.
  • Run a third party secrets manager, such as Conjur loaded into Pods as a sidecar.

Wrapping up

Secure storage of secrets is critical to running containers in Kubernetes because almost all applications require access to external resources—databases, services, and so on. Using Kubernetes Secrets allows you to manage sensitive application information across your cluster, minimizing the risks of maintaining secrets in a non-centralized fashion.

This article was originally published on New Relic’s blog.

How to work with us

  • Contact us to set up a call.
  • We will analyze your needs and recommend a content contract solution.
  • Sign on with ContentLab.
  • We deliver topic-curated, deeply technical content to you.

To get started, complete the form to the right to schedule a call with us.

Send this to a friend