The current article will tackle one of the most import feature of kubernetes which is called Secrets. Before dig diving in this feature first, I would like to mention that throughout this article I’ll be using the terms Kubernetes and K8s interchangeably.
Well, the purpose of the article is to walk throughout the most important concepts behind K8s Secrets and how it handles sensitive information inside our cluster. I’ll be demonstrating these concepts using minikube which is a tool to run a local K8s cluster. Yeah all K8s’ features up and running in my laptop which a very nice thing isn’t it ? The installation is very straightforward and please feel free to ping me if your are stuck somewhere.
So, what we will learn:
- Explore Secrets in K8s ecosystem
- Understand the concept behind Secrets
- Play with Secrets with a real word use cases
Well, the rest of this article is organized as follows:
- Secrets creation
- Secrets usability
- As volumes
- As environment variables
Secrets is designed to store and handle sensitive information that can be need by internal or external resources, from pods, images and containers standing point. For instance credentials, passwords, tokens, keys, ssh certificates etc. needed/used by APIs, endpoints, servers, databases etc. In fact, Secrets provide not only a flexible way for managing sensitive data but most importantly Secrets manage such information in a safer manner than incorporating it in plain-old text inside containers or pods.
Overview and concept map
Roughly speaking, Secrets is an object that contains a small amount of sensitive data such as passwords, keys and tokens etc. A brief concept map for Secrets inside K8s ecosystem can be drawn like below:
Mainly pods are part of a namespace which is enviously part of a cluster node. Containers belonging to a pods might share mounted volumes, these containers operates on the Secrets objects to interact with internal or external systems. To make this happen, the pods must references the needed secrets. Therefore, there are mainly three ways of doing, the first a one by using volumes, the second one by using environment variables, and the last one through kubelet.
Regarding the secret object itself we can distinguish between two types, user’s and system ’s secrets, for instance K8s create its own secrets automatically for accessing the K8s API server (the main entry point for managing the closer under K8s) and all the user’s created pods are behind the scene overrides to use the build-in secrets. Let’s check if there are any system secrets in my environment, before create any secret object, to do so, we can follow the K8s kubectl get command API, which is kubectl get secrets
➜ ~ kubectl get secrets NAME TYPE DATA AGE default-token-xny9c kubernetes.io/service-account-token 3 13d tls-certs Opaque 4 13d
I can see that service account for instance is already created: default-token-xny9c which is a build in secret.
Let’s assume that a pods need access to redis database, mainly a username and password which they are stored in files for instance ./username.txt and ./password.txt. Please note that for simplicity reason, I'll be using the same files in my demonstration for the rest of this article. So, let’s create and put some faked data into these two files:
➜ ~ echo -n 'zombie' > ./username.txt ➜ ~ echo -n '1f2d1e2e67df' > ./password.txt ➜ ~ cat username.txt zombie% ➜ ~ cat password.txt 1f2d1e2e67df
In fact, there are two ways for creating a secret in K8s, the first one by using the command kubectl create secret and the second one manually from a spec file; either JSON or YAML data serialisation are allowed.
Creating secret object using the command line
In order to create a secret object we use the command like so:
➜ ~ kubectl create secret generic db-zombie-pass --from-file=./username.txt --from-file=./password.txt secret "db-zombie-pass" created
Once again to check the create secrets:
➜ ~ kubectl get secrets NAME TYPE DATA AGE db-zombie-pass Opaque 2 27m default-token-xny9c kubernetes.io/service-account-token 3 14d tls-certs Opaque 4 13d
Now that we have created our first secret object, let’s describe it using kubectl describe command:
➜ ~ kubectl describe secret db-zombie-pass Name: db-zombie-pass Namespace: default Labels: Annotations: Type: Opaque Data ==== password.txt: 12 bytes username.txt: 6 bytes
Please note that the last command shows the files bundled in our secret object but not the content itself, which is hugely important as it prevent the secret from being exposed to other users using the k8s environment.
Creating a secret object manually using a spec file
Creating a secret object manually can be done using a spec file either using JSON or YAML data serialisation. Secret values are rather encoded in base64 string. Therefore, first in order to create a secret object using a spec file, the user need to encode the secret values as illustrated below:
➜ ~ echo -n 'zombie' | base64 em9tYmll ➜ ~ echo -n '1f2d1e2e67df' | base64 MWYyZDFlMmU2N2Rm
Second, open up your favourite editor and edit the secret file as follows, let’s call it my-secret.yaml
apiVersion: v1 kind: Secret metadata: name: mysecret type: Opaque data: username: em9tYmll password: MWYyZDFlMmU2N2Rm
Then we can create the secret object from the spec file by running the command below:
➜ ~ vim my-secrte.yaml ➜ ~ kubectl create -f ./my-secrte.yaml secret "mysecret" created
Well, I saw that kubectl describe does not display the content of the secret object, but what if someone want to check this content; well, we can use the command kubectl get secret by providing the secret object name, for instance for our first created secret db-zombie-pass we can check the content like this:
➜ ~ kubectl get secret db-zombie-pass -o yaml apiVersion: v1 data: password.txt: MWYyZDFlMmU2N2Rm username.txt: em9tYmll kind: Secret metadata: creationTimestamp: 2016-11-30T17:07:17Z name: db-zombie-pass namespace: default resourceVersion: "364840" selfLink: /api/v1/namespaces/default/secrets/db-zombie-pass uid: 72b890fd-b71f-11e6-84fe-2aa787ee170e type: Opaque
You might remember I used "zombie" as username but and I'm getting "em9tYmll" ... any idea..! base64 string encoding as mentioned above. Therefore, in order to check the values we must decode these values like we did before for the encoding. For instance, let’s decode the username:
➜ ~ echo 'em9tYmll' | base64 --decode zombie%
Well, as mentioned above, Secrets can be used either as mounted volumes or as environment variables which are the most used fashion of secrets in K8s ecosystem that we will describe in the current article.
As mounted volume:
- First of all, we need to create a secret as described above.
- Second, pod spec need to be modified to add a volume under the volumes array by specifying
the field secret.secretName to refer the name of the secret object.
- Third, we must affect the secret volume to each container in the pod under
containers.volumeMounts and we must specify also both containers.volumeMounts.readOnly = true so that the volume can be in mode read-only and the folder path of the mounted volume in containers.volumeMounts.mountPath
A final example of such setting using YAML spec looks like below:
apiVersion: v1 kind: Pod metadata: name: "mypod" namespace: "production" spec: containers: - name: mypod image: "redis" volumeMounts: - name: foo mountPath: "/etc/baz" readOnly: true volumes: - name: foo secret: secretName: "mysecret"
- If there are several containers which they need secret data, each one of them must specify volumeMounts
- It’s possible to bundle many files in one secret object or use many secrets in one pods spec file
- It’s also possible to use different keys within different files’ path, this concept is known as secret's keys projection. Now the username will stored under /etc/baz/specific- path/username instead of /etc/baz/username (see shell snippet below).
- Please, not that password is not projected and then it can not be used therefore, the rule is, once the items array is specified only the specified key from the secret will be available for the pod and its underlying containers
- If a specified key does not exist in the secret object the volume will never be created
… volumes: - name: foo secret: secretName: “mysecret” items: - key: “username” path: “specific-path/username”
As environment variables
Like for mounted volumes, we must put a little change to pods' spec file to be able to use secrets as env-variables inside pods and its underlying containers by adding env tag like illustrated below, let's call this spec file redis-pod.yaml:
- name: mycontainer
- name: SECRET_USERNAME
- name: SECRET_PASSWORD
Once the pods is created the env variables SECRET_USERNAME and SECRET_PASSWORD will available inside the pods and ready to use.
In the shell snippet below, I’ll create the pods from the spec file above and then ssh the pods to check the two env-variables using the command kubectl exec name_of_the_pods -i -t – sh.
➜ ~ kubectl create -f redis-pod.yaml pod "secret-env-pod" created ➜ ~ kubectl exec secret-env-pod -i -t -- sh # echo $SECRET_USERNAME zombie # echo $SECRET_PASSWORD 1f2d1e2e67df #
Please note that pods creation might take a little bit of time therefore, you might need to wait a little bit before being able to ssh the pods so be patient. The status of a specific pods can be checked by running the command kubectl get like below:
➜ ~ kubectl get pods secret-env-pod NAME READY STATUS RESTARTS AGE secret-env-pod 1/1 Running 0 5m
One of the real use case of using secret in the K8s ecosystem is to handle ssh public and private keys, to illustrate this, I’m going to generate a ssh RSA key let's say, for Gitlab and after that I’m going to create a secret object to store the private key as well as the public one:
➜ ~ ssh-keygen -t rsa -b 4096 -C “firstname.lastname@example.org” Generating public/private rsa key pair. Enter file in which to save the key (.ssh/id_rsa): gitlab_rsa Enter passphrase (empty for no passphrase): … ➜ ~ kubectl create secret generic ssh-key-secret --from-file=ssh-privatekey=gitlab_rsa --from-file=ssh-publickey=gitlab_rsa.pub secret "ssh-key-secret" created
This secret can than be used like bellow under volumes array in the pods' spec file:
spec: containers: - name: mypod image: "redis" volumeMounts: - name: foo mountPath: “/etc/ssh-secret-vol“ readOnly: true volumes: - name: foo secret: secretName: "ssh-key-secret"
Once the volumes are mounted the folders
/etc/ssh-secret- vol/gitlab_rsa.pub will be available.
Sometimes the user may need let’s say, a username and password credentials to perform some task for example: debugging, database inspection etc. this can be achieved using --from-literal argument like below:
➜ ~ kubectl create secret generic debugger-secret --from-literal=username=debugger --from-literal=password=super-strong-pwd secret "debugger-secret" created
I would like to conclude this article by saying that really the kubectl APIs is very well designed which makes it simple and especially easy to use for instance, even if I did not mention how we can manually delete a secret object the user might guess it from the used commands above such as kubectl get pods name_of_the_pods or kubectl create … which is Kubectl delete pods name_of_the_pods.
Regarding to the official documentation K8s bring more security precautions with secret objects under the hood for instance, avoiding creating secrets in disk as much as possible, sending a secret to only the pods requiring it, the secrets transfer is protected using internal SSL/TLS channel, K8s also ensures the update of secrets mounted as volumes or as environment variables whenever the associated secrets have been updated.