Kubernetes Secrets on OpenStack

In this blog:

• What Are Secrets
• Kubernetes Secrets
• Creating a Kubernetes Secret
• OpenStack Key Manager (Barbican)
• Getting Started With OpenStack Barbican on OpenMetal
• Configuring Barbican and Kubernetes

For any containerized, or for that matter, non-containerized application that you’re deploying, chances are you’ll have some sort of sensitive information that you need to pass into the app. Whether it’s a password, API key, connection string, or any form of sensitive data. It could even be sensitive data that you need for the app to authenticate elsewhere. Because of that, you’ll need a way to store that sensitive data for your containerized workloads. 

In this blog post, you’ll learn about what secrets are, how to create standard Kubernetes secrets, and how to get started with the OpenStack Key Manager.

Prerequisites

To follow along with this blog post, you’ll have to set up the Kolla environment. The Kolla environment configuration is needed to prepare the environment to run Kolla-ansible. To do that, you can follow the the Kolla-ansible guide from OpenMetal found here.

What Are Secrets

Secrets can consist of any data that you don’t want others to see, that shouldn’t be plain-text, or that should be encrypted at rest and in-transit. Typical secrets are: 

• Passwords 
• API keys 
• Connection strings 
• Database passwords 

However, any piece of configuration data that you’d like to consider a secret can be a secret. For example, maybe you want a username to be encrypted and not shown in plain-text. You could create a secret that’s value is the username and utilize it as a secret. 

When thinking about secrets, it’s essentially any data that you want to stay hidden from everything and everyone other than what/who needs it. 

If you have an application that needs to connected to a database to store data, the app will need a database connection string to authenticate. You wouldn’t want to put the database connection string as a plain string in source control. Instead, you’d want to store it as a secret and have your app call upon the secret wherever it may be stored.

Kubernetes Secrets

Kubernetes secrets are a k8s object/resource used to store any sensitive data that you don’t want transmitted/in-transit as plain-text. You can then use the secret inside of a Pod. When you use a secret, you don’t have to put any sensitive data inside of your application code or store any sensitive data in source control, which is an ideal scenario from a security perspective. 

The pitfall of Kubernetes secrets is by default, they have the Opaque standard turned on. The Opaque standard, by default, does not encrypt secrets at rest. That means when the Kubernetes secret is used by a Pod, it’s encrypted. However, at rest, the secret is stored in plain-text, which means any bad entity can see it and use it for a malicious purpose. This is a huge reason why engineers typically stay away from Kubernetes secrets and use another secrets manager solution, which you’ll learn about in an upcoming section.

Creating A Kubernetes Secret

To create a Kubernetes secret, there are two primary ways: 

• A Kubernetes Manifest 
• The CLI 

Below is how you can create a Kubernetes secret with a Kubernetes Manifest. It utilizes the Secret resource from Kubernetes Core API group. 

If you save the below Manifest as, for example, secrets.yaml and deploy it, you’ll have a newly created Kubernetes secret.

apiVersion: v1 
kind: Secret 
metadata: 
name: testsecret 
type: Opaque 
data: 
username: admin 
password: MySuperSecretPassword123 

Aside from using a Kubernetes Manifest, you can create a Kubernetes secret right on the terminal. The below command will create a Kubernetes secret called openstack-pass with a username of admin and a password of SuperSecretPassword12 .

kubectl create secret generic openstack-pass \ 

--from-literal=username=admin \ 

--from-literal=password='SuperSecretPassword12' 

you can also use the –-from-file flag, which can consume the username and password from a plain-text file. 

There’s another huge problem with the above two methods for creating secrets – both of the ways you create the secrets are in plain-text. Aside from Opaque storing them in plain-text, you have to create the secrets as plain-text. If you create a secret using a Kubernetes Manifest, the password is stored right in plain-text. If you use the CLI, you either have to pass in the value in plain-text, or you have to store it in a plain-text file. 

Kubernetes secrets are insecure and not a good way of managing secret values.

OpenStack Key Manager (Barbican)

Now that you know what secrets are and how Kubernetes deals with secrets, let’s talk about the OpenStack secrets manager Barbican. Barbican is a REST API that is designed for secure storage. You can use it to manage and provision anything from passwords to X.509 certificates. You can store: 

• Symmetric keys 
• Asymmetric keys 
• Certificates 
• Raw secrets 

If you’re using OpenStack and deploying Kubernetes clusters on OpenStack, it makes sense to use Barbican as it’s ready out of the box in OpenStack environments and you don’t have to worry about setting up any environments or paying for any products.

 Barbican was designed with the idea in mind that the current state of key and secrets management is not the best. There are some good solutions out there, but solutions out of the box on Linux are almost non-existent. This problem was found from an internal Rackspace need, and with that, development started on fixing the problem. The product became Barbican.

Getting Started With OpenStack Barbican on OpenMetal

Let’s take a look at how you can incorporate Barbican with Kubernetes for a successful secrets management deployment. 

OpenMetal configures a few different settings for OpenStack out of the box, and one of the configurations is to use Kolla Ansible. Kolla Ansible is used to create production ready containers and dev environment for OpenStack. In this case, you can use Kolla Ansible to get Barbican installed.

Preparing Barbican

First, SSH into your OpenStack environment via the OpenMetal platform.

Next, use Vim to open up the Kolla config at /etc/kolla/globals.yml 

sudo vim /etc/kolla/globals.yml 

Add in the following line in globals.yml to ensure Barbican is turned on as an option.

 enable_barbican: 'yes' 

Save the file and run the following command to run the Ansible Playbook inside of the Python virtual environment that was created via the Prerequisite instructions. 

cd /opt/kolla-ansible 

Initialize a Python virtual environment 

virtualenv .venv 

source .venv/bin/activate 

kolla-ansible -i /etc/fm-deploy/kolla-ansible-inventory reconfigure

You should see an output similar to the screenshot below (the screenshot was cut off due to verbosity). 

If you see that localhost gets an update, but one of the servers do not, it can most likely be because you’re running the Playbook on the host itself instead of remotely, which would cause an output of changed=0 and ok=0 on the host you’re running the Playbook on, but localhost will update. 

Configuring Barbican and Kubernetes 

Now that Barbican is set up and configured, let’s learn how to create a configuration that Kubernetes can use to create secrets with Barbican. 

SSH into your OpenStack server as you’ll need to run commands from there on the terminal. 

First, verify that the Barbican endpoints exist. 

openstack endpoint list --service barbican 

Next, install the Barbican Python client using Pip. 

pip install python-barbicanclient 

To validate that secret creation is working properly with Barbican, try creating a new secret with the OpenStack CLI. 

openstack secret store --name my_secret --payload 'This is a secure statement' 

Next, you’ll need to set up the KMS container/service. Without it, Kubernetes has no method to communicate with Barbican. Create a 256bit CBC key and store it in Barbican.

openstack secret order create --name k8s_key --algorithm aes --mode cbc --bit-length 256 --payload-content-type=application/octet-stream key 

Retrieve the key-id via the  uuid in the  Secret href output.

openstack secret order get http://hostname:9311/v1/orders/e477a578-4a46-4c3f-b071-79e220207b0e 

Save the key-id value as you’ll be using it for the configuration coming up in the next step. 

Using, vim , open the cloud-config in /etc/kubernetes .

vim /etc/kubernetes/cloud-config 

Next, you’ll have to set up a global configuration that authenticates to your Kubernetes cluster from OpenStack. Below is an example of a configuration. 

[Global] 
auth-url=http://200.225.44.4:5000/v3 
username=<REDACTED> 
password=<REDACTED> 
region=iad3 
tenant-name=<REDACTED> 
domain-id=default 
tls-Insecure=true 

[BlockStorage] 
bs-version=v2 

[KeyManager] 
key-id = <REDACTED> 

Once the configuration is in place, verify that the /var/lib/kms/kms.sock exists.

docker run -d --volume=/var/lib/kms:/var/lib/kms \ 
--volume=/etc/kubernetes:/etc/kubernetes \ 
-e socketpath=/var/lib/kms/kms.sock \ 
-e cloudconfig=/etc/kubernetes/cloud-config \ 
docker.io/k8scloudprovider/barbican-kms-plugin-amd64:latest

For the last step, you can create a Kubernetes Manifest to create a new secret with the EncryptionConfig Kubernetes resource. 

Create a new file called encrypt-config.yaml 

vim /etc/kubernetes/encryption-config.yaml

Inside of the encryption-config.yaml file, add in the following YAML. 

apiVersion: v1 
kind: EncryptionConfig 
resources: 
    - resources: 
       - secrets 
      providers: 
      - kms: 
          name : barbican 
          endpoint: unix:///var/lib/kms/kms.sock 
          cachesize: 100 
      - identity: {}

To deploy the Kubernetes Manifest, run the following:

kubectl apply -f encryption-config.yaml

Enable –-experimental-encryption-provider-config flag in the Kubernetes Control Plane (API Server) and restart it.

Barbican is now successfully implemented as a Kubernetes secrets option in OpenStack.

About Our Guest Writer

Michael Levan is a consultant, researcher, and content creator with a career that has spanned countless business sectors, and includes working with companies such as Microsoft, IBM, and Pluralsight. An engineer at heart, he is passionate about enabling excellence in software development, SRE, DevOps, and actively mentors developers to help them realize their full potential. You can learn more about Michael on his website at: https://michaellevan.net/

Interested in Learning More?

OpenMetal and OpenStack are the perfect fit for Kubernetes. But don’t take our word for it:

• Schedule a meeting with the OpenMetal team to discuss your unique needs or a custom Proof of Concept (PoC)
• Request an OpenMetal On-Demand Private Cloud Trial to experience the performance and benefits for yourself or your organization.