How to manage Kubernetes secrets on AWS EKS

Overview

Kubernetes has become a worldwide standard for running container workloads. Nevertheless, working with Kubernetes has its challenges such as dealing with sensitive information that your workload needs to perform proper operations on it and consuming the secret such as passwords, API keys and tokens etc. There are a lot of different solutions that exist in the marketplace to solve this specific problem such as HashiCorp Vault, AWS Secret Manager, Mozilla Secrets Operations etc.

To safely use secrets on Kubernetes, take at least the following steps:  

• Enable Encryption at Rest for Kubernetes secrets.  
• Enable or configure RBAC rules with least-privilege access to secrets.  
• Restrict secret access to specific containers.  
• Consider using external secret store providers.  

For more guidelines to manage and improve the security of your secrets, refer to Good practices for Kubernetes Secrets.

In this guide, we will use Mozilla SOPS to secure our secrets on EKS. Previously, you had to store your secrets as plaintext in configuration files because by default Kubernetes secrets are stored unencrypted in the API server’s underlying data store etcd. Anyone who has access to the API can retrieve or modify a secret and hence can access etcd (Etcd is a consistent and highly-available key value store used as Kubernetes’ backing store for all cluster data). In addition, it creates another problem i.e. if anyone authorised to create a Pod in a namespace can use that access to read any secret in that namespace; this includes indirect access such as the ability to create a deployment inside the Kubernetes cluster.

Pre-requisites:

We need a few prerequisites and tools to successfully run through these steps. Ensure you have the following in your working environment:

• An existing EKS cluster version 1.21 or higher  
• AWS Customer Managed Key  
• AWS RDS MySQL  
• Terraform  
• kubectl  
• eksctl  
• sops  
• helm  

What is SOPS and Why SOPS?

‍SOPS is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault, age, and PGP. There are too many options that exist in the markets for storing your secrets, however for Kubernetes SOPS is a very good one for the following reasons: it simplifies CI/CD pipelines, has ease of use, no complexity, and no scalability issues.

Encrypting and decrypting with SOPS

Let’s first try to understand how SOPS encryption works. And, how can we encrypt our secrets?
To test SOPS without having to do a bunch of setups, you can quickly deploy example files and the PGP key provided with the repository:

git clone https://github.com/mozilla/sops.git
cd sops
gpg - import pgp/sops_functional_tests_key.asc
sops example.yaml

In this guide, we will create a set of AWS customer-managed keys for different environments which will contain different environment secrets for our application and these KMS keys will be controlled and grant permission to specific IAM users or roles for encrypting and decrypting the secrets in a particular environment.

Before we begin, please make sure you have a clone of our repository from here: https://github.com/tal-lightlytics/eks-sops-demo
Under this repository, we will create a .sops.yaml configuration file at the root directory of our sops-demo folder to define which KMS keys are used for which filename.

For example:

• file name dev_secrets.yaml should use one set of KMS lightlytics-kms-development  
• file name staging_secrets.yaml should use one set of KMS lightlytics-kms-staging  
• file name preproduction_secrets.yaml should use one set of KMS lightlytics-kms-preproduction
• file name production_secrets.yaml should use one set of KMS lightlytics-kms-production

Under those circumstances, a file placed at sops-demo/.sops.yaml can manage the four sets of configurations for the four types of files:

creation_rules:
- path_regex: .*/development/*
kms: "arn:aws:kms:us-east-1:xxxxxxxxxxxx:alias/lightlytics-kms-demo"
- path_regex: .*/staging/*
kms: "arn:aws:kms:us-east-1:xxxxxxxxxxxx:alias/lightlytics-kms-staging"  
- path_regex: .*/preproduction/*
kms: "arn:aws:kms:us-east-1:xxxxxxxxxxxx:alias/lightlytics-kms-preproduction"  
- path_regex: .*/production/*
kms: "arn:aws:kms:us-east-1:xxxxxxxxxxxx:alias/lightlytics-kms-production"

To create the KMS Keys you can simply upload this CloudFormation template: https://github.com/tal-lightlytics/eks-sops-demo/blob/main/kms-cf.yaml and update the IAM user or IAM role permissions for individual KMS keys inside the CF template.

For example, if you want to give control to your developers to handle the secrets by themselves only in the development environment, then simply grant their IAM role or user ARN inside the KMS key policy for that specific key.

Using AWS RDS MySQL as Backend DB

So far, we got our EKS Cluster up and running, CMK keys, and we know how to encrypt or decrypt any secrets via SOPS and pass them as environment variables.
Now, we will deploy our application i.e. WordPress inside the EKS and for the backend database, we will be using AWS RDS MySQL.
To deploy quickly your RDS you can follow the ClickOps strategy or you can deploy IaC via Terraform from here: https://github.com/tal-lightlytics/eks-sops-demo/tree/main/tf-aws-rds-mysql

Why use RDS MySQL?

Database maintenance for your WordPress site is critical. Your database instance holds all of your important data for your WordPress site. If the database goes down, your website may go down with it, and you could even lose your data.

Database maintenance can also be difficult, and database administrators have years of specialized experience. When setting up a WordPress site, you want to stay focused on designing your page and generating your content, not worrying about database performance and backups.

Amazon RDS for MySQL helps with both of these problems. With Amazon RDS for MySQL, you get:  

• Automated backup and recovery so that you won’t lose data in the event of an accident  
• Regular updates and patches, keeping your database secure and performant  
• Easy installation with smart default parameters.  
• It provides cost-efficient and resizable capacity while automating time-consuming administration tasks such as hardware provisioning, database setup, patching and backups.

As WordPress stores data at the backend in the MySQL Database server. Therefore, you need to set up a MySQL server using AWS RDS service.
As part of the AWS Free Tier, the Amazon RDS Free Tier helps new AWS customers get started with a managed database service in the cloud for free. In addition, please make sure your RDS is running in the same VPC as your EKS Cluster.

Since we are using the secrets.yaml configuration inside our code and we want them to decrypt encrypted Helm value files on the fly which is possible via Helm secret plugin.

Once, you have the helm secret plugin installed. We can start deploying our helm-release by simply running the below command and pass the Dedicated values.yaml for environment-specific deployment.

For instance, If I want to create a release staging environment then I will pass all configuration files related to that specific environment.

Have a look at the code below.

helm secrets install WordPress-staging-demo ./kubernetes/sops-demo/  \      
        -f kubernetes/sops-demo/values/staging/values.yaml                        \      
         -f kubernetes/sops-demo/values/staging/staging_secrets.yaml

After installing the release we can see our WordPress demo application pod is up and running and it’s passing the writing environments variable for my deployment based on my environment type.

All these environment variables are set up through the ConfigMap configuration file for each environment specific and mapping them dynamically at the time of installing or upgrading the helm release.

apiVersion: v1  
kind: ConfigMap  
metadata:  
name: {{ include "sops-demo.fullname" . }}-cm  
labels:
{{- include "sops-demo.labels" . | nindent 4 }}  
data:  
{{- range $key, $val := .Values.environment }}  
{{ $key }}: {{ tpl $val $ | quote }}  
{{- end }}

Controlling your secrets information that is coming from config maps can be controlled via Role Based Access Control where you can create a specific Role or ClusterRole and control the granular level of access you wanted to grant to an individual or team members who can see this ConfigMaps in which namespaces.

Since our helm release is installed, let’s go for quick testing by simply doing the port-forwarding on your pod or service and see whether the application is configured correctly or not.

kubectl port-forward wordpress-pod-name -n namespace :8

Note: If you do not see this it means some error in the version not supported or you give the wrong credentials, or they are not in same VPC or some networking issue. In any of those cases, you need to debug further for troubleshooting and check your code again

Here are some best practices for storing secrets inside Kubernetes

1. Use Kubernetes Secrets: Use Kubernetes Secrets to store your sensitive data. Secrets is a Kubernetes resource specifically designed to store and manage secrets.

2. Limit access: Limit access to the secrets to only those who need it. This can be done by setting RBAC (Role-Based Access Control) rules.

3. Use encryption: Store your secrets in an encrypted format. Kubernetes Secrets are base64 encoded by default, but this encoding is not a form of encryption. To secure the secrets, use a tool like SOPS to encrypt the data and then store the encrypted file in a version control system like Git.

4. Use the least privilege: When configuring the access control rules for secrets, use the principle of least privilege. This means granting access only to the minimum set of users or applications that need it.

5. Rotate secrets: Regularly rotate the secrets to ensure that any unauthorized access to the data is limited. This can be done manually or using a tool like the Kubernetes External Secrets operator.

6. Don’t hardcode secrets: Avoid hardcoding secrets in your application code or deployment of YAML files. Instead, use environment variables to access the values of the secrets.

7. Monitor access: Monitor and audit access to secrets to detect any unauthorized access attempts or suspicious activities.

By following these best practices, you can effectively store secrets inside Kubernetes and ensure the security of your applications.

Conclusion

In conclusion, using SOPS to encrypt secrets inside Kubernetes provides a secure and automated way to manage sensitive information. By encrypting the data, you can store it in a version control system like Git without exposing it. The use of tools like kustomize, Helm, and the Kubernetes External Secrets operator makes it possible to manage the deployment process while ensuring that the secrets remain secure. Encrypting secrets using SOPS provides an additional layer of security, making it a recommended practice for managing sensitive data in a Kubernetes environment.