5.4 Managing Secrets in Terraform with Vault, AWS Secrets Manager, or SSM

Kubernetes Secrets: Stop Hardcoding, Start Securing with Terraform!

Let's talk secrets. We're not talking James Bond-level clandestine meetings here. We're talking about your application's passwords, API keys, and other sensitive data. In Kubernetes, these are often handled as... well, Kubernetes Secrets!

But how do you manage these Secrets when you're building and deploying your infrastructure with Terraform? You wouldn't want to hardcode them in your Terraform configurations, would you? That's like leaving the keys to your house under the doormat!

This blog post will guide you through managing Kubernetes Secrets using Terraform and secure external secrets management solutions like Vault, AWS Secrets Manager, and SSM. We'll keep it simple, clear, and practical.

Why not just hardcode the secrets in Terraform?

Imagine Terraform is a blueprint for your house (your infrastructure). Would you write the combination to your safe (your secrets) directly on the blueprint? No! Anyone who gets hold of that blueprint would have access to everything. Same logic applies here!

The Goal: Securely inject secrets into Kubernetes using Terraform

We want to use Terraform to deploy our applications and Kubernetes Secrets, but without ever actually exposing the values of those secrets in our Terraform code.

The Players: Vault, AWS Secrets Manager, and SSM

Think of these as highly secure vaults, each with their own strengths. They store and manage your secrets securely.

• HashiCorp Vault: A comprehensive secrets management system that can store, generate, and lease secrets. It's like your personal, ultra-secure bank vault.

• AWS Secrets Manager: A managed secrets management service in AWS. Think of it as AWS providing a secure vault specifically for you.

• AWS Systems Manager Parameter Store (SSM): A service on AWS that provides secure, hierarchical storage for configuration data management and secrets management. It's more like a secure, organized file cabinet within AWS.

How does it work? (The General Idea)

1. Store Secrets: You store your sensitive data (database password, API key, etc.) in Vault, AWS Secrets Manager, or SSM.

2. Terraform Configuration: Your Terraform configuration reads the secrets from your chosen vault without ever storing the actual value in the Terraform state file.

3. Kubernetes Secret Creation: Terraform uses the retrieved secret value to create the Kubernetes Secret object.

4. Application Access: Your application then accesses the Kubernetes Secret in the standard way.

Architectural Diagram:

Real-World Example: Database Password

Let's say you need to create a Kubernetes Secret containing the password for your database. We'll use AWS Secrets Manager for this example.

1. Store the Password in AWS Secrets Manager:

   Create a secret in AWS Secrets Manager named "my-database-password" and store your database password as the secret value.

2. Terraform Configuration (using the AWS Provider):

    data "aws_secretsmanager_secret_version" "db_password" {
      secret_id = "my-database-password"
    }
   
    resource "kubernetes_secret" "db_secret" {
      metadata {
        name = "db-secret"
      }
      type = "Opaque"
      data = {
        password = data.aws_secretsmanager_secret_version.db_password.secret_string
      }
    }
   

   Explanation:

   • data "aws_secretsmanager_secret_version" "db_password": This data source retrieves the secret value from AWS Secrets Manager. The secret_id is the name you gave your secret in AWS Secrets Manager.

   • resource "kubernetes_secret" "db_secret": This creates the Kubernetes Secret named "db-secret".

   • data = { password = data.aws_secretsmanager_secret_version.db_password.secret_string }: This line is crucial. It populates the password field in the Kubernetes Secret with the value retrieved from AWS Secrets Manager. The actual password value is never stored directly in your Terraform code or the Terraform state file!

3. Apply the Terraform Configuration:

   Run terraform apply to create the Kubernetes Secret.

4. Your Application Accesses the Secret:

   Your application can now access the "db-secret" Kubernetes Secret and retrieve the database password.

Challenge and Solution: Secret Rotation

One big challenge is secret rotation. What happens when you need to change the database password? You'll need to update the secret in AWS Secrets Manager and have Terraform update the Kubernetes Secret.

Solution:

• Scheduled Terraform Runs: You can schedule Terraform runs to periodically refresh the Kubernetes Secrets with the latest values from your secrets manager. This ensures your Kubernetes Secrets stay synchronized with your external secret store.

• External Secrets Operator (ESO): ESO is a Kubernetes operator that automatically synchronizes secrets from external stores (like Vault, AWS Secrets Manager, etc.) into Kubernetes Secrets. It detects changes in the external store and automatically updates the Kubernetes Secret. This is the preferred method for robust secret rotation.

Key Takeaways

• Never hardcode secrets in Terraform.

• Use Vault, AWS Secrets Manager, or SSM to securely store and manage your secrets.

• Leverage Terraform data sources to retrieve secrets from these services.

• Implement a strategy for secret rotation to keep your secrets fresh and secure.

• Consider using External Secrets Operator for automated secret synchronization.

Managing Kubernetes Secrets with Terraform and external secrets managers might seem a bit complex at first, but it's essential for building secure and maintainable infrastructure. Start small, experiment with the examples, and choose the solution that best fits your needs. Happy deploying!