5.3 Terraform Provisioners: When and How to Use Them Safely

Terraform Provisioners and Kubernetes: Using Them Wisely (and Safely!)

Okay, you're using Terraform to manage your Kubernetes infrastructure – awesome! You're declaring what you want and Terraform makes it happen. But sometimes, you need to go beyond just creating resources like deployments, services, and namespaces. Sometimes you need to configure something after it's created. That's where Terraform provisioners come in.

Think of it like this: Terraform is building a house (your Kubernetes cluster). It can lay the foundation (VPC), put up the walls (nodes), and install the plumbing (ingress controllers). But what if you need to configure the smart home system inside the house after it's built? That's where provisioners come in.

What are Terraform Provisioners?

Terraform provisioners are like mini-scripts that run after a resource is created. They allow you to perform actions on that resource, like:

Running commands remotely: Installing software, starting services, configuring files.
Copying files: Uploading configuration files, scripts, or data to the resource.
Executing local commands: Running scripts on your local machine that interact with the resource.

When to Use Provisioners (and When to Avoid Them)

The golden rule: Use provisioners as a last resort. Why? Because they introduce complexities and potential for errors. Ideally, your Kubernetes applications should be configured through declarative methods (e.g., ConfigMaps, Secrets, Helm charts) that are managed directly by Kubernetes.

Here are scenarios where provisioners might be necessary:

Bootstrapping: Initializing a new cluster or node with essential software.
Legacy Systems: Interacting with systems that don't have a good Terraform provider.
One-Off Configuration: Performing a specific configuration task that isn't easily automated through other means.

Types of Provisioners (The Basics)

Let's look at the most common provisioners:

remote-exec: Executes commands on a remote host using SSH or other protocols. This is the most common and potentially most problematic.
local-exec: Executes commands on your local machine. Useful for tasks like generating certificates or preparing configuration files.
file: Copies files from your local machine to a remote host.

A Practical Example: Initializing a Kubernetes Node with a Custom Package

Let's say you have a custom monitoring agent you want installed on each Kubernetes worker node during cluster creation.

resource "aws_instance" "worker_node" {
  ami                    = "ami-0c55b5f8b1e5456a3"  # Example AMI
  instance_type          = "t3.medium"
  key_name               = "my-ssh-key"

  provisioner "remote-exec" {
    connection {
      type        = "ssh"
      user        = "ec2-user"  # Adjust user for your AMI
      private_key = file("~/.ssh/id_rsa")
      host        = self.public_ip
    }

    inline = [
      "sudo yum update -y",
      "sudo yum install -y my-custom-agent"  # Replace with your agent install
    ]
  }
}

Explanation:

We're defining an aws_instance representing a worker node.
The remote-exec provisioner connects to the node using SSH.
The inline section specifies a list of commands to run on the node after it's created. We're updating the package manager and installing our custom agent.

Important Safety Considerations: Handle with Care!

Provisioners can be tricky! Here's how to use them safely:

Idempotency: Ensure your provisioner scripts are idempotent. This means running them multiple times has the same effect as running them once. Use checks within your scripts to avoid errors if they're accidentally re-run.
Connection Details: Securely manage your SSH keys and other connection credentials. Avoid hardcoding sensitive information in your Terraform code. Use variables and external secrets management.
Error Handling: Implement proper error handling in your provisioner scripts. Don't just let errors silently fail. Log errors and consider failing the Terraform apply if a critical error occurs.
Use only_if or when Attributes (if available): Certain provisioners may have conditions to control execution. Use these to ensure provisioners only run under appropriate conditions.
Test Thoroughly: Test your Terraform code, including provisioners, in a non-production environment before deploying to production.

A Common Challenge: Connection Issues

Challenge: Often, you'll face connection issues during the provisioner execution. This might be due to network connectivity problems, SSH key issues, or the instance not being fully ready when the provisioner tries to connect.

Solution:

depends_on: Use depends_on to ensure that the resource is fully created before the provisioner runs. For example, if you are creating a VPC, ensure that the VPC is fully available.

retry blocks: Retry the connection attempt with a delay and a maximum number of attempts. Here's how to do it in the connection block:

  connection {
    type        = "ssh"
    user        = "ec2-user"
    private_key = file("~/.ssh/id_rsa")
    host        = self.public_ip
    timeout     = "5m"  # Overall timeout for the connection

    # Retry the connection a few times with a delay
    retry {
      attempts = 3
      delay    = "30s"
    }
  }

Alternatives to Provisioners: Embrace Declarative Solutions!

Before reaching for provisioners, always explore declarative alternatives within Kubernetes itself:

ConfigMaps and Secrets: Use these to inject configuration data into your Pods.
Init Containers: Run special containers that perform initialization tasks before your main application container starts.
Operators: Build custom Kubernetes controllers that automate complex configuration and management tasks.
Helm Charts: Package and deploy applications with pre-defined configurations.

In Conclusion:

Terraform provisioners are powerful tools, but they should be used with caution. Prioritize declarative solutions and only resort to provisioners when absolutely necessary. By following these guidelines, you can ensure your Kubernetes infrastructure is managed safely, reliably, and efficiently. Remember to always test, test, and test again!