Building Golden AMIs with Packer and Terraform for EKS Node Groups

When you're running EKS in production, the default Amazon Linux AMIs work - but they come with tradeoffs that'll bite you at the worst possible moment.
Every node spends precious minutes at boot time installing monitoring agents, pulling container images, and applying configurations. Sounds harmless until you're watching a scaling event where 20 nodes are spinning up during a traffic spike, each one taking 5-7 minutes to become ready. Your users are waiting. Your autoscaler is panicking. And you're wondering why you didn't fix this sooner.
Golden AMIs solve this problem elegantly. You bake everything once with Packer, then deploy consistently with Terraform. Nodes join your cluster in under 2 minutes instead of 7. Every node is identical. No more "works on some nodes" debugging sessions.
_____
Why Golden AMIs for EKS?
Let me break down why this approach matters:
- Faster node startup - Pre-installed agents and cached container images mean nodes are ready almost immediately. When your cluster needs to scale, it actually scales.
- Security compliance - You can bake in CIS benchmarks, pre-configure security agents, and lock down package versions. Your security team will thank you.
- Consistency - Every single node starts from the same baseline. No configuration drift. No mysterious differences between nodes.
- Reduced external dependencies - No boot-time downloads that fail because a package repository is having a bad day. Everything your node needs is already on disk.
______
Architecture Overview
The flow is straightforward:

Packer builds the image with all your customizations. The AMI gets stored in your AWS account. Terraform references that AMI when creating node groups. Simple, repeatable, version-controlled.
_____
Part 1: Building the Golden AMI with Packer

The Packer Template
Create a file called eks-node.pkr.hcl. This is where the magic happens:
1packer {
2 required_plugins {
3 amazon = {
4 version = ">= 1.2.0"
5 source = "github.com/hashicorp/amazon"
6 }
7 }
8}
9
10variable "eks_version" {
11 type = string
12 default = "1.29"
13}
14
15variable "aws_region" {
16 type = string
17 default = "ap-south-1"
18}
19
20# Fetch the latest EKS-optimized AMI as our base
21data "amazon-ami" "eks_base" {
22 filters = {
23 name = "amazon-eks-node-${var.eks_version}-*"
24 virtualization-type = "hvm"
25 architecture = "x86_64"
26 }
27 owners = ["602401143452"] # AWS EKS AMI account
28 most_recent = true
29 region = var.aws_region
30}
31
32source "amazon-ebs" "eks_node" {
33 ami_name = "eks-golden-${var.eks_version}-{{timestamp}}"
34 instance_type = "t3.medium"
35 region = var.aws_region
36 source_ami = data.amazon-ami.eks_base.id
37 ssh_username = "ec2-user"
38
39 ami_description = "Golden AMI for EKS ${var.eks_version} nodes"
40
41 tags = {
42 Name = "eks-golden-${var.eks_version}"
43 EKSVersion = var.eks_version
44 BuildTime = "{{timestamp}}"
45 ManagedBy = "Packer"
46 }
47}
48
49build {
50 sources = ["source.amazon-ebs.eks_node"]
51
52 # Update system packages
53 provisioner "shell" {
54 inline = [
55 "sudo yum update -y",
56 "sudo yum install -y aws-cli jq wget"
57 ]
58 }
59
60 # Install monitoring agents
61 provisioner "shell" {
62 script = "scripts/install-cloudwatch-agent.sh"
63 }
64
65 # Install security tools
66 provisioner "shell" {
67 script = "scripts/install-security-agents.sh"
68 }
69
70 # Pre-cache common container images
71 provisioner "shell" {
72 script = "scripts/cache-container-images.sh"
73 }
74
75 # Apply CIS hardening
76 provisioner "shell" {
77 script = "scripts/cis-hardening.sh"
78 }
79
80 # Cleanup
81 provisioner "shell" {
82 inline = [
83 "sudo yum clean all",
84 "sudo rm -rf /var/cache/yum",
85 "sudo rm -rf /tmp/*"
86 ]
87 }
88}Notice how we're starting with the official EKS-optimized AMI and layering our customizations on top. This ensures we always have the correct kubelet version and EKS bootstrap scripts.
__
Key Provisioning Scripts
CloudWatch Agent Installation (scripts/install-cloudwatch-agent.sh):
1#!/bin/bash
2set -e
3
4# Install CloudWatch agent
5sudo yum install -y amazon-cloudwatch-agent
6
7# Add base configuration
8cat <<EOF | sudo tee /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
9{
10 "logs": {
11 "logs_collected": {
12 "files": {
13 "collect_list": [
14 {
15 "file_path": "/var/log/messages",
16 "log_group_name": "/eks/nodes/system"
17 }
18 ]
19 }
20 }
21 }
22}
23EOFContainer Image Caching (scripts/cache-container-images.sh):
This script is the real time-saver. Pre-pulling common images means pods schedule instantly instead of waiting for image pulls:
1#!/bin/bash
2set -e
3
4# Pre-pull common images to speed up pod scheduling
5IMAGES=(
6 "602401143452.dkr.ecr.ap-south-1.amazonaws.com/amazon-k8s-cni:v1.15.1"
7 "602401143452.dkr.ecr.ap-south-1.amazonaws.com/eks/kube-proxy:v1.29.0"
8 "public.ecr.aws/eks/aws-load-balancer-controller:v2.7.0"
9)
10
11# Authenticate with ECR
12aws ecr get-login-password --region ap-south-1 | \
13 sudo docker login --username AWS --password-stdin 602401143452.dkr.ecr.ap-south-1.amazonaws.com
14
15for image in "${IMAGES[@]}"; do
16 echo "Pre-caching: $image"
17 sudo ctr --namespace k8s.io image pull "$image" || true
18done___
Building the AMI
Once your template and scripts are ready:
1# Initialize Packer plugins
2packer init eks-node.pkr.hcl
3
4# Validate the template (catch errors early)
5packer validate eks-node.pkr.hcl
6
7# Build the AMI
8packer build -var="eks_version=1.29" eks-node.pkr.hclPacker will output the AMI ID when it finishes. Save that - you'll need it for Terraform.
_____
Part 2: Deploying with Terraform
EKS Module with Custom AMI
Here's how to wire up your golden AMI with the popular terraform-aws-modules EKS module:
1# variables.tf
2variable "golden_ami_id" {
3 description = "Golden AMI ID built by Packer"
4 type = string
5}
6
7variable "cluster_name" {
8 type = string
9 default = "production-eks"
10}1# eks.tf
2module "eks" {
3 source = "terraform-aws-modules/eks/aws"
4 version = "~> 20.0"
5
6 cluster_name = var.cluster_name
7 cluster_version = "1.29"
8
9 vpc_id = module.vpc.vpc_id
10 subnet_ids = module.vpc.private_subnets
11
12 eks_managed_node_groups = {
13 application = {
14 name = "app-nodes"
15 instance_types = ["t3.large"]
16
17 min_size = 2
18 max_size = 10
19 desired_size = 3
20
21 # This is where we use our Golden AMI
22 ami_id = var.golden_ami_id
23 enable_bootstrap_user_data = true
24
25 # Additional bootstrap arguments
26 bootstrap_extra_args = "--kubelet-extra-args '--node-labels=workload=application'"
27
28 labels = {
29 Environment = "production"
30 ManagedBy = "terraform"
31 }
32
33 tags = {
34 AMIType = "golden"
35 }
36 }
37 }
38}__
Automating AMI Discovery
Hardcoding AMI IDs gets old fast. Instead, have Terraform discover the latest golden AMI automatically:
1# data.tf
2data "aws_ami" "golden_eks" {
3 most_recent = true
4 owners = ["self"]
5
6 filter {
7 name = "name"
8 values = ["eks-golden-1.29-*"]
9 }
10
11 filter {
12 name = "tag:ManagedBy"
13 values = ["Packer"]
14 }
15}
16
17# Then reference: data.aws_ami.golden_eks.idNow whenever you build a new AMI with Packer, Terraform will automatically pick up the latest version on the next apply.
_____
Part 3: CI/CD Pipeline Integration
Automate the entire flow with GitHub Actions:
1# .github/workflows/golden-ami.yml
2name: Build Golden AMI
3
4on:
5 schedule:
6 - cron: '0 2 * * 0' # Weekly builds every Sunday at 2 AM
7 workflow_dispatch: # Manual trigger option
8
9jobs:
10 build:
11 runs-on: ubuntu-latest
12 steps:
13 - uses: actions/checkout@v4
14
15 - name: Configure AWS credentials
16 uses: aws-actions/configure-aws-credentials@v4
17 with:
18 role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
19 aws-region: ap-south-1
20
21 - name: Setup Packer
22 uses: hashicorp/setup-packer@main
23
24 - name: Build AMI
25 run: |
26 packer init eks-node.pkr.hcl
27 packer build -machine-readable eks-node.pkr.hcl | tee build.log
28
29 - name: Extract AMI ID
30 run: |
31 AMI_ID=$(grep 'artifact,0,id' build.log | cut -d: -f2)
32 echo "New Golden AMI: $AMI_ID"Weekly builds ensure you pick up the latest security patches from the base EKS AMI while keeping your customizations.
_____
Best Practices
After running this pattern in production, here's what I've learned:
- Version your AMIs - Include EKS version and build timestamp in AMI names. When something goes wrong (and it will), you need to know exactly which AMI version is running and have a path to rollback.
- Test before promoting - Build AMIs in a dev account first, run validation tests, then copy to production accounts. Never deploy an untested AMI to production.
- Rotate regularly - Security patches matter. Schedule weekly or bi-weekly builds to stay current.
- Keep the base minimal - Only bake what's truly needed at boot time. Configuration that changes frequently should stay in user-data or SSM Parameter Store. The goal is stability, not cramming everything into the AMI.
- Tag extensively - Include metadata like EKS version, build date, git commit SHA, and anything else that helps you trace an AMI back to its source.
_____
Wrapping Up
Golden AMIs transform EKS operations from reactive firefighting to proactive infrastructure management. Nodes boot faster. Environments stay consistent. Security teams sleep better knowing every node starts from an audited, hardened baseline.
The combination of Packer for building and Terraform for deploying gives you a fully automated, version-controlled infrastructure pipeline. Start with the basics - monitoring agents and security tools - then expand to image caching and compliance hardening as your needs grow.
Your future self, watching nodes join the cluster in under 2 minutes during that 3 AM scaling event, will thank you.
