Upgrade EKS to 1.16 using Terraform

Table of contents

Reading Time: 5 minutes

Hello Readers, In this blog, I’ll be sharing the steps to upgrade EKS using Terraform.
Here, I’m using Terraform version v0.12.9

As we know that AWS keeps updating its EKS service, and keep on deprecating the older versions, so we also need to upgrade our EKS cluster service side by side.

In this blog, I’ll be sharing the steps to upgrade the EKS cluster but before that, here are some points to note.

1. Incremental Upgrade: The EKS needs to be upgraded incrementally, which means you can increment the version only by 1 at a time.
This means that you can follow this blog only if you are currently on EKS 1.15 version. If not, you need to upgrade to 1.14 first, and for that you can refer to my another blog on Upgrade EKS to 1.15 using Terraform.

2. Permissions: To upgrade the cluster, the controller node, from where you are running the Terraform commands, must have eks:UpdateClusterVersion permission. You can check the IAM roles for verifying this.

3. Deprecated API Versions: In the latest version of EKS, a number of ApiVersions for some Kubernetes objects have been deprecated. Let’s see these objects in detail.

Deprecated API Versions

Network Policy:
NetworkPolicy in extensions/v1beta1 API Version is no longer served. They need to use API Version as networking.k8s.io/v1.

PodSecurityPolicy:
PodSecurityPolicy in extensions/v1beta1 API Version is no longer served. They need to use API Version as policy/v1beta1.

DaemonSet:
DaemonSet in extensions/v1beta1 and apps/v1beta API Versions are no longer served. They need to use API Version as apps/v1.
Additionally, there are important changes inside the spec definition:

spec.templateGeneration is removed
spec.selector is required and immutable after creation
spec.updateStrategy.type now defaults to RollingUpdate

Deployment:
Deployment in extensions/v1beta1, apps/v1beta1 and apps/v1beta2 API Versions are no longer served. They need to use API Version as apps/v1. Additionally, here also there are important changes inside the spec definition:

spec.rollbackTo is removed
spec.selector is now required and immutable after creation
spec.progressDeadlineSeconds now defaults to 600 seconds
spec.revisionHistoryLimit now defaults to 10
maxSurge and maxUnavailable now default to 25%

StatefulSet:
StatefulSet in apps/v1beta1 and apps/v1beta2 API Versions are no longer served. They need to use API Version as apps/v1. Additionally, here also there are important changes inside the spec definition:

spec.selector is now required and immutable after creation
spec.updateStrategy.type now defaults to RollingUpdate

ReplicaSet:
ReplicaSet in extensions/v1beta1, apps/v1beta1 and apps/v1beta2 API Versions are no longer served. They need to use API Version as apps/v1. Additionally, here also there are important changes inside the spec definition:

spec.selector is now required and immutable after creation.

Now, In your running cluster, you need to make sure that all your running services must be running on supported API Version for the above listed objects and not the depricated version. Here’s a way to do that:

Upgrading deprecated APIs using `kubectl`

If you are simply using kubectl to manage your Kubernetes deployments, you can do the following

To check the API Version for above listed kubernetes object:
Note: You need to have jq installed on your controller node. If not, it can be installed by using the following command:
sudo apt install jq

kubectl get <object-name> --all-namespaces -o yaml | grep -A 1 last-applied-configuration| grep apiVersion | jq '.apiVersion+" "+.metadata.name'

## For eg: To get ApiVersion for all deployments, run:
kubectl get <object-name> --all-namespaces -o yaml | grep -A 1 last-applied-configuration| grep apiVersion | jq '.apiVersion+" "+.metadata.name'

Now, for those object, which are using deprecated APIs, you can upgrade them to use the new API version by using the following command on their manifest file.

kubectl convert -f <manifest-file.yaml> --output-version apps/v1 | kubectl apply -f -

Upgrading deprecated APIs using `helm`

If you are using helm for managing your Kubernetes deployments, you can do the following

Check for the helm charts of deployed services and verify if the above-listed objects are using the compatible API version. If not, change the API Version and re-deploy that chart.
Even if you don’t upgrade the services whose objects are using older API Versions, it won’t make any effect on them. They will still run fine. The only problem you’ll face is while upgrading such charts. Helm won’t allow you to upgrade those charts and then the only option left is to purge the chart and re-deploy it.

After the deprecated APIs are removed, you are now ready to upgrade the EKS version.

Step 1: Verify the current version is 1.15

To begin with the upgrade, you first need to verify if you are on EKS version 1.15. You can verify this either from AWS Console, or run the following command from the controller node.

kubectl version --short

Step 2: Update `version` value to 1.16

In the Terraform variables, you need to change the value of the variable corresponds to version from 1.15 to 1.16. Here’s how you will do it.

## main.tf
resource "aws_eks_cluster" "eks_cluster" {
  name        = var.cluster_name
  version     = var.cluster_version
}

## variables.tf
variable "cluster_version" {
  description = "Kubernetes version to use for the EKS cluster."
  default     = "1.16"
}

Step 3: Apply the Terraform changes

Now, to upgrade the cluster, ssh into the controller node and trigger the following commands.

cd <terraform-module-directory>
terraform init
terraform apply

Step 4: Verify the upgraded EKS version

After applying the terraform changes, you need to verify if the version has upgraded to 1.16 using the following command.

kubectl version --short

## Expected output
Client Version: v1.16.0
Server Version: v1.16.0

In case your client version is lower than 1.16, you need to download the new kubectl binaries using the following command.

KUBEPATH=$(which kubectl) && cd $(dirname $KUBEPATH) && curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.16.0/bin/linux/amd64/kubectl && cd - && unset KUBEPATH

Kubernetes Version	Kube-Proxy Version	CoreDNS Version	Amazon VPC CNI Version
1.15	v1.16.8	v1.6.6	v1.6.1

Components Version for EKS 1.16

Step 5: Upgrade `Kube-Proxy` version

Check the current Kube-Proxy version using this command:

kubectl describe ds kube-proxy  -n kube-system | grep Image | awk -F":" '{print$3}'

If the version is less than 1.16.8 , replace us-east-1 with your EKS region and run the following command.

kubectl patch daemonset kube-proxy -n kube-system
-p '{"spec": {"template": {"spec": {"containers": [{"image": "602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/kube-proxy:v1.16.8","name":"kube-proxy"}]}}}}'

Step 6: Upgrade `CoreDNS` version

Check the current CoreDNS version using this command:

kubectl describe deploy coredns -n kube-system | grep Image | awk -F":" '{print$3}'

If the version is less than 1.6.6 , replace us-east-1 with your EKS region and run the following command.

kubectl set image --namespace kube-system deployment.apps/coredns \
coredns=602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/coredns:v1.6.6

Step 7: Upgrade `Amazon VPC CNI` version

Check the current Amazon VPC CNI version using this command:

kubectl describe daemonset aws-node -n kube-system | grep Image | awk -F":" '{print$3}'

If the version is less than 1.6.1, run the following command.

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/master/config/v1.6/aws-k8s-cni.yaml

Final Step: Upgrading the nodes

Terminate the nodes one by one, and they will turn up with the newer version. To verify that, run the following command

Important Note: Make sure you terminate only those nodes which are stateless. For stateful services, you’ll need to figure out a way to back up the per-node data and then re-spawn that node else there are chances to lose your data.

kubectl get nodes

You’ll see that the version of nodes is now v1.16

These concludes all the steps you need to follow for upgrading EKS to v1.16. Still, if you have any doubts or queries, feel free to contact me at yatharth.sharma@knoldus.in.

Here are my references :
1. Amazon EKS Upgrade journey from 1.15 to 1.16 by Marcin Cuber

Also, I would like to thank you for sticking to the end. If you like this blog, please do show your appreciation by giving thumbs-ups and share this blog and give me suggestions on how I can improve my future posts to suit your needs. Follow me to get updates on different technologies.