How to work with Karpenter

Reading Time: 3 minutes

Karpenter automatically provisions new nodes in response to unschedulable pods. It can also monitor the events in the Kubernetes cluster and then, it will send commands to the underlying cloud provider.

Karpenter works with:

• Watching for pods that the Kubernetes scheduler has kept as unschedulable
• Evaluating scheduling constraints asked by the pods
• Provisioning nodes that fulfill the requirements of the pods
• Scheduling the pods to run on the new nodes
• Removing the nodes that we don’t need

It has two control loops that maximize the availability and efficiency of your cluster.

• Allocator – fast-acting controller
• Reallocator – slow-acting controller

Benefits of using it:

1. Designed to handle the full flexibility of the cloud: It can manage the full range of instance types available through AWS.
2. Group-less node provisioning: It can handle each instance directly, without using extra orchestration tools like node groups.
3. Scheduling enforcement

Required Utilities for it:

1. AWS CLI
2. kubectl – the k8s CLI
3. eksctl – the CLI for AWS EKS
4. helm – the package manager for K8s

Steps that you need to follow:

Step 1: Create a cluster with eksctl

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: ${CLUSTER_NAME}
  region: ${AWS_DEFAULT_REGION}
  version: "1.21"
  tags:
    karpenter.sh/discovery: ${CLUSTER_NAME}
managedNodeGroups:
  - instanceType: m5.large
    amiFamily: AmazonLinux2
    name: ${CLUSTER_NAME}-ng
    desiredCapacity: 1
    minSize: 1
    maxSize: 10
iam:
  withOIDC: true

Step 2: Create a KarpenterNode IAM Role

TEMPOUT=$(mktemp)

curl -fsSL https://karpenter.sh/"${KARPENTER_VERSION}"/getting-started/getting-started-with-eksctl/cloudformation.yaml  > $TEMPOUT \
&& aws cloudformation deploy \
  --stack-name "Karpenter-${CLUSTER_NAME}" \
  --template-file "${TEMPOUT}" \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides "ClusterName=${CLUSTER_NAME}"

eksctl create iamidentitymapping \
  --username system:node:{{EC2PrivateDNSName}} \
  --cluster "${CLUSTER_NAME}" \
  --arn "arn:aws:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME}" \
  --group system:bootstrappers \
  --group system:nodes

Step 3: Create a KarpenterController IAM Role

eksctl create iamserviceaccount \
  --cluster "${CLUSTER_NAME}" --name karpenter --namespace karpenter \
  --role-name "${CLUSTER_NAME}-karpenter" \
  --attach-policy-arn "arn:aws:iam::${AWS_ACCOUNT_ID}:policy/KarpenterControllerPolicy-${CLUSTER_NAME}" \
  --role-only \
  --approve

export KARPENTER_IAM_ROLE_ARN="arn:aws:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-karpenter"

Step 4: Installing Karpenter Helm Chart

helm repo add karpenter https://charts.karpenter.sh/
helm repo update

Next, install the chart passing in the cluster and the Karpenter role ARN.

helm upgrade --install --namespace karpenter --create-namespace \
  karpenter karpenter/karpenter \
  --version ${KARPENTER_VERSION} \
  --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=${KARPENTER_IAM_ROLE_ARN} \
  --set clusterName=${CLUSTER_NAME} \
  --set clusterEndpoint=${CLUSTER_ENDPOINT} \
  --set aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-${CLUSTER_NAME} \
  --wait # for the defaulting webhook to install before creating a Provisioner

Step 5: Provisioner

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  requirements:
    - key: karpenter.sh/capacity-type
      operator: In
      values: ["spot"]
  limits:
    resources:
      cpu: 1000
  provider:
    subnetSelector:
      karpenter.sh/discovery: ${CLUSTER_NAME}
    securityGroupSelector:
      karpenter.sh/discovery: ${CLUSTER_NAME}
  ttlSecondsAfterEmpty: 30

Karpenter is now active and ready to start provisioning nodes. We can Create pods using a deployment, and can also watch provision nodes in reply.

For more, you can refer to: https://karpenter.sh/v0.9.1/

Also, For a more technical blog, you can refer to the knoldus blog: https://blog.knoldus.com/

In conclusion, how it can automatically provision new nodes and can monitor it also. If you liked this blog, Please like and share, and if you have any doubts feel free to reach out to me.