AWS EKS VPC IP’s vanish mystery

In my environment, the networking part is handled by another team So they created the 3 subnets with a CIDR of /21 total number of IP’s will be 6144 and the instance type for nodes was M5.large
and due to some promotion week(newspaper, advertisement, some offers), all the things were bombarding out the infrastructure and unfortunately, there was some issue in the nodes autoscaling group it was not working as expected so we are manually increasing the EKS cluster nodes

Majorly people are not aware of this that when every we spin up and node and by default setting in AWS EKS it caches the IP’s and that depends on instance type for more details click here

Solutions

They were few solutions to overcome this issue

  1. )Increase to large instance type
  2. )Create the new subnet with a big CIDR range
  3. )Change the CNI settings
  1. )Increase to large instance type
    The instance type was M5.large instead of using this use another instance type c5.2xlarge and depends on your use case.
  2. )Create the new subnet with a big CIDR range
    Create the new private subnet with a big CIDR range like /16 which have 65536 to calculate the CIDR range link

Note:- This will take some downtime.

  1. )Change the CNI settings

I would recommend setting WARM_IP_TARGET and MINIMUM_IP_TARGET both to ‘10’ in the configuration for your aws-node DaemonSet. You would also want to remove WARM_ENI_TARGET. [1]

This is covered in the documentation at [1] and you will find previous correspondence on this case which dives deeper into the settings mentioned above.

Check and validation applied changes

Provided the below curl command which can be run from the shell of a worker node to get the details you’re looking for. This would need to be scripted to retrieve the data from each worker node remotely if you were looking to gather all the data on every node.

Resources:

[1] https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/eni-and-ip-target.md

--

--

--

DevOps engineer

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Guide to identify and solve dynamic programming problems in an interview

This is a screenshot of Leetcode problem 746. Min Cost Climbing Stairs

Working on My Portfolio!

Hidden Onion Service using Tor, nftables and NGINX

What I learnt from Firebase deployment

Ansible In Action

Filter Designing (Butterworth)

My First Tech Blog

Little Scraps of Wisdom

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Saket Jain

Saket Jain

DevOps engineer

More from Medium

Kubernetes test clusters In less than 20 seconds.

Managing and Troubleshooting AWS EKS Access

Connecting the Dots with Terraform(IaC)!!

From Monolith to Kubernetes Architecture — Part IV — GKE / GCP