AWS EKS VPC IP’s vanish mystery

In my environment, the networking part is handled by another team So they created the 3 subnets with a CIDR of /21 total number of IP’s will be 6144 and the instance type for nodes was M5.large
and due to some promotion week(newspaper, advertisement, some offers), all the things were bombarding out the infrastructure and unfortunately, there was some issue in the nodes autoscaling group it was not working as expected so we are manually increasing the EKS cluster nodes

Majorly people are not aware of this that when ever we spin up and node and by default setting in AWS EKS it caches the IP’s and that depends on instance type for more details click here

Solutions

They were few solutions to overcome this issue

1. )Increase to large instance type
2. )Create the new subnet with a big CIDR range
3. )Change the CNI settings

1. )Increase to large instance type
   The instance type was M5.large instead of using this use another instance type c5.2xlarge and depending on your use case.
2. )Create the new subnet with a big CIDR range
   Create the new private subnet with a big CIDR range like /16 which have 65536 to calculate the CIDR range link

Note:- This will take some downtime.

3.)Change the CNI settings

I would recommend setting WARM_IP_TARGET and MINIMUM_IP_TARGET both to ‘10’ in the configuration for your aws-node DaemonSet. You would also want to remove WARM_ENI_TARGET. [1]

This is covered in the documentation at [1] and you will find previous correspondence on this case which dives deeper into the settings mentioned above.

Check and validation applied changes

Provided the below curl command which can be run from the shell of a worker node to get the details you’re looking for. This would need to be scripted to retrieve the data from each worker node remotely if you were looking to gather all the data on every node.

Resources:

[1] https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/eni-and-ip-target.md