So the day has come when I noticed that one of my pods was not running and I received the above mentioned message insufficient pods”.
What I then realised was that I run out of maximal number of pods I can run :O which in AWS EKS is associated with ENI value.
To get the number of maximal pods you can run execute the following:
❯ kubectl get node -o yaml | grep pods
pods: "17" => this is allocatable pods that can be allocated in node
pods: "17" => this is how many running pods you have created
The details of number of pods per instance can be found via https://github.com/awslabs/amazon-eks-ami/blob/master/files/eni-max-pods.txt
In kubernetes v1.19 we will get GA of EvenPodsSpread which will definitely help in managing how pods are distributed
Also In my troubleshooting I found helpful to use some of the below scripts.
# find number of pods running per node
❯ kubectl get pod --all-namespaces -o json | jq -r '.items[] |select( .kind=="Pod")| "\(.status.hostIP),\(.metadata.name)"'| awk -F, '{a[$1]++;}END{for (i in a)print i, a[i];}'
# find pods running on specific node
> kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=ip-10-10-1-55.eu-central-1.compute.internal
# Find pods running wiuth specific status ( or not )
> kubectl get pods --all-namespaces -o wide --field-selector status.phase!=Running