0

Kubernetes with Calico using kind

Recently after analysing requirements for application that I manage I realised I’m in need of a way to secure communication within my cluster – so in a nutshell is not an open wilderness.

While looking at several alternatives one was very appealing especially after watching the following video….

And yes it is project Calico.

So I decided to do some more testing with it. And spin it up in a locally running cluster. To have some more fun this time – there are more nodes πŸ™‚

The difference in the below config is that we disable the default CNI.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  podSubnet: "10.240.0.0/16"
  disableDefaultCNI: true
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true,zone=cookie,region=oo-space-1"
  extraPortMappings:
  - containerPort: 30080
    hostPort: 88
    protocol: TCP
  - containerPort: 30443
    hostPort: 444
    protocol: TCP
- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "zone=alpha,region=eu-west-1"
- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "zone=alpha,region=eu-west-1"
- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "zone=beta,region=eu-west-1"
- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "zone=beta,region=eu-west-1"
- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "zone=gamma,region=eu-centra
l-1"
- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "zone=gamma,region=eu-central-1"
- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "zone=gamma,region=eu-central-1"

Once the cluster is up and running I used kapp to deploy Calico by issuing the following command:

kapp deploy -a calico -f <(curl https://docs.projectcalico.org/v3.17/manifests/calico.yaml)

Shortly after the nodes applied configuration change Calico was running on all nodes

That gets you going right away! But in order to really understand now the power you have I can highly recommend looking at example networkPolicies

Once you have done that there is also a great tool to validate not only NetworkPolicies but your kubernetes cluster configuration in general called sonobuoy

sonobuoy run --e2e-focus "NetworkPolicy" --e2e-skip ""

Happy securing of your k8s cluster!

0

AWS – EKS and “insufficient pods”

So the day has come when I noticed that one of my pods was not running and I received the above mentioned message insufficient pods”.

What I then realised was that I run out of maximal number of pods I can run :O which in AWS EKS is associated with ENI value.

To get the number of maximal pods you can run execute the following:

❯ kubectl get node -o yaml | grep pods
      pods: "17" => this is allocatable pods that can be allocated in node
      pods: "17" => this is how many running pods you have created

The details of number of pods per instance can be found via https://github.com/awslabs/amazon-eks-ami/blob/master/files/eni-max-pods.txt

In kubernetes v1.19 we will get GA of EvenPodsSpread which will definitely help in managing how pods are distributed

Also In my troubleshooting I found helpful to use some of the below scripts.

# find number of pods running per node 
❯ kubectl get pod --all-namespaces -o json | jq -r '.items[] |select( .kind=="Pod")| "\(.status.hostIP),\(.metadata.name)"'| awk -F, '{a[$1]++;}END{for (i in a)print i, a[i];}'

# find pods running on specific node 
> kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=ip-10-10-1-55.eu-central-1.compute.internal

# Find pods running wiuth specific status ( or not )
> kubectl get pods --all-namespaces -o wide --field-selector status.phase!=Running
0

AWS – Migrate VPN from a virtual private gateway to a transit gateway

When you migrate from setup where you have been using VPG ( Virtual Private Gateway ) to TG ( Transit Gateway ) it might be desirable not to reconfigure the VPN connection.

If you read through documentation it’s possible with just one caveat – tunnel will flip down/up during this process.

First find the details from your current VPN setup:

❯ aws ec2 describe-vpn-connections | jq '.VpnConnections[] | {VpnConnectionId, VpnGatewayId , TransitGatewayId}' 

The above should provide you with information about your VpnID and VpgID

{
  "VpnConnectionId": "vpn-1234567890",
  "VpnGatewayId": "vgw-123456",
  "TransitGatewayId": null
}

Having this info is now sufficient to execute command which will move your VPN connection to transit gateway ( you need to have that ID at hand )

 aws ec2 modify-vpn-connection --vpn-connection-id vpn-1234567890 -transit-gateway-id tgw-1234567890f 

Detailed information you can find in the article here https://aws.amazon.com/premiumsupport/knowledge-center/transit-gateway-migrate-vpn/

0

Kubernetes context per terminal session

If you are like me working with multiple kubernetes clusters it becomes really unhandy to work with them at the same time.

For a while I have been using ktx to manage that – however as mentioned above – at the moment of writing it sets context to be global.

So as a small workaround I have those 2 functions in my `.zshrc` file

k8ctx-switch() { 
# create a temp file for our config 
TEMP_CONFIG="$(mktemp "kubectx.$1")" 
kubectl config view --minify --flatten --context=$1 > $TEMP_CONFIG export KUBECONFIG="${TEMP_CONFIG}:${KUBECONFIG}" cat ${TEMP_CONFIG}
}

k8ctx-list() {
KUBECONFIG="${HOME}/.kube/config" kubectl config get-contexts
}

This is a MVP πŸ™‚ to get multiple tabs opened with different k8s clusters to manage. Pretty handy πŸ™‚

0

Terraform – iam policy for AWS user

Just a quick writeup when for example providing conditional access to s3 you would like to restrict access to AWS user name in the path you can refer to this quick snippet

   statement {
       actions = [
           "s3:ListBucket",
       ]
       resources = [
           "arn:aws:s3:::${var.s3_bucket_name}",
       ]
       condition {
           test = "StringLike"
           variable = "s3:prefix"
           values = [
               "",
               "home/",
               "home/&{aws:username}/",
           ]
       }
   },
   statement {
       actions = [
           "s3:*",
       ]
       resources = [
           "arn:aws:s3:::${var.s3_bucket_name}/home/&{aws:username}",
           "arn:aws:s3:::${var.s3_bucket_name}/home/&{aws:username}/*",
       ]
   }
0

Compiling NGINX with — with-http_auth_request_module on Centos 7

While looking at SSO solutions I decided to investigate a bit more options how I could use Nginx with solutions like Okta to protect my resources. One of interesting ones was using authentication proxy with Nginx.

The afore functionality is available through use of http_auth_request_module. However this module is not compiled by default. This got me the idea that would be nice to exercise going step by step through compiling Nginx with auth module Centos 7.

Yes – I do know that there are solutions on the market/internet which would save me from this – however I value the learning process in this challenge as well πŸ™‚ If you have interesting links to alternatives please leave them in the comment section.

Getting the sources

Our journey begins with getting the sources. I have tried following the official Nginx documentation but I find it …. somehow not up to the task. Hence there are some modifications or additions that I did to get this through πŸ™‚

mkdir nginx-from-source && cd $_

Once we have our new folder we can download the pre-reqs

Here we are taking Nginx version 1.19.0 – please be sure to check whats the latest version before running the command

   wget https://ftp.pcre.org/pub/pcre/pcre-8.44.tar.gz
   wget http://zlib.net/zlib-1.2.11.tar.gz
   wget http://www.openssl.org/source/openssl-1.1.1g.tar.gz
   wget https://nginx.org/download/nginx-1.19.0.tar.gz
   tar zxf nginx-1.19.0.tar.gz

Compile PCRE

tar -zxf pcre-8.44.tar.gz
cd pcre-8.44
./configure
make
sudo make install

Compile ZLIB

tar -zxf zlib-1.2.11.tar.gz
cd zlib-1.2.11
./configure
make
sudo make install

Compiling OpenSSL

OpenSSL deserves spot for bit more insights than just dry code. We will use never version than the one running on the box right now.

Pre-reqs

We will start off by installing required packages via yum and extracting the content of downloaded archive

yum group install 'Development Tools'
yum install perl-core zlib-devel -y
tar -xf openssl-1.1.1g.tar.gz 
cd openssl-1.1.1g

Configure & install OpenSSL

sudo ./config --prefix=/usr/local/ssl --openssldir=/usr/local/ssl shared zlib
sudo make
sudo make test
sudo make install

Configure shared libraries

Navigate to /etc/ld.so.conf.d and run the following

sudo echo "/usr/local/ssl/lib" >> /etc/ld.so.conf.d/openssl-1.1.1g.conf

ldconfig is used to create, update and remove symbolic links for the current shared libraries based on the lib directories present in the /etc/ld.so.conf

Reload with verbose

sudo ldconfig -v

Configure OpenSSL binary

Start with backing up the current OpenSSL

sudo mv /bin/openssl /bin/openssl.backup

Create script which will be executed on the system…

sudo vi /etc/profile.d/openssl.sh

… and set contents to

OPENSSL_PATH="/usr/local/ssl/bin"
export OPENSSL_PATH
PATH=$PATH:$OPENSSL_PATH
export PATH

Once done we need to make sure that the script is allowed to be executed

sudo chmod +x /etc/profile.d/openssl.sh

Next reload the profile to get the openSSL new binary with your PATH

source /etc/profile.d/openssl.sh

Verify openSSL version

 which openssl
 openssl version -a

If you reached this moment then we are ready to move on the next part πŸ™‚

Compiling Nginx with extra modules

Create user under which the process will be running

useradd -s/sbin/nologin -d/usr/local/nginx -M nginx

Navigate to folder with nginx sources created during download of our pre-reqs and run the config command

./configure 
--user=nginx 
--group=nginx 
--error-log-path=/var/log/nginx/error.log 
--http-log-path=/var/log/nginx/access.log 
--sbin-path=/usr/local/nginx/nginx 
--pid-path=/usr/local/nginx/nginx.pid 
--with-pcre=../pcre-8.44 
--with-zlib=../zlib-1.2.11 
--with-http_auth_request_module 
--with-http_geoip_module 
--with-http_gzip_static_module 
--with-http_gunzip_module 
--with-http_realip_module 
--with-http_secure_link_module 
--with-http_slice_module 
--with-http_ssl_module 
--with-http_v2_module

The above compiles Nginx with extra modules. For a comprehensive list with detailed information about each of the extra modules please refer to official Nginx documentation.

Once the above command finishes run

  make
  make install

When the above process finishes you should have nginx installed in /usr/local/nginx

Initial configuration of Nginx

In order to use Nginx we need to configure it. Right now our system knows nothing about running it.

Run vi /etc/systemd/system/nginx.service and set the content to

[Unit]
Description=The NGINX HTTP and reverse proxy server
After=syslog.target network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target

[Service]
Type=forking
PIDFile=/usr/local/nginx/nginx.pid
ExecStartPre=/usr/local/nginx/nginx -t
ExecStart=/usr/local/nginx/nginx
ExecReload=/usr/local/nginx/nginx -s reload
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Now when you run systemctl status nginx you should see our service available.

Go ahead and run it! Type systemctl start nginx

At this moment you should have Nginx running with extra modules compiled!

1

Kubernetes kind with Traefik

If you have been working with kubernetes you already know that sometimes you just need a quick way to spin a cluster.

Well if it is so you must have came across kind . It allows you to spin k8s cluster locally.

kind is a tool for running local Kubernetes clusters using Docker container β€œnodes”.

With just one line you can get and spin up your cluster πŸ™‚


Once thats up and running I was interested to have some more fun with Traefik however on the “kind” documentation you could see only subset was mentioned as supported ones ….

So this felt like a good sunday challenge πŸ™‚ since I really had some plans to explore Traefik and its potential more.

I started off by finding the details of creating a new cluster with exposed mapped ports

cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 30080
    hostPort: 88
    protocol: TCP
  - containerPort: 30443
    hostPort: 444
    protocol: TCP
- role: worker
- role: worker
- role: worker
EOF

After the above it will create you 4 “nodes” from which one ( in my case control plane node ) will have the mapping and the specific node label ( the node label you can change to your needs )

Once your cluster is ready you can apply the bundle from the code below.It does not have host defined for ingress route or the middleware for dashboard but it is something you can easily add with CRDs

# Traefik bundle
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutes.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRoute
    plural: ingressroutes
    singular: ingressroute
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: middlewares.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: Middleware
    plural: middlewares
    singular: middleware
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutetcps.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteTCP
    plural: ingressroutetcps
    singular: ingressroutetcp
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressrouteudps.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteUDP
    plural: ingressrouteudps
    singular: ingressrouteudp
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsoptions.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSOption
    plural: tlsoptions
    singular: tlsoption
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsstores.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSStore
    plural: tlsstores
    singular: tlsstore
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: traefikservices.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TraefikService
    plural: traefikservices
    singular: traefikservice
  scope: Namespaced
---
apiVersion: v1
kind: Namespace
metadata:
  name:  traefik
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - traefik.containo.us
    resources:
      - middlewares
      - ingressroutes
      - traefikservices
      - ingressroutetcps
      - ingressrouteudps
      - tlsoptions
      - tlsstores
    verbs:
      - get
      - list
      - watch
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name:  traefik
  namespace: traefik
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik
subjects:
  - kind: ServiceAccount
    name: traefik
    namespace: traefik
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: traefik
  namespace: traefik
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app.kubernetes.io/name: traefik
  template:
    metadata:
      labels:
        app.kubernetes.io/name: traefik
    spec:
      containers:
        - args:
            - --global.checknewversion
            - --global.sendanonymoususage
            - --entryPoints.traefik.address=:9000/tcp
            - --entryPoints.web.address=:9080/tcp
            - --entryPoints.websecure.address=:9443/tcp
            - --api.dashboard=true
            - --ping=true
            - --providers.kubernetescrd
            - --providers.kubernetesingress
            - --log.level=DEBUG
          image: traefik:2.2.11
          imagePullPolicy: IfNotPresent
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /ping
              port: 9000
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 2
          name: traefik
          ports:
          - containerPort: 9000
            name: traefik
            protocol: TCP
          - containerPort: 9080
            name: web
            protocol: TCP
          - containerPort: 9443
            name: websecure
            protocol: TCP
          readinessProbe:
            failureThreshold: 1
            httpGet:
              path: /ping
              port: 9000
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 2
          resources: {}
          securityContext:
            capabilities:
              drop:
              - ALL
            readOnlyRootFilesystem: true
            runAsGroup: 65532
            runAsNonRoot: true
            runAsUser: 65532
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      tolerations:
        - key: "node-role.kubernetes.io/master"
          effect: "NoSchedule"
          operator: "Exists"
      nodeSelector:
        ingress-ready: "true"
      restartPolicy: Always
      securityContext:
        fsGroup: 65532
      serviceAccountName: traefik
---
apiVersion: v1
kind: Service
metadata:
  name: traefik
  namespace: traefik
spec:
  type: NodePort
  selector:
    app.kubernetes.io/name: traefik
  ports:
    - name: web
      port: 9080
      nodePort: 30080
    - name: websecure
      port: 9443
      nodePort: 30443
---
apiVersion: v1
kind: Service
metadata:
  name: traefik-dashboard
  namespace: traefik
  labels:
    app.kubernetes.io/name: traefik
spec:
  type: ClusterIP
  ports:
  - port: 9000
    name: traefik
  selector:
     app.kubernetes.io/name: traefik
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard
spec:
  routes:
  - match: (PathPrefix(`/api`) || PathPrefix(`/dashboard`))
    kind: Rule
    services:
    - name: [email protected]
      kind: TraefikService
    # middlewares:
    #   - name: auth
# ---
# apiVersion: traefik.containo.us/v1alpha1
# kind: Middleware
# metadata:
#   name: auth
# spec:
#   basicAuth:
#     secret: secretName # Kubernetes secret named "secretName"

Once the installation is completed you can access the dashboard. Of course that dashboard is exposed via IngressRoute so the fun can begin πŸ™‚

Hope this will enable you to quickly discover Traefik/Kubernetes/Kind πŸ™‚

Happy coding!

0

Golang – GoEnv with versions 1.11+

If you have wondered why does you goenv is stuck on version 1.11.X then you probably missed on one of the open issues ( which btw is quite annoying to find )

brew install jq
if [ -n "$(brew info --json goenv | jq -r '.[0].installed[] | select(.version | test("^HEAD-") | not)')" ]; then
  brew uninstall goenv
fi
brew install --HEAD goenv

Once you execute the code above – you will be able to use up to date versions of go πŸ™‚

0

golang modules – accessing local packages

Recently I was developing locally framework which I just wanted to test before pushing it anywhere to the clouds πŸ™‚ It just so happens that testing when using modules is quite easy. The only thing you need to do is to update your go.mod file like that

module test-package

require github.com/abc/xyz v1.0.2
replace github.com/abc/xyz v1.0.2 => /Users/rafpe/xyz

From that point onward you can debug/make changes and try it all out before release.

Happy coding!

0

openSSL pb7 certificate : unable to load certificate

Recently when working with certificates I received them in pb7 format. If you just try to take them as is u might get

unable to load certificate
140735207381436:error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1319:
140735207381436:error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:381:Type=X509_CINF
140735207381436:error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:751:Field=cert_info, Type=X509
140735207381436:error:0906700D:PEM routines:PEM_ASN1_read_bio:ASN1 lib:pem_oth.c:83:

When trying to validate a certificate using openssl, this is because it is in the wrong format, whilst the certificate file visually appears to be in x.509 format, you will find it contains a far longer base64 string than x.509 certificates of the same bit length.
The format in this case is p7b (PCKS #7); to use the certificate with apache you’re going to have to convert this.

openssl pkcs7 -print_certs -in certificate.p7b -out certificate.cer

Within the resulting .cer file you will file you x.509 certificate bundled with relevant CA certificates, break these out into your relevant .crt and ca.crt files and load as normal into apache.