0

AWS – EKS and “insufficient pods”

So the day has come when I noticed that one of my pods was not running and I received the above mentioned message insufficient pods”.

What I then realised was that I run out of maximal number of pods I can run :O which in AWS EKS is associated with ENI value.

To get the number of maximal pods you can run execute the following:

❯ kubectl get node -o yaml | grep pods
      pods: "17" => this is allocatable pods that can be allocated in node
      pods: "17" => this is how many running pods you have created

The details of number of pods per instance can be found via https://github.com/awslabs/amazon-eks-ami/blob/master/files/eni-max-pods.txt

In kubernetes v1.19 we will get GA of EvenPodsSpread which will definitely help in managing how pods are distributed

Also In my troubleshooting I found helpful to use some of the below scripts.

# find number of pods running per node 
❯ kubectl get pod --all-namespaces -o json | jq -r '.items[] |select( .kind=="Pod")| "\(.status.hostIP),\(.metadata.name)"'| awk -F, '{a[$1]++;}END{for (i in a)print i, a[i];}'

# find pods running on specific node 
> kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=ip-10-10-1-55.eu-central-1.compute.internal

# Find pods running wiuth specific status ( or not )
> kubectl get pods --all-namespaces -o wide --field-selector status.phase!=Running
0

AWS – Migrate VPN from a virtual private gateway to a transit gateway

When you migrate from setup where you have been using VPG ( Virtual Private Gateway ) to TG ( Transit Gateway ) it might be desirable not to reconfigure the VPN connection.

If you read through documentation it’s possible with just one caveat – tunnel will flip down/up during this process.

First find the details from your current VPN setup:

❯ aws ec2 describe-vpn-connections | jq '.VpnConnections[] | {VpnConnectionId, VpnGatewayId , TransitGatewayId}' 

The above should provide you with information about your VpnID and VpgID

{
  "VpnConnectionId": "vpn-1234567890",
  "VpnGatewayId": "vgw-123456",
  "TransitGatewayId": null
}

Having this info is now sufficient to execute command which will move your VPN connection to transit gateway ( you need to have that ID at hand )

 aws ec2 modify-vpn-connection --vpn-connection-id vpn-1234567890 -transit-gateway-id tgw-1234567890f 

Detailed information you can find in the article here https://aws.amazon.com/premiumsupport/knowledge-center/transit-gateway-migrate-vpn/

0

Terraform – iam policy for AWS user

Just a quick writeup when for example providing conditional access to s3 you would like to restrict access to AWS user name in the path you can refer to this quick snippet

   statement {
       actions = [
           "s3:ListBucket",
       ]
       resources = [
           "arn:aws:s3:::${var.s3_bucket_name}",
       ]
       condition {
           test = "StringLike"
           variable = "s3:prefix"
           values = [
               "",
               "home/",
               "home/&{aws:username}/",
           ]
       }
   },
   statement {
       actions = [
           "s3:*",
       ]
       resources = [
           "arn:aws:s3:::${var.s3_bucket_name}/home/&{aws:username}",
           "arn:aws:s3:::${var.s3_bucket_name}/home/&{aws:username}/*",
       ]
   }
2

Serverless Okta JWT as AWS API Gateway Authorizer

About this solution

In todays technological world it has become very popular ( and quite easy )  to create serverless architectures with Lambdas and expose them via API gateway.

The expose part is something which we could protect better. Solution provided here is basic blueprint which leverages openID  ( in this case set up in Okta ).

 

This software/code is provided to you “as-is” and without warranty of any kind, express, implied or otherwise, including without limitation, any warranty of fitness for a particular purpose

 

Tech involved

Although I would like to keep the solution requirements to minimal this requires some software/services to work nicely together. Below I highlighted what we will need:

  • Serverless – to create and configure the stack
  • AWS – to consume provided services
  • Okta –  to provide authentication

Setting up the solution

Setup of the whole solution involves several steps. I have outlined them below.

Create Okta openID application

  • Login to your Okta organisation and navigate to Admin part
  • Go to application
  • Create new native application with `openID` ( this is the only option atm)
  • For the name we have used `OpenID – myAppName`
  • For the redirect URL you can paste `api://myAppName` ( or anything else if you plan to use different kind of flows )
  • Save the app and proceed to editing.
  • In general settings you need to modify *Allowed grant types* and enable `Resource Owner Password`
  • In client credentials enable `Use Client Authentication` ( make note of Client Id and Client Secret)
  • Assign the application to engineers which should have access to REST API

As a nice feature you could use claims to determine who can execute READ/WRITE actions

 

Verify token generation

In order to verify that you can get tokens from the app you have just created you need to call one of Okta endpoints.

  • Create basic authentication credentials consisting of the following client_id:client_secret
    You can use the following snippet

    echo "client_id:client_secret" | base64
  • With the result of that command you will be able to make calls using Authorization: Basic <result-of-command>
  • Make http call to token endpoint , making sure to replace your-okta-tenant-name , username and password
    curl -X POST \
    https://xxx.okta-emea.com/oauth2/default/v1/token \
    -H 'Authorization: Basic MG9hMmQzN.........Q==' \
    -H 'Cache-Control: no-cache' \
    -H 'Content-Type: application/x-www-form-urlencoded' \
    -d 'username=username&password=Password1&grant_type=password&scope=openid'

     

  • In response you should receive token in the following format
    {
    "access_token": "eyJraW.....BvXdkU2Gg",
    "token_type": "Bearer",
    "expires_in": 3600,
    "scope": "openid",
    "id_token": "eyJr....yg"
    }

 

Obtain a public key from Okta

At this point of time you have fully working Okta openID app and can obtain tokens. Our next task is to obtain public key which will allow us in later stage to verify token signature.

There are many ways to obtain the key – and each of them can either take more time or involves you providing more information. My idea was simple – automate it as much as possible… therefore I came up with go-jwk-pem (available in github ) . It is a simple CLI tool which takes either token or Okta server URL and retrieves public key which have been used to sign the JWT.

In this instance I will just use token from previous step

go-jwk-pem from-token --token eyJraW.....BvXdkU2Gg | /usr/bin/env ruby -e 'p ARGF.read'

Result of this command is single line public key , which is last piece of our puzzle which we need to make our solution working.

"-----BEGIN RSA PUBLIC KEY-----\nMIIBIjA........A4\nzTsuZ+eQLfhNbuA.....wWtcDsd+vMUlS7iJow\n2QIDAQAB\n-----END RSA PUBLIC KEY-----\n\n"

 

Creating AWS stack using Serverless

The time has come when we begin real fun 🙂 Let’s begin by cloning the solution from Github

One thats done we need to modify value in serverless.env.yml

dev:
  OKTA_PUBLIC_KEY: <PASTE-YOUR-PEM-HERE>

Since this functions are written in go we need to build them before deploying

> [SHELL]  RafPe $ make build
env GOOS=linux go build -ldflags="-s -w" -o bin/func1 func1/main.go
env GOOS=linux go build -ldflags="-s -w" -o bin/auth auth/main.go

And now let’s deploy by using one of profile for AWS ( from credentials file ) running simple command

> [SHELL]  RafPe $ sls deploy -s dev --aws-profile myAwsProfile --verbose

Output shows us details about our functions deployes ( this is minimal blueprint so your output can have more )

Serverless: Stack update finished...
Service Information
service: test-auth
stage: dev
region: eu-west-1
stack: test-auth-dev
api keys:
  None
endpoints:
  ANY - https://reiw2emcp3.execute-api.eu-west-1.amazonaws.com/dev/hello
functions:
  func1: test-auth-dev-func1
  auth: test-auth-dev-auth
layers:
  None

Stack Outputs
AuthLambdaFunctionQualifiedArn: arn:aws:lambda:eu-west-1:123:function:test-auth-dev-auth:3
Func1LambdaFunctionQualifiedArn: arn:aws:lambda:eu-west-1:123:function:test-auth-dev-func1:3
ServiceEndpoint: https://reiw2emcp3.execute-api.eu-west-1.amazonaws.com/dev
ServerlessDeploymentBucketName: test-auth-dev-serverlessdeploymentbucket-2g5ap50n5lwn

 

Testing the solution

So right now we can immediately check if our setup works. Let’s start by trying to make simple http call

> [SHELL]  RafPe $ http https://reiw2emcp3.execute-api.eu-west-1.amazonaws.com/dev/hello
HTTP/1.1 401 Unauthorized
Connection: keep-alive
Content-Length: 26
Content-Type: application/json
Date: Sat, 15 Dec 2018 11:01:24 GMT
Via: 1.1 bce55e537f8dfcf0127f649d11fd1821.cloudfront.net (CloudFront)
{
    "message": "Unauthorized"
}

As expected we get Unauthorized message 😉 Let’s try to add token generated by Okta and make the call again

> [SHELL]  RafPe $ http https://reiw2emcp3.execute-api.eu-west-1.amazonaws.com/dev/hello Authorization:'Bearer eyJraWQ....QHMi5ISw'
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 69
Content-Type: application/json
Date: Sat, 15 Dec 2018 11:10:04 GMT
Via: 1.1 94d63cbf92082237b86267ffd4cacc64.cloudfront.net (CloudFront)
X-Cache: Miss from cloudfront
X-MyCompany-Func-Reply: world-handler
{
    "message": "Okay so your other function also executed successfully!"
}

and voilla 😉 we have just created custom authorizer validating our Okta JWT.

 

Summary

Although this is just a blueprint it can be nicely extended. I would like to point out several items you might be interested about this

  • Solution can be nicely extended to use claims to provide appropriate access – I find it really nice
  • You are not limited to use goLang because of one function being written in go. Serverless support multiple frameworks – just need to define those on function level then ( and define what packages you are including )

 

if you have any feedback – please leave comment or add your code into github repo 😉

 

 

 

0

Golang app to authenticate with AWS Cognito Pool

Since started to work with AWS I sometimes hit the same problems more than one time 😉 One of those happen was when working with AWS Cognito – just needed to authenticate and get token – or just verify the user 😉 using command line.  I honestly did not want to be bothered with any complexity to get simple tokens which I planned to use in accessing other systems etc.

For this purposes I have created simple CLI ( right now with just 2 methods ) to help me out in those situations.  Usage is extremely simple you just need to have your AWS profile configured and have details of your AppClient from your user pool.

Authenticate

> [SHELL] RafPe $ go-cognito-authy --profile cloudy --region eu-central-1 auth --username rafpe --password 'Password.0ne!'  --clientID 2jxxxiuui123
{
AuthenticationResult: {
    AccessToken: "eyJraWQiOiJ0QXVBNmxtNngrYkxoSmZ",
    ExpiresIn: 3600,
    IdToken: "eyJraWQiOiJ0bHF2UElTV0pn",
    RefreshToken: "eyJjdHkiOiJKV1QiLCJlbmMiOiJBMjU2R-TpkR_uompG7fyajYeFvn-rJVC_tDO4pB3",
    TokenType: "Bearer"
},
ChallengeParameters: {}
}

which in return should give you response with tokens needed further in your adventures with AWS…. but if you would have user in state that a password needs to be changed 😉 ….

> [INSERT] RafPe $ go-cognito-authy --profile cloudy --region eu-central-1 auth --username rafpe --password 'Password.0ne!'  --clientID 2jxxxiuui123
{
ChallengeName: "NEW_PASSWORD_REQUIRED",
ChallengeParameters: {
    requiredAttributes: "[]",
    userAttributes: "{\"email_verified\":\"true\",\"email\":\"[email protected]\"}",
    USER_ID_FOR_SRP: "rafpe"
},
Session: "bCqSkLeoJR_ys...."
}

 

Administratively set new pass

With the session above and known challenge for new pass you can use it to set desired password

> [INSERT] RafPe $ go-cognito-authy --profile cloudy -region eu-central-1 admin reset-pass --username rafpe --pass-new 'Password.0ne2!' --clientID 2jxxxiuui123 --userPoolID  eu-central-1_CWNnTiR0j --session "bCqSkLeoJR_ys...."

and voilla 😉 we can now continue playing with tokens

 

 

Patches welcome

The whole solution is available on Github https://github.com/RafPe/go-cognito-authy/tree/master  and if you are missing something please create a PR 😉

0

Serverless – own plugin to attach AWS API Gateway Basic Request Validation

Hey,

I have not had so much time to do as many posts as I wanted since I started my recent project of my own app on AWS using serverless framework. As every good engineer while making my solution I looked at many open source options and serverless seemed to be really good … until I wanted to do something which no one seems to have been doing before ( every one of us know it right ? ) …

So what was that special thing ? Well nothing fancy – just wanted to “attach AWS API Gateway Basic Request Validation” … So I thought … how hard can it be 🙂

As everyone I used “google” to tell me who has done something like that before …. and thats how I visited issue related directly to my problem => https://github.com/serverless/serverless/issues/3464 

Had a conversation there with all interested and we all seemed to agree that there is nothing that would work ( at that specific moment 🙂 )

Well by default ( at the moment of writing of this article ) serverless does not support this “out of the box” therefore this has kicked my off to create my own plugin. Since being developer is not my primary focus ( as I do it only as hobbyist 🙂 ) it was a bit of challenge. But it was completed with success. So what was needed to happen to create it ( if you want to skip the story just scroll down 🙂 ) ?

I started by looking into creating a form of “hello world” plugin that would help me to understand how to approach this in best way. Resources for this can be found on the official site of serverless

But I had a feeling this was quite incomplete to create fully functional plugin. So I spend quite a while browsing internet and reading how people created their own plugins from really simple ones – into more advanced which rocks&roll 😉  Here I think the resource that will show you more detailed steps can be found here

With that bits of knowledge I went to the official repo page of plugins and browsed through repositories which were there. This gave me better idea about what I needed to use.

Having all that knowledge I compiled my action plan which basically was:

  • Create plugin core structure
  • Add resource to all functions of CloudFormation template based on serverless.yml
  • Merge template on hook being called before:deploy
  • Publish for people & get a beer 🙂

This is how I managed to create my plugin available in github https://github.com/RafPe/serverless-reqvalidator-plugin and via npm:

npm install serverless-reqvalidator-plugin

All it does require is extremely simple we start off by creating custom resource in serverless.yml

    xMyRequestValidator:  
      Type: "AWS::ApiGateway::RequestValidator"
      Properties:
        Name: 'my-req-validator'
        RestApiId: 
          Ref: ApiGatewayRestApi
        ValidateRequestBody: true
        ValidateRequestParameters: false

With that one done we add plugin to be enabled

plugins:
    - serverless-reqvalidator-plugin

And then in our functions we specify validator

  debug:
    handler: apis/admin/debug/debug.debug
    timeout: 10
    events:
      - http:
          path: admin/debug
          method: get
          cors: true
          private: true 
          reqValidatorName: 'xMyRequestValidator'

 

 

 

 

Voilla 😉 And the code in the plugin that does the magic ? You would not believe how much of code that is ….

resources[methodName].Properties.RequestValidatorId = {"Ref": `${event.http.reqValidatorName}`};

 

 

And thats it folks 😉 Enjoy and happy coding 🙂

 

0

AWS – Writing better code in Node.js for Lambda functions

Hey,

Been a while but finally its time to post some of technical information I accumulated since my last post. Today we will focus on improvements in code from the time you released something aka version 1 into a code base which you can run tests against.

 

The past …

In my last post I described some modular code I started to work on for use with Lambda ( available here ) The code back then contained a lot of redundant blocks ( which I was aware of 😉  ) however it enabled me to focus on making it better. Just to get everyone idea how it did look like here is snippet from initial code commit

....  // code removed for demo purposes // .... 

var timestamp = new Date().getTime();
  const uniqueId  = uuid.v1();

  console.log(`[CreatePlatformEndpoint] [${timestamp}] [${uniqueId}][Info] Starting execution`);
  
      var responseCode = 400;
      var responseBody = "";
  
      var response = {
        statusCode: responseCode,
        body:       responseBody
      };

      if ( !isDef(event.body) )
        { 

          console.log(`[CreatePlatformEndpoint] [${timestamp}] [${uniqueId}][Error] Missing body information (EC.001)`);

          let errorData = {
            code: "EC.001",
            data: {
              message: "Missing body"
            }
          }

          response.body = {
            action:  "CreatePlatformEndpoint",
            status:  "error",
            error:   errorData,
          }

          response.body = JSON.stringify(response.body)

          callback(null,response); 

        }

      var jsonBody = JSON.parse(event.body);

      console.log(`[CreatePlatformEndpoint] [${timestamp}] [${uniqueId}][Info] Parsed body from request`);

      if ( !isDef(jsonBody.deviceToken) || !isDef(jsonBody.platformApplicationArn))
        {

          console.log(`[CreatePlatformEndpoint] [${timestamp}] [${uniqueId}][Error] Missing required parameters in body (EC.002)`);
          
          let errorData = {
            code: "EC.002",
            data: {
              message: "Missing required parameters in body"
            }
          }

          response.body = {
            action:  "CreatePlatformEndpoint",
            status:  "error",
            error:   errorData,
          }

          response.body = JSON.stringify(response.body)

          callback(null,response);

And now this is just only portion of  code which been redundant. It was just in single file. Now imagine we have multiple components with multiple files and we need to make a single change into logic which is repeated across all those ?! Madness :/

Therefore it took some time ( as I’m far away from being a js developer 😉 ) but I changed …

Making it better …

by creating classes and also in this way trying to regain control on controlling the lifetime of object’ instances. This also enabled me to start writing tests for my code which I’m really happy about as it help so much in test driven development ( this was inspired by the following article )

  • Write your business logic so that it is separate from your FaaS provider (e.g., AWS Lambda), to keep it provider-independent, reusable and more easily testable.

  • When your business logic is written separately from the FaaS provider, you can write traditional Unit Tests to ensure it is working properly.

  • Write Integration Tests to verify integrations with other services are working correctly.

 

So how does the new code looks like ? Take a sneak peak on  snippet from shared resource

const uuid = require('uuid');

class xSharedFunctions { 
    
    constructor(component,callback,disableLogging){
        
                this.callback       =  callback;
                this.component      =  (component === null || component === undefined ) ? 'undefined' : component ;
                this.disableLogging =  disableLogging
                
            }

    generateSuccessResponse(dataSuc,respCode){
            let that = this;

            var responseCode = (respCode === null || respCode === undefined  ) ? 200:respCode ;
            var responseBody = "";
        
            var response = {
              statusCode: responseCode,
              headers: {
                "Access-Control-Allow-Origin" : "*",      // Required for CORS support to work
                "Access-Control-Allow-Credentials" : true // Required for cookies, authorization headers with HTTPS
              },
              body:       responseBody
            };
            
            response.body = {
                component: that.component,
                status:  "success",
                data: dataSuc
            };
            
            response.body = JSON.stringify(response.body);
        
            return response;
    }

// REST OF CODE COMES HERE ....

 

Once that is in place we can go ahead and try to …

 

… use the code in our modules/applications

To get all required references we required our resources and use their functions

'use strict';

var xSharedFunctions = require('../xRes/shared/xSharedFunctions');
var xSnsEndpointManager = require('../xRes/xSnsEndpointManager');


const uuid      = require('uuid');
const component  = 'sns'

var xSharedFnc = new xSharedFunctions('sns');

module.exports.create = (event, context, callback) => {
  const uniqueId      = uuid.v1();
  var xSnsEndpointMgr = new xSnsEndpointManager(uniqueId,callback);

  xSharedFnc.logmsg(uniqueId,'info','Starting execution');
  xSharedFnc.logmsg(uniqueId,'info',`${JSON.stringify(event)}`);

// REST OF CODE COMES HERE ...

 

with keeping the above in mind we should not forget to …

 

… test our code 🙂

And that is why for example I got rests which looks like the following now ( using Mocha and Babel ) …

// CODE REMOVED FOR VISIBILITY ... 


describe('xSnsEndpointManager', function() {
    
        describe('#createPlatformEndpoint()', function() {
    
                before(function () {
                    


                    AWS.mock('SNS', 'createPlatformEndpoint', function (params, callback) {
                    callback(null, '{"ResponseMetadata":{"RequestId":"efdb1199-f10e-5b0b-bff9-43addbda438b"},"EndpointArn":"arn:aws:sns:eu-west-1:12345:endpoint/APNS_SANDBOX/blah-app/c08d3ccd-3e07-328c-a77d-20b2a790122f"}')
                    })

                })


                it('should create endpoint if token provided', function(){

                    var xSnsEndpointMgr = new xSnsEndpointManager('1234',function(dummy,responseCallback){
                        expect(responseCallback.statusCode).to.equal(201);

                        let result = JSON.parse(responseCallback.body);
                        expect(result.component).to.equal('sns');
                        expect(result.status).to.equal('success');

                        let resultData = JSON.parse(result.data);
                        expect(resultData.EndpointArn).not.to.equal(null);
                       
                    },true);

                    let res = xSnsEndpointMgr.createPlatformEndpoint('eee','eee');
                    
                });

                after(function () {
                    AWS.restore('SNS', 'createPlatformEndpoint')
                  })

          });
    
});

// CODE REMOVED FOR VISIBILITY ...

 

Closing thoughts …

So as you can see it all starts to look nice and definitely will get you further if you implement tests. For those interested to see how do I do things here are the links to my repositores on git

 

I hope someone would be able to reuse something for their own needs 😉 Happy coding!

 

1

Serverless REST api for Amazon SNS

Hi!

So it has been a while since I posted some technical posts. Now … this is something that touches us all – the lack of time in the jungle of ongoing projects 🙂 However today we will look into something which I find quite nice for developing of new applications.

Solution is based on serverless framework. Now before we go on – we all know that serverless is a nice catchy word for ‘someone’ else computer and operation problem :)’ . But idea is simple – I’m using AWS – and there it spins me up lambda functions with associated API gateway.

I decided to create this solution to have unified way of deploying and interacting with AWS services in a way that would be easiest for me to consume. However for someone else it might not be the best. Also to be on safe side – this code is really version v1.0.0 so it will get updates as we go ( PR always welcome )

The repository for this write up is available under https://github.com/RafPe/serverless-api-sns

Solution folder structure

The solution structure is created as follows

total 32
-rw-r--r--    1 rafpe  450652656   1.0K Sep  9 17:49 LICENSE
-rw-r--r--    1 rafpe  450652656   2.2K Sep 10 15:07 README.md
drwxr-xr-x  234 rafpe  450652656   7.8K Sep  5 23:53 node_modules
-rw-r--r--    1 rafpe  450652656   255B Sep 10 14:19 package.json
-rw-r--r--    1 rafpe  450652656   3.6K Sep 10 14:21 serverless.yml
drwxr-xr-x    7 rafpe  450652656   238B Sep 10 14:52 sns

and the SNS folder:

├── attributes
├── endpoint
│   ├── create.js
│   ├── delete.js
│   └── list.js
├── messages
├── models
│   └── endpoint.create.json
└── topics

Code

Now since this is not a coding school and I have used really simple code I will not be going into details there. I just might say code has some portions which are repeated and could be wrapped into common methods 😉 did not have time to take a look into that one yet.

For the rest it is using standard aws libraries to execute most of the actions

 

Serverless.yml

Is the heart of your deployment. It describes what will be created and how to link those things together. For more advanced examples you should check out docs.serverless.com

# Welcome to Serverless!
#
# This file is the main config file for your service.
# It's very minimal at this point and uses default values.
# You can always add more config options for more control.
# We've included some commented out config examples here.
# Just uncomment any of them to get that config option.
#
# For full config options, check the docs:
#    docs.serverless.com
#
# Happy Coding!

service: api

# You can pin your service to only deploy with a specific Serverless version
# Check out our docs for more details
# frameworkVersion: "=X.X.X"

provider:
  name: aws
  role: xmyCustRole1 
  apiKeys:
    - myApiKey  
  runtime: nodejs6.10
  region: eu-west-1  


stage: dev



functions:
  create:
    handler: sns/endpoint/create.create
    events:
      - http:
          path: endpoint/create
          method: post
          cors: true
          private: true

  delete:
    handler: sns/endpoint/delete.delete
    events:
      - http:
          path: endpoint/delete
          method: delete
          cors: true
          private: true


  list:
    handler: sns/endpoint/list.list
    events:
      - http:
          path: endpoint/list
          method: post
          cors: true
          private: true


resources:
  Resources:
    # PetsModelNoFlatten: 
    #   Type: "AWS::ApiGateway::Model"
    #   Properties: 
    #     RestApiId: {Ref: ApiGatewayRestApi}
    #     ContentType: "application/json"
    #     Description: "Schema for Pets example"
    #     Name: "PetsModelNoFlatten"
    #     Schema: 
    #       Fn::Join: 
    #         - ""
    #         - 
    #           - "{"
    #           - "   \"$schema\": \"http://json-schema.org/draft-04/schema#\","
    #           - "   \"title\": \"PetsModelNoFlatten\","
    #           - "   \"type\": \"array\","
    #           - "   \"items\": {"
    #           - "       \"type\": \"object\","
    #           - "       \"properties\": {"
    #           - "           \"number\": { \"type\": \"integer\" },"
    #           - "           \"class\": { \"type\": \"string\" },"
    #           - "           \"salesPrice\": { \"type\": \"number\" }"
    #           - "       }"
    #           - "   }"
    #           - "}"
    xmyCustRole1:
      Type: AWS::IAM::Role
      Properties:
        Path: /my/cust/path/
        RoleName: xmyCustRole1
        AssumeRolePolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Principal:
                Service:
                  - lambda.amazonaws.com
              Action: sts:AssumeRole
        Policies:
          - PolicyName: myPolicyName
            PolicyDocument:
              Version: '2012-10-17'
              Statement:
                - Effect: Allow # note that these rights are given in the default policy and are required if you want logs out of your lambda(s)
                  Action:
                    - logs:CreateLogGroup
                    - logs:CreateLogStream
                    - logs:PutLogEvents
                  Resource: arn:aws:logs:*:*:log-group:/aws/lambda/*:*:*
                - Effect: Allow # note that these rights are given in the default policy and are required if you want logs out of your lambda(s)
                  Action:
                    - sns:CreatePlatformEndpoint
                  Resource: arn:aws:sns:*:*:*           
                - Effect: "Allow"
                  Action:
                    - "s3:PutObject"
                  Resource:
                    Fn::Join:
                      - ""
                      - - "arn:aws:s3:::"
                        - "Ref" : "ServerlessDeploymentBucket"

                        

 

IAM policy

To make this all a bit more secure I defined here my specific IAM Role with custom permissions for actions – So if you would need to extend permisions required you would need to look into that resources as well

 

Validations

In my code you will find that I validate if specific parameters are received from the request. Now this is again something that

  1. Could be done better by taking this logic out into common functions or …
  2. even better to use the API gateway validators

I therefore went ahead and created my self json schema using the following online schema generator. With that one done I had to ‘escape’ those characters and then create a policy using serverless resource

resources:
  Resources:
    PetsModelNoFlatten: 
      Type: "AWS::ApiGateway::Model"
      Properties: 
        RestApiId: {Ref: ApiGatewayRestApi}
        ContentType: "application/json"
        Description: "Schema for Pets example"
        Name: "PetsModelNoFlatten"
        Schema: 
          Fn::Join: 
            - ""
            - 
              - "{"
              - "   \"$schema\": \"http://json-schema.org/draft-04/schema#\","
              - "   \"title\": \"PetsModelNoFlatten\","
              - "   \"type\": \"array\","
              - "   \"items\": {"
              - "       \"type\": \"object\","
              - "       \"properties\": {"
              - "           \"number\": { \"type\": \"integer\" },"
              - "           \"class\": { \"type\": \"string\" },"
              - "           \"salesPrice\": { \"type\": \"number\" }"
              - "       }"
              - "   }"
              - "}"

This is all nice but the problem I experience now is that I cannot in programatic way find out how to apply required body validators to specific methods using serverless. Maybe something I will find out later.

 

Deploying

Deploying is easy as running

serverless deploy

and the output should look like

Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (37.84 KB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
..........................
Serverless: Stack update finished...
Service Information
service: api
stage: dev
region: eu-west-1
api keys:
  myApiKey: ID0d9P4Vgi82l2YvndLwi81FA63lCup1adNQX7eD
endpoints:
  POST - https://isr61ohvhl.execute-api.eu-west-1.amazonaws.com/dev/endpoint/create
  DELETE - https://isr61ohvhl.execute-api.eu-west-1.amazonaws.com/dev/endpoint/delete
  POST - https://isr61ohvhl.execute-api.eu-west-1.amazonaws.com/dev/endpoint/list
functions:
  create: api-dev-create
  delete: api-dev-delete
  list: api-dev-list

 

Fun

Now this is the part I like the most 🙂 Fun starts here when you play around with the working solution. If you got any feedback I would be more than happy to hear about it.

 

 

 

 

 

0

AWS – API Gateway returning 502 from Lambda proxy

Hey,

If you have been scratching your head why does API Gateway returns 502 and within your code there are no exceptions ?

Does your API gateway response contain something like below ?

 

Then make sure that you are returning correct response object containing body and status code i.e.

      var response = {
        statusCode: 200,
        body:       '\0/'
      };

If you still see problem then consider if you are returning complex objects in your body ? If so the following should be additionally applied before returning

response.body = JSON.stringify(response.body)

And thats it 🙂 Solved the problem for me

 

 

19

.Net core JWT authentication using AWS Cognito User Pool

While working with .net core I needed to create API. For this what I aimed to have was proper authentication. Therefore I decided to use JSON Web Token (JWT) authentication.

However I wanted to avoid creating any of this logic by myself or spending too much time on it. That’s why I decided to use AWS Cognito User Pools to provide me with user management and to generate JWT I need.

It took me some time to gather information how to wire it all together so I will try to outline the most important.

AWS setup

  1. Create user pool in AWS Cognito
  2. Get the newly created user pool ID and run the following command
    curl https://cognito-idp.<region>.amazonaws.com/<user-pool-id>/.well-known/jwks.json > result.json

    * if you want to you can also just navigate to the URL (https://cognito-idp.<region>.amazonaws.com/<user-pool-id>/.well-known/jwks.json ) . Just replace region and user pool ID with correct information.

  3. The information you receive will be used by us to validate the tokens given by AWS.
  4. Save the results for later use.

.Net core API project

  1. Create new .net core webapi project
    dotnet new webapi
  2. Install additional packages
    dotnet add package Microsoft.AspNetCore.Authentication.JwtBearer
    dotnet add package Microsoft.IdentityModel.Tokens
    dotnet add package Microsoft.AspNetCore.Identity

     

  3. add JWT authentication policy ( we will decorate our controllers with it )
                services.AddAuthorization(auth =>
                {
                    auth.AddPolicy("Bearer", new AuthorizationPolicyBuilder()   
                        .AddAuthenticationSchemes(JwtBearerDefaults.AuthenticationScheme‌​)
                        .RequireAuthenticatedUser()
                        .Build());
                });

     

  4. Before making further modifications we will add 2 methods used which will be used to validate the signature and issuer ( this has potential to be made much better 🙂 )Key is the “n” value and Expo is the “e” value in the keys you got form the url in AWS setup
            public RsaSecurityKey SigningKey(string Key, string Expo)
            {
                    RSA rrr = RSA.Create();
    
                    rrr.ImportParameters(
                        new RSAParameters()
                        {
                            Modulus =  Base64UrlEncoder.DecodeBytes(Key),
                            Exponent = Base64UrlEncoder.DecodeBytes(Expo)
                        }
                    );
        
                    return new RsaSecurityKey(rrr);  
            }
    
            public TokenValidationParameters TokenValidationParameters(string issuer)
            {
                    // Basic settings - signing key to validate with, audience and issuer.
                    return new TokenValidationParameters
                    {
                        // Basic settings - signing key to validate with, IssuerSigningKey and issuer.
                        IssuerSigningKey = this.SigningKey(<key-comes-here>,<expo-comes-here>),
                        ValidIssuer      = issuer,
                            
                        // when receiving a token, check that the signing key
                        ValidateIssuerSigningKey = true,
        
                        // When receiving a token, check that we've signed it.
                        ValidateIssuer = true,
        
                        // When receiving a token, check that it is still valid.
                        ValidateLifetime = true,
                            
                        // Do not validate Audience on the "access" token since Cognito does not supply it but it is      on the "id"
                        ValidateAudience = false,
        
                        // This defines the maximum allowable clock skew - i.e. provides a tolerance on the token expiry time 
                        // when validating the lifetime. As we're creating the tokens locally and validating them on the same 
                        // machines which should have synchronised time, this can be set to zero. Where external tokens are
                        // used, some leeway here could be useful.
                        ClockSkew = TimeSpan.FromMinutes(0)
                    };
                
            }

     

  5. Modify Configure method to enable JWT
                app.UseJwtBearerAuthentication(new JwtBearerOptions()
                { 
                    
                    TokenValidationParameters = this.TokenValidationParameters(<issuer-comes-here>)
                });

    The issuer format has the following format : https://cognito-idp.<region>.amazonaws.com/<user-pool-id>

     

  6. Modify controller and enable the authentication by using the following decorator
        [Authorize(Policy = "Bearer")]

     

Testing the solution

With the authentication enabled we get the following while requesting controller

> http http://localhost:5000/api/values

HTTP/1.1 401 Unauthorized
Content-Length: 0
Date: Sun, 30 Jul 2017 11:41:33 GMT
Server: Kestrel
WWW-Authenticate: Bearer

 

And if we pass the JWT 🙂

http --auth-type=jwt -v http://localhost:5000/api/values

GET /api/values HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Authorization: Bearer ey.....
Host: localhost:5000
User-Agent: HTTPie/0.9.9



HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Date: Sun, 30 Jul 2017 11:45:41 GMT
Server: Kestrel
Transfer-Encoding: chunked

 

 

Code

Full gist below 🙂