AWS networking basics with terraform and a tiny bit of microservices

I have a moniker for myself - the accidental software engineer. I never imagined 5 years ago that I would be in this industry neither did I image that I would soon have to learn DevOps and aws architecture but I guess that is par for the course in a start up. I found myself in an awkward situation late last year when our DevOps gave notice that he was leaving. I had to pick up this skill otherwise, I would not have been able to deploy the code I had written. This exposure to our aws infrastructure piqued my interest to dive deeper into the subject matter and after months of hacking around, here we are - the accidental DevOps.

In this article, I will write and try to explain where I can the basic networking architecture in aws. I will implement a very simple application that sends an email once you interact with the server. The image below explains what I will achieve.

In the above image, we have a virtual private cloud inside an aws region, we then have public subnets where our flask web servers will be located. A load balancer is used to distribute the traffic going to these two servers. We have a private subnet where we run a python service that basically communicates with an sqs instance, extracts the payload and sends an email. The code for the servers and service will be implemented in python whilst the creation of the aws resources will be implemented with terraform an infrastructure as code software tool(yes you can create all these resources on aws without clicking a button for the most part). You can replicate this project by opening an aws account, pulling the code from this repository here and making slight changes to the code. You will have to undo all your changes in aws else you might incur some charges especially with anything named elastic(aren't all the resources there named this way).

When you create an aws account, you are assigned a default region and aws also creates a virtual private cloud(VPC) for you. A VPC is basically a data centre where you can host your servers. Each region also has availability zones. These are selected zones in your region where the physical hardware is located optimised to create a buffer in case of floods, bad weather etc to protect from data loss. Inside every VPC is a subnet. These could be private or public depending on the route table configuration and security group in which they are associated to. Since your network is private, you need some way to communicate with the internet. You basically have 3 options one is through an internet gateway for public subnets and the other two are through a vpc endpoint and a NAT Gateway. The later option is very expensive and should be avoided in my opinion especially if you are doing this on a private account. Now let us see some code.

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
      version = "~> 3.27"
    }
  }

  required_version = ">= 0.14.9"
}


provider "aws" {
  profile = "default"
  region = "eu-north-1"
}


resource "aws_vpc" "ecs-container-vpc" {
  cidr_block = "10.0.0.0/16"
    enable_dns_support = true
    enable_dns_hostnames = true


  tags = {
    Name = "ecs-container-vpc"
  }
}

resource "aws_internet_gateway" "ecs-container-gw" {
  vpc_id = aws_vpc.ecs-container-vpc.id

  tags = {
  Name = "ecs-container-gw"
  }
}

resource "aws_subnet" "ecs-container-subnet-1" {
  vpc_id     = aws_vpc.ecs-container-vpc.id
  cidr_block = "10.0.1.0/24"
  availability_zone = "eu-north-1a"

  tags = {
    Name = "ecs-container-subnet-1"
  }
}

resource "aws_subnet" "ecs-container-subnet-2" {
  vpc_id     = aws_vpc.ecs-container-vpc.id
  cidr_block = "10.0.2.0/24"
  availability_zone = "eu-north-1a"

  tags = {
    Name = "ecs-container-subnet-2"
  }
}

resource "aws_subnet" "ecs-container-subnet-3" {
  vpc_id     = aws_vpc.ecs-container-vpc.id
  cidr_block = "10.0.20.0/24"
  availability_zone = "eu-north-1b"

  tags = {
    Name = "ecs-container-subnet-3"
  }
}

resource "aws_subnet" "ecs-container-subnet-4" {
  vpc_id     = aws_vpc.ecs-container-vpc.id
  cidr_block = "10.0.21.0/24"
  availability_zone = "eu-north-1b"

  tags = {
    Name = "ecs-container-subnet-4"
  }
}


resource "aws_route_table" "ecs-container-route-table" {
  vpc_id = aws_vpc.ecs-container-vpc.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.ecs-container-gw.id
  }

  route {
    ipv6_cidr_block        = "::/0"
    gateway_id = aws_internet_gateway.ecs-container-gw.id
  }

  tags = {
    Name = "ecs-container-route-table"
  }
}

resource "aws_route_table" "ecs-container-route-table-private" {
  vpc_id = aws_vpc.ecs-container-vpc.id

  tags = {
    Name = "ecs-container-route-table-private"
  }
}

resource "aws_route_table_association" "ecs-container-route-association" {

  for_each    = toset([aws_subnet.ecs-container-subnet-1.id, aws_subnet.ecs-container-subnet-3.id])  

  subnet_id   = each.key
  route_table_id  = aws_route_table.ecs-container-route-table.id
}

resource "aws_route_table_association" "ecs-container-route-association-private" {

  for_each    = toset([aws_subnet.ecs-container-subnet-2.id, aws_subnet.ecs-container-subnet-4.id])  

  subnet_id   = each.key
  route_table_id  = aws_route_table.ecs-container-route-table-private.id
}

In the first part of this code, I have used the eu-north-1 region - Stockholm which is a favorite city of mine. We create the VPC, an internet gateway and four subnets two in the availablilty zones 1a and 1b. The cidr block is out of the scope of this article but the subnets usually inherit a subset of the cidr block from the VPC. To instruct the VPC where to direct all incoming traffic, we need to create a route table. By default AWS creates a local route so the router can redirect all traffic coming from within. If you want a route to the outside world, you need to create a route that accepts all ip addresses like so "0.0.0.0/0" and associate this with the id of your internet gateway. Finally, we associate the route tables with their respective subnets. The configuration of these route tables is what classifies a subnet as public or private.

resource "aws_vpc_endpoint" "s3-api" {
    vpc_id       = aws_vpc.ecs-container-vpc.id
    service_name = "com.amazonaws.<region-name>.s3"
  tags = {
    Name = "s3-api"
  }

}

resource "aws_vpc_endpoint_route_table_association" "ecr-s3" {
  route_table_id  = aws_route_table.ecs-container-route-table.id
  vpc_endpoint_id = aws_vpc_endpoint.s3-api.id
}


resource "aws_vpc_endpoint_route_table_association" "ecr-s3-private" {
  route_table_id  = aws_route_table.ecs-container-route-table-private.id
  vpc_endpoint_id = aws_vpc_endpoint.s3-api.id
}



resource "aws_vpc_endpoint" "ecr-api" {
    vpc_id       = aws_vpc.ecs-container-vpc.id
    service_name = "com.amazonaws.<region-name>.ecr.api"
    vpc_endpoint_type = "Interface"
    private_dns_enabled = true
    subnet_ids        = [aws_subnet.ecs-container-subnet-1.id, aws_subnet.ecs-container-subnet-4.id]
    security_group_ids = [
        aws_security_group.private.id, aws_security_group.allow-web.id
    ]
  tags = {
    Name = "ecr-api"
  }

}

resource "aws_vpc_endpoint" "ecr-dkr" {
    vpc_id       = aws_vpc.ecs-container-vpc.id
    service_name = "com.amazonaws.<region-name>.ecr.dkr"
    vpc_endpoint_type = "Interface"
    private_dns_enabled = true
    subnet_ids        = [aws_subnet.ecs-container-subnet-1.id, aws_subnet.ecs-container-subnet-4.id]
    security_group_ids = [
        aws_security_group.private.id, aws_security_group.allow-web.id
    ]
  tags = {
    Name = "dkr-api"
  }

}

resource "aws_vpc_endpoint" "logs" {
    vpc_id       = aws_vpc.ecs-container-vpc.id
    service_name = "com.amazonaws.<region-name>.logs"
    vpc_endpoint_type = "Interface"
    private_dns_enabled = true
    subnet_ids        = [aws_subnet.ecs-container-subnet-1.id, aws_subnet.ecs-container-subnet-4.id]
    security_group_ids = [
        aws_security_group.private.id, aws_security_group.allow-web.id
    ]
  tags = {
    Name = "logs"
  }

}

resource "aws_vpc_endpoint" "ssm" {
    vpc_id       = aws_vpc.ecs-container-vpc.id
    service_name = "com.amazonaws.<region-name>.ssm"
    vpc_endpoint_type = "Interface"
    private_dns_enabled = true
    subnet_ids        = [aws_subnet.ecs-container-subnet-1.id, aws_subnet.ecs-container-subnet-4.id]
    security_group_ids = [
        aws_security_group.private.id, aws_security_group.allow-web.id
    ]
  tags = {
    Name = "ssm"
  }

}

resource "aws_vpc_endpoint" "secretsmanager" {
    vpc_id       = aws_vpc.ecs-container-vpc.id
    service_name = "com.amazonaws.<region-name>.secretsmanager"
    vpc_endpoint_type = "Interface"
    private_dns_enabled = true
    subnet_ids        = [aws_subnet.ecs-container-subnet-1.id, aws_subnet.ecs-container-subnet-4.id]
    security_group_ids = [
        aws_security_group.private.id, aws_security_group.allow-web.id
    ]
  tags = {
    Name = "secretsmanager"
  }

}

resource "aws_vpc_endpoint" "sqs" {
    vpc_id       = aws_vpc.ecs-container-vpc.id
    service_name = "com.amazonaws.<region-name>.sqs"
    vpc_endpoint_type = "Interface"
    private_dns_enabled = true
    subnet_ids        = [aws_subnet.ecs-container-subnet-1.id, aws_subnet.ecs-container-subnet-4.id]
    security_group_ids = [
        aws_security_group.private.id, aws_security_group.allow-web.id
    ]
  tags = {
    Name = "sqs"
  }

}

resource "aws_vpc_endpoint" "ses" {
    vpc_id       = aws_vpc.ecs-container-vpc.id
    service_name = "com.amazonaws.<region-name>.email-smtp"
    vpc_endpoint_type = "Interface"
    private_dns_enabled = true
    subnet_ids        = [aws_subnet.ecs-container-subnet-1.id, aws_subnet.ecs-container-subnet-4.id]
    security_group_ids = [
        aws_security_group.private.id, aws_security_group.allow-web.id
    ]
  tags = {
    Name = "ses"
  }

}




resource "aws_security_group" "private" {
  name        = "private_traffic"
  description = "Limited private traffic"
  vpc_id      = aws_vpc.ecs-container-vpc.id

  ingress {
    description      = "Self"
    from_port        = 0
    to_port          = 0
        protocol         = "-1"
    self             = true
  }

  ingress {
    description     = "HTTP"
    from_port       = 80
    to_port         = 80
    protocol        = "all"
    security_groups = [aws_security_group.load-balancer-web.id]
  }

    egress {
        from_port        = 0
        to_port          = 0
        protocol         = "-1"
        cidr_blocks      = ["0.0.0.0/0"]
        ipv6_cidr_blocks = ["::/0"]
    }

  tags = {
    Name = "private-network"
  }
}

resource "aws_security_group" "allow-web" {
  name        = "allow_web_traffic"
  description = "Allow Web inbound traffic"
  vpc_id      = aws_vpc.ecs-container-vpc.id

  ingress {
    description      = "Self"
    from_port        = 0
    to_port          = 0
        protocol         = "-1"
    self             = true
  }

  ingress {
    description      = "HTTP"
    from_port        = 80
    to_port          = 80
    protocol         = "tcp"
    cidr_blocks      = ["0.0.0.0/0"]
  }

  ingress {
    description     = "HTTP"
    from_port       = 80
    to_port         = 80
    protocol        = "all"
    security_groups = [aws_security_group.load-balancer-web.id]
  }

    egress {
        from_port        = 0
        to_port          = 0
        protocol         = "-1"
        cidr_blocks      = ["0.0.0.0/0"]
        ipv6_cidr_blocks = ["::/0"]
    }

  tags = {
    Name = "allow-web"
  }
}

resource "aws_security_group" "load-balancer-web" {
  name        = "elbs-allow_web_traffic"
  description = "Allow Web inbound traffic"
  vpc_id      = aws_vpc.ecs-container-vpc.id

  ingress {
    description      = "HTTP"
    from_port        = 80
    to_port          = 80
    protocol         = "tcp"
    cidr_blocks      = ["0.0.0.0/0"]
  }

    egress {
        from_port        = 0
        to_port          = 0
        protocol         = "-1"
        cidr_blocks      = ["0.0.0.0/0"]
        ipv6_cidr_blocks = ["::/0"]
    }

  tags = {
    Name = "load-balancer-web"
  }
}


resource "aws_lb" "ecs-container-load-balancer" {
  name               = "ecs-container-load-balancer"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [aws_security_group.load-balancer-web.id]
  subnets            = [aws_subnet.ecs-container-subnet-1.id, aws_subnet.ecs-container-subnet-3.id]

  enable_deletion_protection = true
    tags = {
      Environment = "development"
  }
}

resource "aws_lb_target_group" "alb-target-group" {
  name        = "alb-target-group"
  port        = 80
  protocol    = "HTTP"
  target_type = "ip"
  vpc_id      = aws_vpc.ecs-container-vpc.id
    health_check {
        path = "/api"
    }

  depends_on = [aws_lb.ecs-container-load-balancer]
}

resource "aws_lb_listener" "http" {
  load_balancer_arn = aws_lb.ecs-container-load-balancer.arn
  port              = "80"
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.alb-target-group.arn

  }
}

In this second section, we create vpc endpoints, security groups and load balancers. Remember I said for private subnets to communicate with the internet, aws needs either a NAT Gateway which is very expensive as my savings sadly discovered or a vpc endpoint. Basically the vpc endpoint creates a private link communication from your vpc to aws resources such as s3, sqs, dynamodb. For this to work, you need to enable dns support and hostname for your vpc and enable private dns when configuring your endpoints. There is something worth nothing here and I spent a lot of time trying to get this part work, once you enable private dns for the vpc endpoints, both your public and private subnets will access these vpc endpoints via this configuration for communication with aws services according to this article. This also means you have to attach the security groups and respective private and public subnets to these vpc endpoint configurations.

We then configure the security groups. For the public security group we open port 80 for incoming http traffic. This is also the port we expose in our docker container hosting our flask server. For outgoing traffic we open all the ports. Another gotcha here is that you have to assign the security group of the load balancer to the security group of the public subnets so it can accept traffic from the load balancer. This setup actually means that if we wanted, our web servers could have also been hosted in a private subnet since the load balancer is public. Finally we configure our load balancer attaching a listener that also listens for http traffic on port 80 and forwards it to our load balancer. We assign our vpc as the target group for this load balancer. In this project, I only listen to the endpoint "/api" in the load balancers health check. The default is "" but this will always fail because I don't listen to this route in the server.

# Create sqs
resource "aws_sqs_queue" "email-queue" {
    name                      = "email-queue"
    delay_seconds             = 90
    max_message_size          = 2048
    message_retention_seconds = 86400
    receive_wait_time_seconds = 10


    tags = {
        Environment = "development"
    }
}

resource "aws_iam_role" "ecs-flask-container-role" {
  name = "ecs-flask-container-role"

  # Terraform's "jsonencode" function converts a
  # Terraform expression result to valid JSON syntax.
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
    {
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Sid    = ""
      Principal = {
      Service = "ecs-tasks.amazonaws.com"
      }
    },
    ]
  })
   managed_policy_arns = [
                aws_iam_policy.sqs-policy.arn, 
                aws_iam_policy.ses-policy.arn, 
                aws_iam_policy.ecr-policy.arn, 
                aws_iam_policy.s3-policy.arn,
                aws_iam_policy.cloudwatch-policy.arn,
                aws_iam_policy.secrets-policy.arn,
            ]

  tags = {
    tag-key = "ecs-flask-container-role"
  }
}


resource "aws_iam_policy" "sqs-policy" {
  name = "sqs-policy"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action   = ["sqs:*"]
        Effect   = "Allow"
        Resource = "*"
      },
    ]
  })
}

resource "aws_iam_policy" "s3-policy" {
  name = "s3-policy"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action   = ["s3:*"]
        Effect   = "Allow"
        Resource = "*"
      },
    ]
  })
}

resource "aws_iam_policy" "ses-policy" {
  name = "ses-policy"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action   = ["ses:SendEmail", "ses:SendRawEmail"]
        Effect   = "Allow"
        Resource = "*"
      },
    ]
  })
}

resource "aws_iam_policy" "ecr-policy" {
  name = "ecr-policy"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action   = [
          "ecr:GetAuthorizationToken",
                  "ecr:BatchCheckLayerAvailability",
                  "ecr:GetDownloadUrlForLayer",
                  "ecr:BatchGetImage",
                  "logs:CreateLogStream",
                  "logs:PutLogEvents"
          ]
        Effect   = "Allow"
        Resource = "*"
      },
    ]
  })
}

resource "aws_iam_policy" "cloudwatch-policy" {
  name = "cloudwatch-policy"

    policy = jsonencode({
        Version = "2012-10-17",
        Statement = [
            {
                Action = [
                    "autoscaling:Describe*",
                    "cloudwatch:*",
                    "logs:*",
                    "sns:*",
                    "iam:GetPolicy",
                    "iam:GetPolicyVersion",
                    "iam:GetRole"
                ],
                Effect = "Allow",
                Resource = "*"
            },
            {
                Effect = "Allow",
                Action = "iam:CreateServiceLinkedRole",
                Resource = "arn:aws:iam::*:role/aws-service-role/events.amazonaws.com/AWSServiceRoleForCloudWatchEvents*",
                Condition = {
                    StringLike = {
                        "iam:AWSServiceName": "events.amazonaws.com"
                    }
                }
            }
        ]
    })
}




resource "aws_iam_policy" "secrets-policy" {
  name = "secrets-policy"

    policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
        {
            Action = [
                "secretsmanager:*",
                "cloudformation:CreateChangeSet",
                "cloudformation:DescribeChangeSet",
                "cloudformation:DescribeStackResource",
                "cloudformation:DescribeStacks",
                "cloudformation:ExecuteChangeSet",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeVpcs",
                "kms:DescribeKey",
                "kms:ListAliases",
                "kms:ListKeys",
                "lambda:ListFunctions",
                "rds:DescribeDBClusters",
                "rds:DescribeDBInstances",
                "redshift:DescribeClusters",
                "tag:GetResources"
            ],
            Effect = "Allow",
            Resource = "*"
        },
        {
            Action = [
                "lambda:AddPermission",
                "lambda:CreateFunction",
                "lambda:GetFunction",
                "lambda:InvokeFunction",
                "lambda:UpdateFunctionConfiguration"
            ],
            Effect = "Allow",
            "Resource": "arn:aws:lambda:*:*:function:SecretsManager*"
        },
        {
            Action =  [
                "serverlessrepo:CreateCloudFormationChangeSet",
                "serverlessrepo:GetApplication"
            ],
            Effect = "Allow",
            Resource = "arn:aws:serverlessrepo:*:*:applications/SecretsManager*"
        },
        {
            Action = [
                "s3:GetObject"
            ],
            Effect = "Allow",
            Resource = [
                "arn:aws:s3:::awsserverlessrepo-changesets*",
                "arn:aws:s3:::secrets-manager-rotation-apps-*/*"
            ]
        }
    ]
})
}

In the third section, we create the resources and iam role that our containers need to operate. We need only one aws resource which is sqs(simple queue service). When a user hits the api endpoint of our server like this www.whatever.com/api?name=George&email=xxx@xxx.com the flask server reads this and puts it on the queue. The python microservice then extracts this information from the queue and sends an email to the address given. To run our containers on aws we need a service role that can access s3 to download the files needed to set up the serverless machine, sqs to read and write to the queuing system, secrets to download secrets needed for the repository, CloudWatch to store our logs, ecr to extract the code pushed to the amazon private repository and ses to send emails to the clients. I have given this service role full access to all these resources but usually you want to give only what is needed for example the server needs only push access to the sqs and the python service needs only read access but I didn't want to complicate the project with these details.

# Create repository

resource "aws_ecr_repository" "flask-server" {
  name                 = "flask-server"
  image_tag_mutability = "MUTABLE"

  image_scanning_configuration {
    scan_on_push = true
  }
}

resource "aws_ecr_repository" "event-consumer" {
  name                 = "event-consumer"
  image_tag_mutability = "MUTABLE"

  image_scanning_configuration {
    scan_on_push = true
  }
}


# Create cluster

resource "aws_ecs_cluster" "flask-fargate-cluster" {
  name = "flask-fargate-cluster"

  setting {
    name  = "containerInsights"
    value = "enabled"
  }
}

resource "aws_ecs_task_definition" "task-definition-flask-server" {
  family = "ecs-container-task-definition"
  container_definitions = jsonencode([
    {
      name      = "flask-server"
      image     = "<account-id>.dkr.<region-name>.amazonaws.com/flask-server:latest"
      cpu       = 1024
      memory    = 2048
      essential = true
      portMappings = [
        {
        containerPort = 80
        hostPort      = 80
        }
      ],
            logConfiguration = {
                logDriver = "awslogs",
                options = {
                    awslogs-group = "ecs",
                    awslogs-stream-prefix = "flask-server",
                    awslogs-region = "eu-north-1"
                }
            }
    }
  ])
    network_mode    = "awsvpc"
  cpu       = 1024
  memory    = 2048
    requires_compatibilities = ["FARGATE"]
    execution_role_arn = aws_iam_role.ecs-flask-container-role.arn
    task_role_arn  = aws_iam_role.ecs-flask-container-role.arn
}

resource "aws_ecs_task_definition" "task-definition-event-consumer" {
  family = "task-definition-event-consumer"
  container_definitions = jsonencode([
    {
      name      = "event-consumer"
      image     = "<account-id>.dkr.<region-name>.amazonaws.com/event-consumer:latest"
      cpu       = 1024
      memory    = 2048
      essential = true
      portMappings = [],
            logConfiguration = {
                logDriver = "awslogs",
                options = {
                    awslogs-group = "ecs",
                    awslogs-stream-prefix = "event-consumer",
                    awslogs-region = "eu-north-1"
                }
            }
    }
  ])
    network_mode    = "awsvpc"
  cpu       = 1024
  memory    = 2048
    requires_compatibilities = ["FARGATE"]
    execution_role_arn = aws_iam_role.ecs-flask-container-role.arn
    task_role_arn  = aws_iam_role.ecs-flask-container-role.arn
}

resource "aws_ecs_service" "flask-server" {
  name            = "flask-server"
  cluster         = aws_ecs_cluster.flask-fargate-cluster.id
  task_definition = aws_ecs_task_definition.task-definition-flask-server.arn
  launch_type     = "FARGATE"
  desired_count   = 2

  load_balancer {
    target_group_arn = aws_lb_target_group.alb-target-group.arn
    container_name   = "flask-server"
    container_port   = 80
  }

  network_configuration {
    subnets            = [aws_subnet.ecs-container-subnet-1.id, aws_subnet.ecs-container-subnet-3.id]
    security_groups    = [aws_security_group.allow-web.id]
    assign_public_ip = true
  }
}

resource "aws_ecs_service" "event-consumer" {
  name            = "event-consumer"
  cluster         = aws_ecs_cluster.flask-fargate-cluster.id
  task_definition = aws_ecs_task_definition.task-definition-event-consumer.arn
  launch_type     = "FARGATE"
  desired_count   = 2

  network_configuration {
    subnets            = [aws_subnet.ecs-container-subnet-2.id, aws_subnet.ecs-container-subnet-4.id]
        security_groups    = [aws_security_group.private.id]
  }
}

resource "aws_cloudwatch_log_group" "ecs" {
  name = "ecs"

  tags = {
    Environment = "development"
  }
}

resource "aws_cloudwatch_log_stream" "flask-server" {
  name           = "flask-server"
  log_group_name = aws_cloudwatch_log_group.ecs.name
}

resource "aws_cloudwatch_log_stream" "event-consumer" {
  name           = "event-consumer"
  log_group_name = aws_cloudwatch_log_group.ecs.name
}

In the last terraform section, we create an ecr (elastic container registry). This is where we host our docker images. Afterwards we create our cluster and two tasks. The tasks contain the serverless instances that hosts our server and service. Make sure you activate logs to be able to see what is happening in your instance. We also need to create two services. These services help start up the tasks and make sure they are running. Whenever a task goes down, it is the responsibility of the service to start it back up. It is also in the service that we assign the load balancer. For the web service, we attach the load balancer, the public subnets and the public security group. For the python service, we attach the private subnets and the private security group.

from flask import Flask, request
import boto3
import json


app = Flask(__name__)

@app.route('/api', methods=['GET'])
def api():
    args = request.args
    name = args.get("name")
    email = args.get("email")
    if not name or not email:
        return "<p>Set parameters</p>"
    post_query(name, email)
    return "<p>Your queue request was successful</p>"


def post_query(name, email):
    
    message = {
        "name": name,
        "email": email
    }
    sqs = boto3.client('sqs')

    print(f"About to upload message {message}")
    response = sqs.send_message(
        QueueUrl="https://sqs.<region-name>.amazonaws.com/<account-id>/email-queue",
        MessageBody=json.dumps(message)
    )
    
    
if __name__ == '__main__':
      app.run(host='0.0.0.0', port=80)

This is our server. It is a python flask server listening on port 80. It retrieves query strings from the endpoint "api" and posts the name and email to sqs.

xxxxxxxxxx
import boto3
import traceback
import json
from email.utils import formataddr
from smtplib import SMTP_SSL
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

def send_email(name, email):
    
    # code adapted from https://stackoverflow.com/questions/9087158/amazon-ses-smtp-python-usage
    SENDER = 'johndoe@johndoe.com'
    SENDERNAME = 'John Doe'

    RECIPIENT = email

    USERNAME_SMTP = "<AWS ASSIGNED SMTP USER NAME>"

    PASSWORD_SMTP = "<AWS ASSIGNED SMTP PASSWORD>"

    HOST = "email-smtp.eu-north-1.amazonaws.com"
    PORT = 465

    SUBJECT = 'Email delivery microservice'


    BODY_TEXT = (f"Hello {name} here your email!")

    BODY_HTML = f"""<html>
    <head></head>
    <body>
    <p>Hello {name} here is your email!</p>
    </body>
    </html>"""

    msg = MIMEMultipart('alternative')
    msg['Subject'] = SUBJECT
    msg['From'] = formataddr((SENDERNAME, SENDER))
    msg['To'] = RECIPIENT

    part1 = MIMEText(BODY_TEXT, 'plain')
    part2 = MIMEText(BODY_HTML, 'html')


    msg.attach(part1)
    msg.attach(part2)

    # Try to send the message.
    try:
        with SMTP_SSL(HOST, PORT) as server:
            server.login(USERNAME_SMTP, PASSWORD_SMTP)
            server.sendmail(SENDER, RECIPIENT, msg.as_string())
            server.close()
            print("Email sent!")
    except:
        error = traceback.format_exc()
        print(f"Failed because of {error}")


def main():
    # Create SQS client
    region = "eu-north-1"
    sqs = boto3.client('sqs', region_name=region)


    queue_url = "https://sqs.<region-name>.amazonaws.com/<account-id>/email-queue"
    # Long poll for message on provided SQS queue
    while True:
        response = sqs.receive_message(
            QueueUrl=queue_url,
            AttributeNames=[
                'SentTimestamp'
            ],
            MaxNumberOfMessages=10,
            MessageAttributeNames=[
                'All'
            ],
            WaitTimeSeconds=20
        )
        try:
            if "Messages" in response:
                for message in response['Messages']:
                    
                
                    response = json.loads(message['Body'])
                    receipt = message['ReceiptHandle']
                    name = response['name']
                    email = response['email']
                    send_email(name, email)
                    print(f"Deleting message with id: {receipt}")
                    sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=receipt)
        except:
            error = traceback.format_exc()
            print(f"Failed because of {error}")
            continue


if __name__ == "__main__":
    main()

This is the python service that retrieves and sends the email. It is worth noting that you can't use boto3 ses here because the vpc endpoint only allows smtp servers so you have to go to aws and configure your own smtp user. Unfornately I didn't add this part to the terraform script and you have to do it on aws.

# syntax=docker/dockerfile:1

FROM python:3.8-slim-buster


WORKDIR .

COPY requirements.txt requirements.txt

RUN pip3 install -r requirements.txt

COPY . .

ENV HOME /home/app

EXPOSE 80
CMD ["python", "server.py"]

To build the image, you have to run a docker file like this for both the server and python service and create a requirements.txt file. Don't worry the code is on GitHub in case yo have doubts on how to do it.

aws ecr get-login-password --region <aws-zone> | docker login --username AWS --password-stdin <aws-id>.dkr.ecr.<aws-zone>.amazonaws.com
docker build -t event-consumer .
docker tag event-consumer <aws-id>.dkr.ecr.<aws-zone>.amazonaws.com/event-consumer
docker push <aws-id>.dkr.ecr.<aws-zone>.amazonaws.com/event-consumer

Finally to build and push your images you have to do the above which is an example to push the image for the python service into the ecr repository created in terraform.

At the end of the day this is a lot of work just to create a server and service that sends an email that contains "Hello George here is your email" but I hope you enjoyed this article. Write a comment in the comments section below if you have a comment/question or there is something that didn't work or just shoot me an email. Maybe you too could become an accidental DevOps.