Stack walkthrough

Some components do not show up in the diagrams, as the intent of this walkthrough is to clarify items discussed in the kickoff presentation.

Network Layer

The following diagram depicts the foundational elements of the VPC. It does not contain verbose detail.

The network stack establishes the backbone for the solution. Without networking, you cannot communicate with instances that need to connect with Amazon ES. For this workshop, the network stack deploys subnets across three availability zones. The stack creates both “public” and “private” subnets (ranges of IP addresses used for communication in a VPC). When someone states the term “public subnet”, it means the ability of the subnet to be reachable from an Internet Gateway.

This solution strives to keep the majority of the internet traffic private and public access to a minimum. The aggregate solution provides a secure foundation as an example of how you create a secure environment. Access to the internet is limited from instances in the VPC by using a NAT Gateway. The NAT Gateway provides a public IP address that does not allow inbound traffic.

The right hand side of the image shows the term “Service VPCs”. The majority of the AWS services deploy in a “service VPC” – one managed by the service team – and provide additional layers of isolation for security and service stability.

Authentication Layer

This diagram shows where the authentication layer deploys.

The authentication layer is Amazon Cognito. The layer creates a user pool (user names and passwords) and an identity pool (trusted sources that allow access via IAM). It also creates a default user and password. You can access these attributes in the output section of the AWS CloudFormation stack used to create this foundational element. In lab 1, you will pair this service with the Amazon Elasticsearch Service. This gives you control of who accesses your Amazon Elasticsearch Service domain.

Elasticsearch Layer

This layer provisions the Elasticsearch cluster via the Amazon Elasticsearch Service.

When you select the VPC option for your domain, the service presents ENIs (elastic network interfaces) in your VPC. VPC Private Endpoints provide the ENIs to your subnets. Each data node in your Elasticsearch domain gets an ENI for communication in the network. Private endpoints give you the ability to control the scope of traffic. It stays private. This gives you a secure foundation for your solution.

Proxy Layer

The proxy layer creates an NGINX server in a public subnet.

You use the proxy server to access Kibana (with Amazon Cognito authentication). The Kibana URL given to you when you deploy your domain in VPC is always private. You need a means by which you can access Kibana, with authentication, in a public environment for this lab. Your lab captures logs from NGINX and pushes them to the buffer layer. In lab 3, you monitor requests from this layer.

Buffer & S3 Bucket Notification Layer

The buffer and S3 bucket notification layer provides a landing zone for VPC Flow Logs and a notification pipeline that puts events in an SQS queue when objects land in the S3 bucket.

Flow Logs Layer

This template sets up the flow logs so they deliver to an Amazon S3 bucket. It requires a bucket to exist. That bucket is created in the prior template.