Ingesion Configuration

image

Select “Author from scratch” - default, give the function the name “flowlogs” and select “Java 8” for the run time as the jar file used in this lab was built using that runtime engine.

image

Assign a role to the function and configure timeouts and memory

Scroll down to the “Execution role” section. Choose existing role and select the role matching “*FlowlogsLambdaExecution*". In the basic settings section, choose 1 minute for the timeout and keep the function at 512MB. Save the configuration.image

After you create the execution role for the function, you need to trigger it. Click on the “Add trigger” button.

image

Select SQS as the event source and clicked on it

image

This exposes the final configuration. If you are not using a fresh account, select the SQS queue that matches what was created in your CloudFormation template as seen here in the s3sqs stack resources tab:

image

image

Make sure you do NOT enable the trigger and keep the batch size set to one. You will configure concurrency later on in the lab to deal with throughput.

Once you have added the trigger, you will see the following. The trigger needs to be disabled while you configure the rest of the function:

image

Provision concurrency settings

Click on the flowlogs section to expose the settings and scroll down to Concurrency section. Set reserve concurrency to 5 and save the setting.

image

image

Provision VPC settings

Since you run this solution in VPC, you will not use the public APIs. Lambda is created using ENI / private endpoints so that the Lambda traffic can stay private. Scroll down to the VPC settings and select the VPC created by this lab (detail are in the network stack under AWS CloudFormation. Select the private subnets.

image

Next, setup the security groups using the security group labeled like “*SQSPrivate*” and save the configuration.image

Configure the Environment variables and function code

You will need to pass in the Amazon Elasticsearch domain endpoint so that the Java code can communicate with Elasticsearch using the Java High Level Rest client. Earlier in the lab, you copied the endpoint for the domain. You will use that endpoint in this configuration. Scroll down to the Environment variables section and create a key named ELASTICSEARCH_ENDPOINT. Use the value from the output section in the CloudFormation template for the creation of the Elasticsearch cluster.

Now configure the code. In the dropdown under the Function code, select “Upload a file from S3”. Paste the following URL in the link section.

https://s3.amazonaws.com/search-sa-log-solutions/flowlogs/code/lambda-0.0.1-SNAPSHOT.jar

For the Handler, paste in the following:

com.amazonaws.sa.lambda.sqsinput.FlowLogsIndexer::handleRequest

Save the configuration changes so that do not get lost.

image

Enable the function to receive requests from SQS

Now that you configured the function, it is time to run it. Enable Lambda to read from SQS. Save the configuration.

image

Monitor the function to ensure data is flowing

Under the monitoring tab, ensure that you have events and invocations. If you jump to view the logs in CloudWatch, details on the invocations are present.

image

Click on the “View logs in CloudWatch” button and click on any of the log output to see what the code is doing:

image

Click on one of the logs to reveal details such as these:

image

Now you have data flowing to Elasticsearch. The logs do not indicate any errors, so let us verify that is the case.