Saturday, May 25, 2024
HomeBig DataRun Kinesis Agent on Amazon ECS

Run Kinesis Agent on Amazon ECS


Kinesis Agent is a standalone Java software program software that gives an easy technique to accumulate and ship knowledge to Amazon Kinesis Information Streams and Amazon Kinesis Information Firehose. The agent repeatedly screens a set of recordsdata and sends new knowledge to the specified vacation spot. The agent handles file rotation, checkpointing, and retry upon failures. It delivers your whole knowledge in a dependable, well timed, and easy method. It additionally emits Amazon CloudWatch metrics that can assist you higher monitor and troubleshoot the streaming course of.

This publish describes the steps to ship knowledge from a containerized software to Kinesis Information Firehose utilizing Kinesis Agent. Extra particularly, we present learn how to run Kinesis Agent as a sidecar container for an software working in Amazon Elastic Container Service (Amazon ECS). After the information is in Kinesis Information Firehose, it may be despatched to any supported vacation spot, equivalent to Amazon Easy Storage Service (Amazon S3).

So as to current the important thing factors required for this setup, we assume that you’re conversant in Amazon ECS and dealing with containers. We additionally keep away from the implementation particulars and packaging strategy of our check knowledge technology software, known as the producer.

Answer overview

As depicted within the following determine, we configure a Kinesis Agent container as a sidecar that may learn recordsdata created by the producer container. On this occasion, the producer and Kinesis Agent containers share knowledge through a bind mount in Amazon ECS.

Solution design diagram

Stipulations

You need to fulfill the next conditions for the profitable completion of this process:

With these conditions in place, you’ll be able to start subsequent step to bundle a Kinesis Agent and your required agent configuration as a container in your native improvement machine.

Create a Kinesis Agent configuration file

We use the Kinesis Agent configuration file to configure the supply and vacation spot, amongst different knowledge switch settings. The next code makes use of the minimal configuration required to learn the contents of recordsdata matching /var/log/producer/*.log and publish them to a Kinesis Information Firehose supply stream referred to as kinesis-agent-demo:

{
    "firehose.endpoint": "firehose.ap-southeast-2.amazonaws.com",
    "flows": [
        {
            "deliveryStream": "kinesis-agent-demo",
            "filePattern": "/var/log/producer/*.log"
        }
    ]
}

Create a container picture for Kinesis Agent

To deploy Kinesis Agent as a sidecar in Amazon ECS, you first should bundle it as a container picture. The container should have Kinesis Agent, which and discover binaries, and the Kinesis Agent configuration file that you just ready earlier. Its entry level should be configured utilizing the start-aws-kinesis-agent script. This command is put in if you run the yum set up aws-kinesis-agent step. The ensuing Dockerfile ought to look as follows:

FROM amazonlinux

RUN yum set up -y aws-kinesis-agent which findutils
COPY agent.json /and so on/aws-kinesis/agent.json

CMD ["start-aws-kinesis-agent"]

Run the docker construct command to construct this container:

docker construct -t kinesis-agent .

After the picture is constructed, it needs to be pushed to a container registry like Amazon ECR so to reference it within the subsequent part.

Create an ECS process definition with Kinesis Agent and the appliance container

Now that you’ve got Kinesis Agent packaged as a container picture, you need to use it in your ECS process definitions to run as sidecar. To do this, you create an ECS process definition along with your software container (referred to as producer) and Kinesis Agent container. All containers in a process definition are scheduled on the identical container host and subsequently can share assets equivalent to bind mounts.

Within the following pattern container definition, we use a bind mount referred to as logs_dir to share a listing between the producer container and kinesis-agent container.

You need to use the next template as a place to begin, however make sure you change taskRoleArn and executionRoleArn to legitimate IAM roles in your AWS account. On this occasion, the IAM position used for taskRoleArn should have write permissions to Kinesis Information Firehose that you just specified earlier within the agent.json file. Moreover, be sure that the ECR picture paths and awslogs-region are modified as per your AWS account.

{
    "household": "kinesis-agent-demo",
    "taskRoleArn": "arn:aws:iam::111111111:position/kinesis-agent-demo-task-role",
    "executionRoleArn": "arn:aws:iam::111111111:position/kinesis-agent-test",
    "networkMode": "awsvpc",
    "containerDefinitions": [
        {
            "name": "producer",
            "image": "111111111.dkr.ecr.ap-southeast-2.amazonaws.com/producer:latest",
            "cpu": 1024,
            "memory": 2048,
            "essential": true,
            "command": [
                "-output",
                "/var/log/producer/test.log"
            ],
            "mountPoints": [
                {
                    "sourceVolume": "logs_dir",
                    "containerPath": "/var/log/producer",
                    "readOnly": false
                }
            ],
            "logConfiguration": {
                "logDriver": "awslogs",
                "choices": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "producer",
                    "awslogs-stream-prefix": "producer",
                    "awslogs-region": "ap-southeast-2"
                }
            }
        },
        {
            "title": "kinesis-agent",
            "picture": "111111111.dkr.ecr.ap-southeast-2.amazonaws.com/kinesis-agent:newest",
            "cpu": 1024,
            "reminiscence": 2048,
            "important": true,
            "mountPoints": [
                {
                    "sourceVolume": "logs_dir",
                    "containerPath": "/var/log/producer",
                    "readOnly": true
                }
            ],
            "logConfiguration": {
                "logDriver": "awslogs",
                "choices": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "kinesis-agent",
                    "awslogs-stream-prefix": "kinesis-agent",
                    "awslogs-region": "ap-southeast-2"
                }
            }
        }
    ],
    "volumes": [
        {
            "name": "logs_dir"
        }
    ],
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "2048",
    "reminiscence": "4096"
}

Register the duty definition with the next command:

aws ecs register-task-definition --cli-input-json file://./task-definition.json

Run a brand new ECS process

Lastly, you’ll be able to run a brand new ECS process utilizing the duty definition you simply created utilizing the aws ecs run-task command. When the duty is began, it is best to be capable to see two containers working underneath that process on the Amazon ECS console.

Amazon ECS console screenshot

Conclusion

This publish confirmed how easy it’s to run Kinesis Agent in a containerized atmosphere. Though we used Amazon ECS as our container orchestration service on this publish, you need to use a Kinesis Agent container in different environments equivalent to Amazon Elastic Kubernetes Service (Amazon EKS).

To study extra about utilizing Kinesis Agent, seek advice from Writing to Amazon Kinesis Information Streams Utilizing Kinesis Agent. For extra details about Amazon ECS, seek advice from the Amazon ECS Developer Information.


Concerning the Creator

Buddhike de Silva is a Senior Specialist Options Architect at Amazon Internet Providers. Buddhike helps clients run giant scale streaming analytics workloads on AWS and make the very best out of their cloud journey.

RELATED ARTICLES

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Most Popular

Recent Comments