Skip to main content

ECS Infra Metrics and Logs Collection using Daemon Service

This documentation will guide you about how to collect metrics and logs from your ECS infrastructure using a daemon service. The daemon service will run a container in each nodes of your ECS cluster. The container will collect metrics and logs from the instance and send them to SigNoz.

Select the type of SigNoz instance you are running: SigNoz Cloud or Self-Hosted.

Prerequisites​

  • An ECS cluster running with at least one task definition
  • ECS cluster of launch type EC2 or External
  • SigNoz Cloud account
note

If you want to collect metrics and other data for Fargate launch type, checkout this documentation.


Select the type of ECS cluster you are running: EC2 or External.

Below are the steps to collect your metrics and logs from ECS infrastructure:

Send Data from your Application deployed on ECS:

Setting up Daemon Service​


Step 1: Daemon Service Template​

This step creates a new service within your Amazon ECS (Elastic Container Service) cluster. The purpose of this service is to deploy a container that functions as a daemon. This service will run a container that will send data such as ECS infrastructure metrics and logs from docker containers and send it to SigNoz Cloud. It also acts as a gateway for any incoming OTLP telemetry data from your applications or services. This means it can receive telemetry data (such as traces and metrics) sent using OTLP from your applications, and then forward this data to SigNoz Cloud for analysis and visualization.

We will use CloudFormation template which includes parameters and configurations that define how the daemon service should be set up. For example, it specifies the container image to use for the daemon, the necessary environment variables, and network settings. The template is designed to be customizable, allowing you to adjust certain parameters to fit your specific ECS setup and requirements to create the daemon service with our custom parameters mentioned in Step 3.

Download the daemon-template.yaml using the command below:

wget https://github.com/SigNoz/benchmark/raw/main/ecs/ec2/daemon-template.yaml

Step 2: Create SigNoz OtelCollector Config​

otelcol-daeomn parameter in AWS Parameter Store
OTel Collector config in AWS Parameter Store

  • Navigate to AWS Parameter Store and create a new parameter named /ecs/signoz/otelcol-daemon.yaml.
wget https://github.com/SigNoz/benchmark/raw/main/ecs/otelcol-daemon.yaml
  • Update {region} and SIGNOZ_INGESTION_KEY values in your YAML configuration file and copy the updated content of the otelcol-sidecar.yaml file and paste it in the value field of the /ecs/signoz/otelcol-daemon.yaml parameter that you created.

You will be able to get {region} and SIGNOZ_INGESTION_KEY values in your SigNoz Cloud account under Settings --> Ingestion Settings.

Ingestion key details
Ingestion details in SigNoz dashboard

info

After successful set up, feel free to remove logging exporter if it gets too noisy. To do so, simply remove the logging exporter from the exporters list in the following pipelines: traces, metrics, and logs from the otelcol-daemon.yaml file.


Step 3: Create Daemon Service​

Now configure the daemon service you created in Step 1 with the OpenTelemetry Collector (OTelCollector) configuration you prepared and stored in AWS Parameter Store in Step 2. The goal is to ensure the daemon service is properly set up to collect, process, and export your ECS infrastructure metrics and logs to SigNoz Cloud.

As a first step to configure the daemon service, you need to set the environment variable by running the command (using your AWS CLI) below:

export CLUSTER_NAME=<YOUR-ECS-CLUSTER-NAME>
export REGION=<YOUR-ECS-REGION>
export COMMAND=--config=env:SIGNOZ_CONFIG_CONTENT
export SIGNOZ_CONFIG_PATH=/ecs/signoz/otelcol-daemon.yaml

Where, <YOUR-ECS-CLUSTER-NAME> - Name of your ECS cluster. For example, my-test-cluster <YOUR-ECS-REGION> - Region in which your ECS cluster is running. For example, us-east-1

note

Make sure you have CLUSTER_NAME and REGION environment variables set to the proper values before running any aws commands.

Now with the environment variables set, you proceed to run an AWS CloudFormation command to create the stack that represents your daemon service. This command uses the aws cloudformation create-stack for creating a new stack.

Now run the following command (using your AWS CLI) to create the daemon service:

aws cloudformation create-stack --stack-name AOCECS-daemon-${CLUSTER_NAME}-${REGION} \
    --template-body file://daemon-template.yaml \
    --parameters ParameterKey=ClusterName,ParameterValue=${CLUSTER_NAME} \
    ParameterKey=CreateIAMRoles,ParameterValue=True \
        ParameterKey=command,ParameterValue=${COMMAND} \
        ParameterKey=SigNozConfigPath,ParameterValue=${SIGNOZ_CONFIG_PATH} \
    --capabilities CAPABILITY_NAMED_IAM \
    --region ${REGION}

Step 4: Verify Daemon Service​

To verify that the daemon service is running, you can run the following command:

aws ecs list-tasks --cluster ${CLUSTER_NAME} --region ${REGION}

You should see the task ARN of the daemon service in the output.

Step 5: Verify Data in SigNoz​

To verify that the data is being sent to SigNoz Cloud, you can go to the dashboard section of SigNoz and import one of the following dashboards below:

Hostmetrics Dashboard showing data for ECS cluster
Hostmetrics Dashboard showing data for ECS cluster

(Optional) Step 6: Clean Up​

In a cloud environment where resources are billed based on usage, cleaning up resources is crucial. This step involves removing the daemon service and any associated resources that were created during the setup process to collect and forward metrics and logs from your ECS infrastructure to SigNoz. To clean up the daemon service, you can run the following command:

aws cloudformation delete-stack --stack-name AOCECS-daemon-${CLUSTER_NAME}-${REGION} --region ${REGION}

Send Data from Applications​

In this section, we will see how to send data from applications deployed in ECS to SigNoz Cloud using Daemon Service we created in the previous section.

Step 1: Add OpenTelemetry Instrumentation to your Application​

To add OpenTelemetry instrumentation to your application, you can follow the docs here.

note

This step include adding the OpenTelemetry SDK as well as the initialization code to your application codebase and rebuilding the application container image.

Step 2: Add Entrypoint to your Application Container​

We need to add an entrypoint to the application container to set the OTEL_EXPORTER_OTLP_ENDPOINT environment variable to the endpoint of the daemon service. Such that the application container can send data to the daemon container running in the same host.

Method 1: Updating entrypoint in task definition

We need to obtain the endpoint or IP address of the instance on which the task is running. We can do this by querying the metadata service of the instance. For EC2, the metadata service is available at 169.254.169.254.

The entryPoint will look like:

{
      ...,
      "entryPoint": [
        "sh",
        "-c",
        "export OTEL_EXPORTER_OTLP_ENDPOINT=\"http://$(curl http://169.254.169.254/latest/meta-data/local-ipv4):4317\"; <Application Startup Commands>"
      ],
            "command": [],
            ...
}
  • Replace <Application Startup Commands> with the commands to start your application.

Step 3: Add Service Name of your Application​

To add the service name of your application, you need to set the OTEL_RESOURCE_ATTRIBUTES environment variable of the application container to service.name=<your-service-name>.

In your task definition, you can add the following lines.


...
    ContainerDefinitions:
        - Name: <your-container-name>
          ...
          Environment:
            - Name: OTEL_RESOURCE_ATTRIBUTES
              Value: service.name=<your-service-name>
          ...
...

In case of JSON task definition, you can add the following lines.

...
    "containerDefinitions": [
        {
            "name": "<your-container-name>",
            ...
            "environment": [
                {
                    "name": "OTEL_RESOURCE_ATTRIBUTES",
                    "value": "service.name=<your-service-name>"
                }
            ],
            ...
        }
    ],
...

Step 4: Rebuild and Deploy Application Container​

After following previous step, you need to rebuild the application container and deploy it to ECS cluster.

Step 5: Verify Data in SigNoz​

You will need to generate some traffic to your application. After which you can go to the SigNoz Cloud Services page and you should see your application in the services list.