Collecting Syslogs

With SigNoz you can collect your syslogs logs and perform different queries on top of it. We will demonstrate how to configure rsyslog to forward system logs to tcp endpoint of otel-collector and use syslog receiver in OpenTelemetry Collector to receive and parse the logs. Below are the steps to collect syslogs.

Collect Syslogs in SigNoz cloud

If you don’t already have a SigNoz cloud account, you can sign up here.

Add otel collector binary to your VM by following this guide.
Add the syslog reciever to config.yaml to otel-collector.
```
receivers:
  syslog:
    tcp:
      listen_address: "0.0.0.0:54527"
    protocol: rfc3164
    location: UTC
    operators:
      - type: move
        from: attributes.message
        to: body
...
```
Here we are collecting the logs and moving message from attributes to body using operators that are available. You can read more about operators here.
For more configurations that are available for syslog receiver please check here.

Next we will modify our pipeline inside config.yaml of otel-collector to include the receiver we have created above.

service:
    ....
    logs:
        receivers: [otlp, syslog]
        processors: [batch]
        exporters: [otlp]

Now we can restart the otel collector so that new changes are applied and we can forward our logs to port 54527.

Modify your rsyslog.conf file present inside /etc/ by running the following command:

sudo vim /etc/rsyslog.conf

and adding the this line at the end

template(
  name="UTCTraditionalForwardFormat"
  type="string"
  string="<%PRI%>%TIMESTAMP:::date-utc% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"
)

*.* action(type="omfwd" target="0.0.0.0" port="54527" protocol="tcp" template="UTCTraditionalForwardFormat")

For production use cases it is recommended to use something like below:

template(
  name="UTCTraditionalForwardFormat"
  type="string"
  string="<%PRI%>%TIMESTAMP:::date-utc% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"
)

*.*  action(type="omfwd" target="0.0.0.0" port="54527" protocol="tcp"
        action.resumeRetryCount="10"
        queue.type="linkedList" queue.size="10000" template="UTCTraditionalForwardFormat")

So that you have retries and queue in place to de-couple the sending from the other logging action. Also we are assuming that you are running the otel binary on the same host. If not, the value of target might change depending on your environment.

Now restart your rsyslog service by running sudo systemctl restart rsyslog.service
You can check the status of service by running sudo systemctl status rsyslog.service
If there are no errors your logs will be visible on SigNoz UI.

Collect Syslogs in Self-Hosted SigNoz

Modify the docker-compose.yaml file present inside deploy/docker/clickhouse-setup to expose a port, in this case 54527 so that we can forward syslogs to this port.

...
otel-collector:
    image: signoz/signoz-otel-collector:0.88.11
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "54527:54527"
...

Add the syslog reciever to otel-collector-config.yaml which is present inside deploy/docker/clickhouse-setup
```
receivers:
  syslog:
    tcp:
      listen_address: "0.0.0.0:54527"
    protocol: rfc3164
    location: UTC
    operators:
      - type: move
        from: attributes.message
        to: body
...
```
Here we are collecting the logs and moving message from attributes to body using operators that are available. You can read more about operators here
For more configurations that are available for syslog receiver please check here.

Next we will modify our pipeline inside otel-collector-config.yaml to include the receiver we have created above.

service:
    ....
    logs:
        receivers: [otlp, syslog]
        processors: [batch]
        exporters: [clickhouselogsexporter]

Now we can restart the otel collector container so that new changes are applied and we can forward our logs to port 54527.

Modify your rsyslog.conf file present inside /etc/ by running sudo vim /etc/rsyslog.conf and adding the this line at the end

template(
  name="UTCTraditionalForwardFormat"
  type="string"
  string="<%PRI%>%TIMESTAMP:::date-utc% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"
)

*.* action(type="omfwd" target="0.0.0.0" port="54527" protocol="tcp" template="UTCTraditionalForwardFormat")

For production use cases it is recommended to using something like

template(
  name="UTCTraditionalForwardFormat"
  type="string"
  string="<%PRI%>%TIMESTAMP:::date-utc% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"
)

*.*  action(type="omfwd" target="0.0.0.0" port="54527" protocol="tcp"
        action.resumeRetryCount="10"
        queue.type="linkedList" queue.size="10000" template="UTCTraditionalForwardFormat")

So that you have retires and queue in place to de-couple the sending from the other logging action.

The value of target might vary depending on where SigNoz is deployed, since it is deployed on the same host I am using 0.0.0.0 for more help you can visit here

Now restart your rsyslog service by running sudo systemctl restart rsyslog.service
You can check the status of service by running sudo systemctl status rsyslog.service
If there are no errors your logs will be visible on SigNoz UI.

Collecting Syslogs

Collect Syslogs in SigNoz cloud​

Collect Syslogs in Self-Hosted SigNoz​

Collect Syslogs in SigNoz cloud

Collect Syslogs in Self-Hosted SigNoz