Managing Logs with Graylog & Fluentd-2

Managing Logs with Graylog & Fluentd-2

In my previous blog “Managing Logs with Graylog & Fluentd” we have configured Graylog Server. Now we will learn how to send logs from application server to graylog server.

graylog server IP : – 192.168.1.2

application server IP : – 192.168.1.3

We are using fluentd as log forwarder.

Fluentd is an open source data collector written in Ruby which requires very few system resources and ship logs across multiple destinations.

Let’s start the magic of log collector

Login to the graylog server and create INPUT

select  GELF UDP  and click on Launch new input

 

Fill only as written  below and save

Node: – select your node

Title: – fill suitable name (ex – access-logs)

Install and Configure Fluentd

Login to application server – 192.168.1.3, I have used Ubuntu 14.4

download fluentd td-agent from website “https://www.fluentd.org/download”

Note:- Download td-agent as per your Linux Distribution

 $ sudo curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent3.sh | sh
 $ sudo /usr/sbin/td-agent-gem install gelf

Now configure the td-agent to send logs

 $sudo vim /etc/td-agent/td-agent.conf

add the below content to conf file

<source>
type tail
path /var/log/apache2/*.log
format none
pos_file /var/log/td-agent/app1.log.pos
tag apache-access
</source>

<match apache-access.**>
type copy
buffer_type memory
buffer_chunk_limit 256m
buffer_queue_limit 128
flush_interval 1s
disable_retry_limit false
retry_limit 17
retry_wait 1s
<store>
type gelf
host 192.168.1.2
port 12201
</store>
<store>
type stdout
</store>
</match>

 
in source field

type – tail: – it will send logs in real-time

path: – the path of logs which you want to send

pos_file: – the position of logs sent to graylog server

tag: – tag your logs

in match field

match apache-access.** : – it will match which u have tagged against apache logs

host: – graylog server IP

port: – gelf input port number generated during input creation.

 

Now restart td-agent

 sudo service td-agent restart

open the graylog server and go to inputs section

click on show received message. now you can search for logs.
graylog search help document : – “http://docs.graylog.org/en/1.0/pages/queries.html

 

 

Troubleshooting

if logs are not landing to graylog server then some have some tips to troubleshoot

  1. check td-agent logs on the application server
 $ tail -f  /var/log/td-agent/td-agent.log

you can see the current apache logs here if logs not printing here then provide the logs files permission to “td-agent” user

 

2. check your connectivity from the application server

make sure udp port (ex – 12201) is open in graylog server

 $ nc -uvz ip port

Leave a Reply

Your email address will not be published. Required fields are marked *