Managing Logs With Graylog & Fluentd

Managing Logs With Graylog & Fluentd

Think about searching logs in multiple servers, spent time and effort to find application error. Now Log Management comes to the picture to save the time for this.Log Management is the approach to deal with large volumes of data from multiple servers and it makes life simple for Developers and DevOps during debugging code, troubleshooting, security analysis.

In the IT industry, there are many tools for Log Management. Here I have used Graylog in my organization and write the installation stapes.

GrayLog: – A Supercharge Open Source Log Management. It aggregates and extracts important data from server logs, which are often sent using the Syslog protocol. It also allows you to search logs and build dashboards in the web interface.

A Simple Diagram for single node Graylog Server is as below.

In This Tutorial, I have used Ubuntu 16.04 for Graylog setup. mongo, elasticsearch and java is required for graylog.follow the stapes

 

Prerequisites

 $ sudo apt-get update
 $ sudo apt-get install apt-transport-https openjdk-8-jre-headless uuid-runtime pwgen

MongoDB

 $ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2930ADAE8CAF5059EE73BB4B58712A2291FA4AD5
 $ echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.6 multiverse" | sudo tee 
       /etc/apt/sources.list.d/mongodb-org-3.6.list
 $ sudo apt-get update
 $ sudo apt-get install -y mongodb-org

enable MongoDB during the operating system’s startup:

 $ sudo systemctl daemon-reload
 $ sudo systemctl enable mongod.service
 $ sudo systemctl restart mongod.service

Elasticsearch

 $ sudo apt-get install default-jre
 $ sudo apt-get install default-jdk
 $ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
 $ echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
 $ sudo apt-get update && sudo apt-get install elasticsearch

Set the cluster name for elasticsearch in the configuration file, open the elasticsearch yml file and set in “Cluster” section

 $ sudo vim /etc/elasticsearch/elasticsearch.yml

ex : – cluster.name: graylog

enable elasticsearch during the operating system’s startup.

 $ sudo systemctl daemon-reload
 $ sudo systemctl enable elasticsearch.service
 $ sudo systemctl restart elasticsearch.service

Graylog

 $ wget https://packages.graylog2.org/repo/packages/graylog-2.5-repository_latest.deb
 $ sudo dpkg -i graylog-2.5-repository_latest.deb
 $ sudo apt-get update && sudo apt-get install graylog-server

Configuring Graylog

Now we will configure graylog server conf file to enable web interface and authentication.

Generate Administrator password using pwgen

 $ sudo -E sed -i -e "s/password_secret =.*/password_secret = $(pwgen -s 128 1)/" /etc/graylog/server/server.conf 
 $ sudo sed -i -e "s/root_password_sha2 =.*/root_password_sha2 = $(echo -n 'password' | shasum -a 256 | cut -d' ' -f1)/" /etc/graylog/server/server.conf

Note: – change “password” with your password that you want to set for user: – admin (default user)

There is some configuration needs to access graylog from the web.

 $ sudo vim /etc/graylog/server/server.conf

Leave everything as default, change as I have mentioned below

rest_listen_uri = http://127.0.0.1:9000/api/
rest_transport_uri = http://127.0.0.1:9000/api/
web_endpoint_uri = http://logs.example.in
web_enable = true
web_listen_uri = http://127.0.0.1:9000/

web_endpoint_uri: – enter your url name from which you want to access graylog web interface.

After the modification, restart graylog-server service:

 $ sudo systemctl restart graylog-server

Now Configure Nginx to access graylog web interface with HTTP

 $ sudo apt-get install nginx

delete default nginx file and create new one as below and paste the content

 $ sudo vim /etc/nginx/sites-enabled/gray.conf

server
{
    listen 80 default_server;
    listen [::]:80 default_server ipv6only=on;
    server_name logs.example.in;

    location / {
      proxy_set_header Host $http_host;
      proxy_set_header X-Forwarded-Host $host;
      proxy_set_header X-Forwarded-Server $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Graylog-Server-URL http://$server_name/api;
      proxy_pass       http://127.0.0.1:9000;
    }
}

Now Restart nginx

 $ sudo nginx -t
 $ sudo service nginx restart 

Please verify mongo,elasticsearch,graylog,nginx is running
 $ sudo netstat -tunlp

ports shuold be look like this if you not chnaged the default port.
mongo : – 127.0.0.1:27017
elasticsearch : – 127.0.0.1:9000 , 127.0.0.1:9200, 27.0.0.1:9300
graylog : – 127.0.0.1:9000
nginx : – 0.0.0.0:80

 

Open the URL in chrome or Mozilla by:- http://logs.example.in

 

Login with user – admin and password that you have set.

Now you have Graylog Server ready.the next thing to ship logs to graylog server from an application running in a different server.i will cover in next blog : – “http://www.devinitiate.com/managing-logs-with-graylog-fluentd-2/

Leave a Reply

Your email address will not be published. Required fields are marked *