Using Rsyslog to forward Elastic Load balancer(ELB) logs in AWS

The ELB logs provides the great insights about the traffic being received by your application. You can identify the location, requests, errors and attacks by analyzing the ELB logs. Your  security team might be interested in analyzing these logs.

The problem is the logs are written either in 1 hour or every 5 minutes. You can also set them at  a definite size of 5MB. If you choose 1 hour than the size of the file  would be big. So it makes sense that logs are written at every 5 minutes since you want to analyze current requests coming on the ELB.

The problem in setting Rsyslog is the AWS logs are generated dyamica pattern and date yyyy/mm/dd

keep on rotating. Other problem is everytime a new log file is generated and thirdly logs are written in the S3 bucket which is a storage only and have very low computing power.

We used the S3fs to mount the S3 as a mount on the server this provided the easy access to logs on the s3. The other problem was all multiple application logs were written in  a single directory. We wanted to process multiple application logs separately for which we have used the rsync command to sync the logs in a separate directory. The advantage of using rsync is we won't have to process the same  log again and again it only takes the latest log and  does not copy log which is already present.

We generated the rsync log which provides the path of file being sync to the directory. So we directly using the cat to read the file content and append it in another file. So this way all the latest log file created by elb gets appended to a single file which can be easily be copied on remote server using the rsyslog or can be directly push to logstash if you are using the ELK setup or to your security team after which they can get the logs to there software for processing.

#####Script to get ELB logs written in single file#####
#####Created By Ankit Mittal######

#!/bin/bash  

Source="/path-to-elb-logs/ELB/AWSLogs/912198634563/elasticloadbalancing/ap-southeast-1"

year=`date +%Y`

month=`date +%m`

date=`date +%d`

path="$Source/$year/$month/$date"

process_check=`ps -ef | grep rsync | wc -l`

lb_sync() {

logname="$1"

echo "------------Sync started at `date`------------- " >> /var/log/rsync-elb.log

rsync -avz --files-from=<(ls $path/*$logname* | cut -d / -f11) $path/ /var/log/$logname/

}

if [ $process_check -gt 1 ]

then

echo "Rsync process already running"

else

lb_sync application1-name >> /tmp/app1  ### application1 name coming in elb pattern through which can grep the name of the file of that application

lb_sync application2-name >> /tmp/app2      

fi

merge_log() {

listname="$1"

echo "-----------Started merging of logs `date`--------------" >> /var/log/merging-elblogs.log

for i in `cat /tmp/$listname | grep $listname`;do cat $path/$i >> /var/log/"$listname".log;done >> /var/log/merge-loop-output.log

}

merge_log application1

merge_log application2

Next you need to make an entry in the rsyslog.conf file in the /etc directory so as to forward the logs on the remote server

# ELB Logs forwarding  

$ModLoad imfile

$InputFileName /var/log/application1.log

$InputFileTag application1_ELB:

$InputFileStateFile application1_ELB

$InputFileFacility local3

$InputRunFileMonitor

$ModLoad imfile

$InputFileName /var/log/application2.log

$InputFileTag application2_ELB:

$InputFileStateFile application2_ELB

$InputFileFacility local3

$InputRunFileMonitor

local3.*    @10.142.1.75:514