Pulling the Messages from Amazon SQS Queue to a file with python

The Amazon SQS Queue is a high throughput messaging Queue from Amazon AWS.

You can send any type of messages or logs to SQS and than use a consumer(Scripts) to pull those messages from  the SQS and take an action based on those Queue. One of the use cases can be like you can push all your ELB Logs to SQS and than from SQS you can send it anywhere including your Events Notifier(SIEM) tools, batch processing , automation etc.

The following Generalized python script will pull 10 messages at a time from SQS(polling period) provided by SQS and write in a file. The scripts pulls messages from SQS and writes them in a file. If you want to increase the number of messages you just need to run more number of processes. Like if you want to download 50 messages / minute than you just need to start 10 processes of your script and it will start downloading 50 messages/ minute. Kindly note python or sqs is not having any limitations in this case and you can increase it n number of process , however your base line operating system is the limiting factor in this case and would depend upon overall cpu available and I/O operations.

After downloading these logs you may analyze them , send it somewhere else through syslog or write your own scripts for automation.



We are going to consider the case scenarios where we are fetching the logs from the s3 bucket and consuming these logs from sqs on to a server. In our case we were forwarding these logs to the SIEM receiver. However you might use it according to your use case.

Requirements for this script is python2.7+ and libraries boto.sqs, pprint. os,sys , json, time . If some of the library is missing on your system than you can use the pip to install it. Simply use pip install followed by the library name and pip will install the missing library for you.

#!/usr/bin/env python
#
#  About                : Downloads the ELB log from the sqs(queuename) and forwards the log to the SIEM
#
#  Name                 : sqsconsumer.py
#  Author               : Ankit
#  Creation Date        : 09Jan2017
#  Last Updated On      : 09Jan2017
#  Last Updated By      : Ankit


from __future__ import print_function
import boto.sqs
import pprint
import os,sys
import json
import time
###Running the Script at every 1second
while 1:
###Establishing the connection with queue
        conn=boto.sqs.connect_to_region("ap-southeast-1",aws_access_key_id='AXXXXXXXXXXXXXX', aws_secret_access_key='oXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX')
        queue=conn.get_queue('queuename')

        old_stdout = sys.stdout
###Opening a file where sqs messages would be written
        log_file = open("/var/log/filename.log", "a")
        sys.stdout = log_file
        with open("/var/log/filename.log", "a") as f:
###Downloading the 10 messages at a time
                result_set = queue.get_messages(10)
                for p in result_set:
                        data = json.loads(p.get_body())
                        print(data)
                        queue.delete_message(p)
sys.stdout = old_stdout
###Closing the log file once the messages has been written
log_file.close()
###Running the script at every 1 second
time.sleep( 1 )