Deploying an EC2 Instance Using Terraform

You can use the following Terraform script to Deploy an Instance in your AWS Account. Terraform will create a t2.medium instance from the official RHEL7.2 AMI using the AMI ID within the specified subnet. And will create a 30GB root block device and a 10GB Ebs volume. The instance will use a predefined key and will add the specified tags to the Instance Being Launched.

 provider "aws" {  
  access_key = "AKXXXXXXXXXXXXXXXXX"  
  secret_key = "2YXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxx"  
  region   = "ap-south-1"  
 }  
 resource "aws_instance" "instance_name" {  
  ami = "ami-cdbdd7a2"  
  count = 1  
  instance_type = "t2.medium"  
  security_groups = ["sg-f70674re"]  
  subnet_id = "subnet-526bcb6d"  
  root_block_device = {  
   volume_type = "standard"  
   volume_size = "30"  
  }   
  ebs_block_device = {  
   device_name = "/dev/sdm"  
   volume_type = "gp2"  
   volume_size = "10"  
  }  
  source_dest_check = true  
  key_name = "Keyname"  
  tags {  
   Name = "tagname"  
  }  
 }  

Custom Cloudwatch Alarm Configuration Part-8

As discussed in the previous post regarding the alarm plugins those plugins are used to push the metrics data to the cloudwatch using the cron running every minute or 5minutes depending upon your requirements.

Next we have to create the alarms in the cloudwatch on the above metrics which works on the logic that if the metrics crosses the threshold value than an event is triggered which could be like send a mail through sns alerting that the value has crossed the threshold and if it agains comes below threshold than it state is changed from alarm to ok which is more like a recovery.

But unlike from the console we are going to trigger this programmatically using the AWS CLI provided by the AWS. The script works sequentially and uses the array which runs in a loop and all the relevant alarms are created.

The most important thing to be considered here is the name of the alarm which is to be created in the cloudwatch. Now you can put any name but the name based on programmatic assumptions following a meaningful pattern should be used so that you are able to easily identify the environment, application, alarm type, service affected is easily convened. And the team receiving can immediately work towards its resolution.

Custom Cloudwatch Plugins CW_tcpConnections Part-7

The following Cloudwatch plugin can be used to determine the established tcp connections.

 #!/bin/bash
#
#  About                : Check TCP connections
#
#  Name                 : cw_tcpconnection.sh

DIR=$(dirname $0);
PLUGIN_NAME='cw_tcpconnection';

# Include configuration file
source ${DIR}/../conf/plugin.conf;


#Get Current Instance ID
INSTANCE_ID=(`wget -q -O - http://169.254.169.254/latest/meta-data/instance-id`);
#Get Hostname
HOST_ID=(`wget -q -O - http://169.254.169.254/latest/meta-data/hostname`);

# Help
usage() {
        echo "Usage: $0 [-n ] [-d ] [-m ] [-h ] [-p ]" 1>&2;
        exit 1;
}

# Logger
logger(){

 SEVERITY=$1;
 MESSAGE=$2;
 DATE=`date +"[%Y-%b-%d %H:%M:%S.%3N]"`;

 echo -e "${DATE} [${SEVERITY}] [${PLUGIN_NAME}] [${INSTANCE_ID}] [${HOST_ID}] ${MESSAGE}" >> ${DIR}/../logs/appcwmon.log;

}

# Process Arguments

if [ $# -eq 0 ]; then
        # When no argument is passed
        logger ERROR "Invalid arguments passed";
        usage;
fi



while getopts ":n:d:m:h:p:" o; do
    case "${o}" in
        n)
            NAMESPACE=${OPTARG}
            if [ -z "${NAMESPACE}" ]; then
                logger ERROR "Invalid Namespace passed";
                usage;
            fi
            ;;
        d)
            DIMENSION=${OPTARG};

            DNAME=${DIMENSION%=*};
            DVALUE=${DIMENSION#*=};

            if [ -z "${DIMENSION}" ] || [ -z "${DNAME}" ] || [ "${DNAME}" == "${DVALUE}" ]; then
                logger ERROR "Invalid dimension passed <${DIMENSION}>";
                usage;
            fi

            # If Dimension name is 'InstanceId' then Value is not required to be passed
            if [ "${DNAME}" != 'InstanceId' ] && [ -z "${DVALUE}" ]; then
                logger ERROR "Invalid dimension passed <${DIMENSION}>";
                usage;
            fi
            ;;
        m)
            METRICS=${OPTARG}
            if [ -z "${METRICS}" ]; then
                logger ERROR "Invalid metrices passed <${METRICS}>";
                usage;
            fi
            ;;
        h)
            HOST=${OPTARG}
            if [ -z "${HOST}" ]; then
                logger ERROR "Invalid hostname passed <${HOST}>";
                usage;
            fi
            ;;
        p)
            PORT=${OPTARG}
            if [ -z "${PORT}" ]; then
                logger ERROR "Invalid port passed <${PORT}>";
                usage;
            fi
            ;;

        *)
            usage
            ;;
    esac
done
shift $((OPTIND-1))

# Input Validation
if [ -z "${NAMESPACE}" ] || [ -z "${DNAME}" ] || [ -z "$METRICS" ] || [ -z "$HOST" ] || [ -z "$PORT" ]; then
                logger ERROR "Invalid argument passed";
    usage
fi


##########################################################
##########################################################


# If "INSTANCE_ID" is passed as Dimension, then use actual AWS Instanec ID as Dimension
if [ "${DNAME}" == "InstanceId" ]; then
        DVALUE=${INSTANCE_ID};
fi


UNIT="Count";

&1); if [ "$?" -ne "0" ]; then logger ERROR "${NAMESPACE} ${METRICS} ${DNAME}=${DVALUE} | value=${VALUE} unit=${UNIT} $HOST $PORT | ${OUTPUT}"; exit 1; fi; logger INFO "${NAMESPACE} ${METRICS} ${DNAME}=${DVALUE} | value=${VALUE} unit=${UNIT} $HOST $PORT"; # Success exit 0;

Custom Cloudwatch Plugins CW_Rabbitmq Queue Message Length Part-6

The following Cloudwatch plugin helps to measure the number of messages in the Rabbitmq unack,Ready,Total Message on which alarms can be configured later using the cloudwatch API.

#!/bin/bash
#
#  About                : Check RabbitMQ Queue Message Length
#
#  Name                 : cw_rabbitmq.sh



DIR=$(dirname $0);
PLUGIN_NAME='cw_rabbitmq';

# Include configuration file
source ${DIR}/../conf/plugin.conf;

#Get Current Instance ID
INSTANCE_ID=(`wget -q -O - http://169.254.169.254/latest/meta-data/instance-id`);
#Get Hostname
HOST_ID=(`wget -q -O - http://169.254.169.254/latest/meta-data/hostname`);

# Help
usage() {
        echo "Usage: $0 [-n ] [-d ] [-m ] [-u ] [-p ] [-q ]" 1>&2;
        exit 1;
}

# Logger
logger(){

 SEVERITY=$1;
 MESSAGE=$2;
 DATE=`date +"[%Y-%b-%d %H:%M:%S.%3N]"`;

 echo -e "${DATE} [${SEVERITY}] [${PLUGIN_NAME}] [${INSTANCE_ID}] [${HOST_ID}] ${MESSAGE}" >> ${DIR}/../logs/appcwmon.log;

}

# Process Arguments

if [ $# -eq 0 ]; then
        # When no argument is passed
        logger ERROR "Invalid arguments passed";
        usage;
fi



while getopts ":n:d:m:u:p:q:" o; do
    case "${o}" in
        n)
            NAMESPACE=${OPTARG}
            if [ -z "${NAMESPACE}" ]; then
                logger ERROR "Invalid Namespace passed";
                usage;
            fi
            ;;
        d)
            DIMENSION=${OPTARG};

            DNAME=${DIMENSION%=*};
            DVALUE=${DIMENSION#*=};

            if [ -z "${DIMENSION}" ] || [ -z "${DNAME}" ] || [ "${DNAME}" == "${DVALUE}" ]; then
                logger ERROR "Invalid dimension passed <${DIMENSION}>";
                usage;
            fi

            # If Dimension name is 'InstanceId' then Value is not required to be passed
            if [ "${DNAME}" != 'InstanceId' ] && [ -z "${DVALUE}" ]; then
                logger ERROR "Invalid dimension passed <${DIMENSION}>";
                usage;
            fi
            ;;
        m)
            METRICS=${OPTARG}
            if [ -z "${METRICS}" ]; then
                logger ERROR "Invalid metrices passed <${METRICS}>";
                usage;
            fi
            ;;
        u)
            USERNAME=${OPTARG}
            if [ -z "${USERNAME}" ]; then
                logger ERROR "Invalid username passed <${USERNAME}>";
                usage;
            fi
            ;;
        p)
            PASSWORD=${OPTARG}
            if [ -z "${PASSWORD}" ]; then
                logger ERROR "Invalid password passed <${PASSWORD}>";
                usage;
            fi
            ;;
        q)
            QUEUE=${OPTARG}
            if [ -z "${QUEUE}" ]; then
                logger ERROR "Invalid queue passed <${QUEUE}>";
                usage;
            fi
            ;;


        *)
            usage
            ;;
    esac
done
shift $((OPTIND-1))

# Input Validation
if [ -z "${NAMESPACE}" ] || [ -z "${DNAME}" ] || [ -z "$METRICS" ] || [ -z "$USERNAME" ] || [ -z "$PASSWORD" ] || [ -z "$QUEUE" ]; then
                logger ERROR "Invalid argument passed";
    usage
fi


##########################################################
##########################################################


# If "INSTANCE_ID" is passed as Dimension, then use actual AWS Instanec ID as Dimension
if [ "${DNAME}" == "InstanceId" ]; then
        DVALUE=${INSTANCE_ID};
fi


UNIT="Count";

## Total Message
TOTAL_MESSAGE=$(/usr/local/bin/rabbitmqadmin --username=${USERNAME} --password="${PASSWORD}" list queues name messages | grep -w "${QUEUE} " | awk '{print $4}' 2>&1);

if [ "$?" -ne "0" ]; then
        logger ERROR "${NAMESPACE} ${METRICS} ${DNAME}=${DVALUE} | ${QUEUE} value=NULL unit=${UNIT} | ${TOTAL_MESSAGE}";
        exit 1;
fi;

OUTPUT=$(/usr/local/bin/aws cloudwatch put-metric-data --namespace ${NAMESPACE} --metric-name Total-${METRICS} --dimensions ${DNAME}=${DVALUE} --value ${TOTAL_MESSAGE} --unit ${UNIT} 2>&1);

if [ "$?" -ne "0" ]; then
        logger ERROR "${NAMESPACE} ${METRICS} ${DNAME}=${DVALUE} | ${QUEUE} value=${TOTAL_MESSAGE} unit=${UNIT} | ${OUTPUT}";
        exit 1;
fi;

## Unacknowledge Message
UNACK_MESSAGE=$(/usr/local/bin/rabbitmqadmin --username=${USERNAME} --password="${PASSWORD}" list queues name messages_unacknowledged | grep -w "${QUEUE} " | awk '{print $4}' 2>&1);
if [ "$?" -ne "0" ]; then
        logger ERROR "${NAMESPACE} ${METRICS} ${DNAME}=${DVALUE} | ${QUEUE} value=NULL unit=${UNIT} | ${UNACK_MESSAGE}";
        exit 1;
fi;

OUTPUT=$(/usr/local/bin/aws cloudwatch put-metric-data --namespace ${NAMESPACE} --metric-name UnACK-${METRICS} --dimensions ${DNAME}=${DVALUE} --value ${UNACK_MESSAGE} --unit ${UNIT} 2>&1);

if [ "$?" -ne "0" ]; then
        logger ERROR "${NAMESPACE} ${METRICS} ${DNAME}=${DVALUE} | ${QUEUE} value=${UNACK_MESSAGE} unit=${UNIT} | ${OUTPUT}";
        exit 1;
fi;

## Ready Message
READY_MESSAGE=$(/usr/local/bin/rabbitmqadmin --username=${USERNAME} --password="${PASSWORD}" list queues name messages_ready | grep -w "${QUEUE} " | awk '{print $4}' 2>&1);
if [ "$?" -ne "0" ]; then
        logger ERROR "${NAMESPACE} ${METRICS} ${DNAME}=${DVALUE} | ${QUEUE} value=NULL unit=${UNIT} | ${READY_MESSAGE}";
        exit 1;
fi;

OUTPUT=$(/usr/local/bin/aws cloudwatch put-metric-data --namespace ${NAMESPACE} --metric-name Ready-${METRICS} --dimensions ${DNAME}=${DVALUE} --value ${READY_MESSAGE} --unit ${UNIT} 2>&1);

if [ "$?" -ne "0" ]; then
        logger ERROR "${NAMESPACE} ${METRICS} ${DNAME}=${DVALUE} | ${QUEUE} value=${READY_MESSAGE} unit=${UNIT} | ${OUTPUT}";
        exit 1;
fi;

## Consumer Count
CONSUMER=$(/usr/local/bin/rabbitmqadmin --username=${USERNAME} --password="${PASSWORD}" list queues name consumers | grep -w "${QUEUE} " | awk '{print $4}' 2>&1);
if [ "$?" -ne "0" ]; then
        logger ERROR "${NAMESPACE} ${METRICS} ${DNAME}=${DVALUE} | ${QUEUE} value=NULL unit=${UNIT} | ${CONSUMER}";
        exit 1;
fi;

OUTPUT=$(/usr/local/bin/aws cloudwatch put-metric-data --namespace ${NAMESPACE} --metric-name ConsumerCount-${METRICS} --dimensions ${DNAME}=${DVALUE} --value ${CONSUMER} --unit ${UNIT} 2>&1);

if [ "$?" -ne "0" ]; then
        logger ERROR "${NAMESPACE} ${METRICS} ${DNAME}=${DVALUE} | ${QUEUE} value=${CONSUMER} unit=${UNIT} | ${OUTPUT}";
        exit 1;
fi;

logger INFO "${NAMESPACE} ${METRICS} ${DNAME}=${DVALUE} | ${QUEUE} value=${TOTAL_MESSAGE},${UNACK_MESSAGE},${READY_MESSAGE},${CONSUMER} unit=${UNIT}";
# Success
exit 0;

   

Custom Cloudwatch Plugins CW_ProcessCount Part-5

You can monitor the number of process running for a service to determine whether the service is running or not on the Server using the following cloudwatch plugin.

 #!/bin/bash
#
#  About                : Check Process Running Status
#
#  Name                 : cw_process.sh

DIR=$(dirname $0);
PLUGIN_NAME='cw_process';

# Include configuration file
source ${DIR}/../conf/plugin.conf;


#Get Current Instance ID
INSTANCE_ID=(`wget -q -O - http://169.254.169.254/latest/meta-data/instance-id`);
#Get Hostname
HOST_ID=(`wget -q -O - http://169.254.169.254/latest/meta-data/hostname`);

# Help
usage() {
        echo "Usage: $0 [-n ] [-d ] [-m ] [-p ]" 1>&2;
        exit 1;
}

# Logger
logger(){

 SEVERITY=$1;
 MESSAGE=$2;
 DATE=`date +"[%Y-%b-%d %H:%M:%S.%3N]"`;

 echo -e "${DATE} [${SEVERITY}] [${PLUGIN_NAME}] [${INSTANCE_ID}] [${HOST_ID}] ${MESSAGE}" >> ${DIR}/../logs/appcwmon.log;

}

# Process Arguments

if [ $# -eq 0 ]; then
        # When no argument is passed
        logger ERROR "Invalid arguments passed";
        usage;
fi



while getopts ":n:d:m:p:" o; do
    case "${o}" in
        n)
            NAMESPACE=${OPTARG}
            if [ -z "${NAMESPACE}" ]; then
                logger ERROR "Invalid Namespace passed";
                usage;
            fi
            ;;
        d)
            DIMENSION=${OPTARG};

            DNAME=${DIMENSION%=*};
            DVALUE=${DIMENSION#*=};

            if [ -z "${DIMENSION}" ] || [ -z "${DNAME}" ] || [ "${DNAME}" == "${DVALUE}" ]; then
                logger ERROR "Invalid dimension passed <${DIMENSION}>";
                usage;
            fi

            # If Dimension name is 'InstanceId' then Value is not required to be passed
            if [ "${DNAME}" != 'InstanceId' ] && [ -z "${DVALUE}" ]; then
                logger ERROR "Invalid dimension passed <${DIMENSION}>";
                usage;
            fi
            ;;
        m)
            METRICS=${OPTARG}
            if [ -z "${METRICS}" ]; then
                logger ERROR "Invalid metrices passed <${METRICS}>";
                usage;
            fi
            ;;
        p)
            PROCESS=${OPTARG}
            if [ -z "${PROCESS}" ]; then
                logger ERROR "Invalid process passed <${PROCESS}>";
                usage;
            fi
            ;;
        *)
            usage
            ;;
    esac
done
shift $((OPTIND-1))

# Input Validation
if [ -z "${NAMESPACE}" ] || [ -z "${DNAME}" ] || [ -z "$METRICS" ] || [ -z "$PROCESS" ]; then
                logger ERROR "Invalid argument passed";
    usage
fi


##########################################################
##########################################################


# If "INSTANCE_ID" is passed as Dimension, then use actual AWS Instanec ID as Dimension
if [ "${DNAME}" == "InstanceId" ]; then
        DVALUE=${INSTANCE_ID};
fi


UNIT="Count";

VALUE=$(ps aux | grep "${PROCESS}" | grep -v cw_process.sh | grep -vc grep 2>&1);

OUTPUT=$(/usr/local/bin/aws cloudwatch put-metric-data --namespace ${NAMESPACE} --metric-name ${METRICS} --dimensions ${DNAME}=${DVALUE} --value ${VALUE} --unit ${UNIT} 2>&1);

if [ "$?" -ne "0" ]; then
        logger ERROR "${NAMESPACE} ${METRICS} ${DNAME}=${DVALUE} | value=${VALUE} unit=${UNIT} | ${OUTPUT}";
        exit 1;
fi;

logger INFO "${NAMESPACE} ${METRICS} ${DNAME}=${DVALUE} | value=${VALUE} unit=${UNIT}";
# Success
exit 0;

   

Custom Cloudwatch Plugins CW_Netconnection Part-4

Cloudwatch can be used to monitor the established connection to the vm. This helps in tracking connections in case your application is network intensive

#!/bin/bash
#
#  About                : Check Local and Foreign Network Connections
#
#  Name                 : cw_netconnection.sh

DIR=$(dirname $0);
PLUGIN_NAME='cw_netconnection';

# Include configuration file
source ${DIR}/../conf/plugin.conf;


#Get Current Instance ID
INSTANCE_ID=(`wget -q -O - http://169.254.169.254/latest/meta-data/instance-id`);
#Get Hostname
HOST_ID=(`wget -q -O - http://169.254.169.254/latest/meta-data/hostname`);

# Help
usage() {
        echo "Usage: $0 [-n ] [-d ] [-m ] [-s ] -t [ LOCAL | FOREIGN ] -p " 1>&2;
        exit 1;
}

# Logger
logger(){

 SEVERITY=$1;
 MESSAGE=$2;
 DATE=`date +"[%Y-%b-%d %H:%M:%S.%3N]"`;

 echo -e "${DATE} [${SEVERITY}] [${PLUGIN_NAME}] [${INSTANCE_ID}] [${HOST_ID}] ${MESSAGE}" >> ${DIR}/../logs/appcwmon.log;

}

# Process Arguments

if [ $# -eq 0 ]; then
        # When no argument is passed
        logger ERROR "Invalid arguments passed";
        usage;
fi



while getopts ":n:d:m:p:s:t:" o; do
    case "${o}" in
        n)
            NAMESPACE=${OPTARG}
            if [ -z "${NAMESPACE}" ]; then
                logger ERROR "Invalid Namespace passed";
                usage;
            fi
            ;;
        d)
            DIMENSION=${OPTARG};

            DNAME=${DIMENSION%=*};
            DVALUE=${DIMENSION#*=};

            if [ -z "${DIMENSION}" ] || [ -z "${DNAME}" ] || [ "${DNAME}" == "${DVALUE}" ]; then
                logger ERROR "Invalid dimension passed <${DIMENSION}>";
                usage;
            fi

            # If Dimension name is 'InstanceId' then Value is not required to be passed
            if [ "${DNAME}" != 'InstanceId' ] && [ -z "${DVALUE}" ]; then
                logger ERROR "Invalid dimension passed <${DIMENSION}>";
                usage;
            fi
            ;;
        m)
            METRICS=${OPTARG};
            if [ -z "${METRICS}" ]; then
                logger ERROR "Invalid metrices passed <${METRICS}>";
                usage;
            fi
            ;;
        s)
            STATE=${OPTARG}
            if [ "${STATE}" != "ESTABLISHED" ] && [ "${STATE}" != "LISTEN" ] && [ "${STATE}" != "TIME_WAIT" ]; then
                logger ERROR "Invalid connection state passed <${STATE}>";
                usage;
            fi
            ;;
        t)
            TYPE=${OPTARG}
            if [ "${TYPE}" != "LOCAL" ] && [ "${TYPE}" != "FOREIGN" ]; then
                logger ERROR "Invalid connection type passed <${TYPE}>";
                usage;
            fi
            ;;
        p)
            PORT=${OPTARG}
            if [ -z "${PORT}" ]; then
                logger ERROR "Invalid process passed <${PORT}>";
                usage;
            fi
            ;;
        *)
            usage
            ;;
    esac
done
shift $((OPTIND-1))

# Input Validation
if [ -z "${NAMESPACE}" ] || [ -z "${DNAME}" ] || [ -z "$METRICS" ] || [ -z "$PORT" ] || [ -z "${STATE}" ] || [ -z "${TYPE}" ]; then
                logger ERROR "Invalid argument passed";
    usage
fi


##########################################################
##########################################################


# If "INSTANCE_ID" is passed as Dimension, then use actual AWS Instanec ID as Dimension
if [ "${DNAME}" == "InstanceId" ]; then
        DVALUE=${INSTANCE_ID};
fi


UNIT="Count";

if [ "${TYPE}" == "LOCAL" ]; then
        VALUE=$(netstat -alntp | grep ${STATE} | grep -v grep | awk '{print $4}' | awk -F[:] '{print $2}' | grep -cw ${PORT} 2>&1);
else
        echo ${TYPE};
        VALUE=$(netstat -alntp | grep ${STATE} | grep -v grep | awk '{print $5}' | awk -F[:] '{print $2}' | grep -cw ${PORT} 2>&1);
fi;

if [ "$VALUE" -ne "$VALUE" ] 2>/dev/null; then
        logger ERROR "${NAMESPACE} ${METRICS} ${DNAME}=${DVALUE} | ${STATE} ${TYPE} ${PORT} | value=NULL unit=${UNIT} | ${VALUE}";
        exit 1;
fi;

OUTPUT=$(/usr/local/bin/aws cloudwatch put-metric-data --namespace ${NAMESPACE} --metric-name ${METRICS} --dimensions ${DNAME}=${DVALUE} --value ${VALUE} --unit ${UNIT} 2>&1);

if [ "$?" -ne "0" ]; then
        logger ERROR "${NAMESPACE} ${METRICS} ${DNAME}=${DVALUE} | ${STATE} ${TYPE} ${PORT} value=${VALUE} unit=${UNIT} | ${OUTPUT}";
        exit 1;
fi;

logger INFO "${NAMESPACE} ${METRICS} ${DNAME}=${DVALUE} | ${STATE} ${TYPE} ${PORT} value=${VALUE} unit=${UNIT}";
# Success
exit 0;

   

Custom Cloudwatch Plugins CW_MemoryUsage Part-3

The following plugin pushes the memory consumption of the vm to the cloudwatch which you can use to set the alarms and also can use for the autoscaling or taking actions when combined with the events.

#!/bin/bash
#
#  About                : Check used memory in percentage
#
#  Name                 : cw_memory.sh


DIR=$(dirname $0);
PLUGIN_NAME='cw_memory';

# Include configuration file
source ${DIR}/../conf/plugin.conf;


#Get Current Instance ID
INSTANCE_ID=(`wget -q -O - http://169.254.169.254/latest/meta-data/instance-id`);
#Get Hostname
HOST_ID=(`wget -q -O - http://169.254.169.254/latest/meta-data/hostname`);

# Help
usage() {
        echo "Usage: $0 [-n ] [-d ] [-m ] " 1>&2;
        exit 1;
}

# Logger
logger(){

 SEVERITY=$1;
 MESSAGE=$2;
 DATE=`date +"[%Y-%b-%d %H:%M:%S.%3N]"`;

 echo -e "${DATE} [${SEVERITY}] [${PLUGIN_NAME}] [${INSTANCE_ID}] [${HOST_ID}] ${MESSAGE}" >> ${DIR}/../logs/appcwmon.log;

}

# Process Arguments

if [ $# -eq 0 ]; then
        # When no argument is passed
        logger ERROR "Invalid arguments passed";
        usage;
fi


while getopts ":n:d:m:a:" o; do
    case "${o}" in
        n)
            NAMESPACE=${OPTARG}
            if [ -z "${NAMESPACE}" ]; then
                logger ERROR "Invalid Namespace passed";
                usage;
            fi
            ;;
        d)
            DIMENSION=${OPTARG};
            DNAME=${DIMENSION%=*};
            DVALUE=${DIMENSION#*=};

            if [ -z "${DIMENSION}" ] || [ -z "${DNAME}" ] || [ "${DNAME}" == "${DVALUE}" ]; then
                logger ERROR "Invalid dimension passed <${DIMENSION}>";
                usage;
            fi

            # If Dimension name is 'InstanceId' then Value is not required to be passed
            if [ "${DNAME}" != 'InstanceId' ] && [ -z "${DVALUE}" ]; then
                logger ERROR "Invalid dimension passed <${DIMENSION}>";
                usage;
            fi
            ;;
        m)
            METRICS=${OPTARG}
            if [ -z "${METRICS}" ]; then
                logger ERROR "Invalid metrics passed <${METRICS}>";
                usage;
            fi
            ;;
        *)
            usage
            ;;
    esac
done
shift $((OPTIND-1))

# Input Validation
if [ -z "${NAMESPACE}" ] || [ -z "${DNAME}" ] || [ -z "$METRICS" ]; then
    logger ERROR "Invalid arguments passed";
    usage
fi


##########################################################
##########################################################


# If "INSTANCE_ID" is passed as Dimension, then use actual AWS Instanec ID as Dimension
if [ "${DNAME}" == "InstanceId" ]; then
        DVALUE=${INSTANCE_ID};
fi

##########################

UNIT="Percent"

# Get Total Memory
#MEM_TOTAL=$(free -m | grep 'Mem' | awk '{print $2}' 2>&1);
MEM_TOTAL=$(awk '/^MemTotal/ {print $2}' /proc/meminfo 2>&1);

if [ "$?" -ne "0" ]; then
        logger ERROR "${NAMESPACE} ${METRICS} ${DNAME}=${DVALUE} | value=NULL | ${MEM_TOTAL}";
        exit 1;
fi;

MEM_FREE=$(awk '/^MemFree/ {print $2}' /proc/meminfo 2>&1);

if [ "$?" -ne "0" ]; then
        logger ERROR "${NAMESPACE} ${METRICS} ${DNAME}=${DVALUE} | value=NULL | ${MEM_FREE}";
        exit 1;
fi;

MEM_BUFFER=$(awk '/^Buffers/ {print $2}' /proc/meminfo 2>&1);

if [ "$?" -ne "0" ]; then
        logger ERROR "${NAMESPACE} ${METRICS} ${DNAME}=${DVALUE} | value=NULL | ${MEM_BUFFER}";
        exit 1;
fi;

MEM_CACHED=$(awk '/^Cached/ {print $2}' /proc/meminfo 2>&1);

if [ "$?" -ne "0" ]; then
        logger ERROR "${NAMESPACE} ${METRICS} ${DNAME}=${DVALUE} | value=NULL | ${MEM_CACHED}";
        exit 1;
fi;

# Memory Used (In Percentage)
let "VALUE=((MEM_TOTAL - (MEM_FREE + MEM_BUFFER + MEM_CACHED))*100)/MEM_TOTAL";

if [ "$?" -ne "0" ]; then
        logger ERROR "${NAMESPACE} ${METRICS} ${DNAME}=${DVALUE} | value=NULL | ${VALUE}";
        exit 1;
fi;

## Pushing Cloudwatch Metric data
OUTPUT=$(/usr/local/bin/aws cloudwatch put-metric-data --namespace ${NAMESPACE} --metric-name ${METRICS} --dimensions ${DNAME}=${DVALUE} --value ${VALUE} --unit ${UNIT} 2>&1);

if [ "$?" -ne "0" ]; then
        logger ERROR "${NAMESPACE} ${METRICS} ${DNAME}=${DVALUE} | value=${VALUE} unit=${UNIT} | ${OUTPUT}";
        exit 1;
fi;

logger INFO "${NAMESPACE} ${METRICS} ${DNAME}=${DVALUE} | value=${VALUE} unit=${UNIT}";

# Success
exit 0;

Custom Cloudwatch Plugins CW_DiskUsage Part-2

Below is the plugin for monitoring the diskusage of specific mount via the cloudwatch. This would go in the bin folder in the cloudwatch and you need to create a file name like cw_diskusage.sh with the following script

 #!/bin/bash
#
#  About                : Percent of Disk usage by Mount based on Mount name
#
#  Name                 : cw_diskuage.sh

DIR=$(dirname $0);
PLUGIN_NAME='cw_diskusage';

# Include configuration file
source ${DIR}/../conf/plugin.conf;


#Get Current Instance ID
INSTANCE_ID=(`wget -q -O - http://169.254.169.254/latest/meta-data/instance-id`);
#Get Hostname
HOST_ID=(`wget -q -O - http://169.254.169.254/latest/meta-data/hostname`);

# Help
usage() {
        echo "Usage: $0 [-n ] [-d ] [-m ] [-f Mount Point]" 1>&2;
        exit 1;
}

# Logger
logger(){

 SEVERITY=$1;
 MESSAGE=$2;
 DATE=`date +"[%Y-%b-%d %H:%M:%S.%3N]"`;

 echo -e "${DATE} [${SEVERITY}] [${PLUGIN_NAME}] [${INSTANCE_ID}] [${HOST_ID}] ${MESSAGE}" >> ${DIR}/../logs/appcwmon.log;

}

# Process Arguments

if [ $# -eq 0 ]; then
        # When no argument is passed
        logger ERROR "Invalid arguments passed";
        usage;
fi


while getopts ":n:d:m:f:" o; do
    case "${o}" in
        n)
            NAMESPACE=${OPTARG}
            if [ -z "${NAMESPACE}" ]; then
                logger ERROR "Invalid Namespace passed";
                usage;
            fi
            ;;
        d)
            DIMENSION=${OPTARG};

            DNAME=${DIMENSION%=*};
            DVALUE=${DIMENSION#*=};

            if [ -z "${DIMENSION}" ] || [ -z "${DNAME}" ] || [ "${DNAME}" == "${DVALUE}" ]; then
                logger ERROR "Invalid dimension passed <${DIMENSION}>";
                usage;
            fi

            # If Dimension name is 'InstanceId' then Value is not required to be passed
            if [ "${DNAME}" != 'InstanceId' ] && [ -z "${DVALUE}" ]; then
                logger ERROR "Invalid dimension passed <${DIMENSION}>";
                usage;
            fi
            ;;
        m)
            METRICS=${OPTARG}
            if [ -z "${METRICS}" ]; then
                logger ERROR "Invalid metric passed <${METRICS}>";
                usage;
            fi
            ;;
        f)
            MOUNT_POINT=${OPTARG}
            if [ -z "${MOUNT_POINT}" ]; then
                logger ERROR "Invalid mount point passed <${MOUNT_POINT}>";
                usage;
            fi
            ;;
        *)
            usage
            ;;
    esac
done
shift $((OPTIND-1))

# Input Validation
if [ -z "${NAMESPACE}" ] || [ -z "${DNAME}" ] || [ -z "$METRICS" ] || [ -z "$MOUNT_POINT" ]; then
    logger ERROR "Invalid arguments passed";
    usage
fi


##########################################################
##########################################################


# If "INSTANCE_ID" is passed as Dimension, then use actual AWS Instanec ID as Dimension
if [ "${DNAME}" == "InstanceId" ]; then
        DVALUE=${INSTANCE_ID};
fi


##########################################################

#Check if mount point is valid
if ! grep -qs ${MOUNT_POINT} /proc/mounts; then
    logger ERROR "Mount point <${MOUNT_POINT}> not found";
    exit 1;
fi;

VALUE=$(df "${MOUNT_POINT}" -m | sed -n 2p | awk '{print $5}' | cut -d% -f1 2>&1);
UNIT="Percent";

## Pushing Cloudwatch Metric data

OUTPUT=$(/usr/local/bin/aws cloudwatch put-metric-data --namespace ${NAMESPACE} --metric-name ${METRICS} --dimensions ${DNAME}=${DVALUE} --value ${VALUE} --unit ${UNIT} 2>&1);

if [ "$?" -ne "0" ]; then
        logger ERROR "${NAMESPACE} ${METRICS} ${DNAME}=${DVALUE} | value=${VALUE} unit=${UNIT} | ${OUTPUT}";
        exit 1;
fi;

logger INFO "${NAMESPACE} ${METRICS} ${DNAME}=${DVALUE} | value=${VALUE} unit=${UNIT}";
# Success
exit 0;

Custom Cloudwatch Plugins Part-1

The Cloudwatch is a hosted tool provided by the aws to monitor different resources in your Cloud Infrastructure. AWS provides you with various metrice(data) related to resources to determine its state on per minute basis which can be used to monitor and raise an alarm whenever a certain threshold is crossed. You can configure the cloudwatch with the SNS to send the notification once the state of the alarm changes.

Further you can configure the events and take any action on these alarms. The only limitation is that AWS provides you with certain metrices to monitor but there are times when you want to monitor the resources which are not provided by AWS. Like your services, established connections, processes, memory etc. For this you need to create your own custom cloudwatch metrics which you can push to the cloudwatch using the AWS Cli.

Once the metrice has been configured in the cloudwatch than you can put the alarms on these metrices. You need to push the metrice regularly using the scheduler(cron) so that the state of the alarm is ok , if its not having the relevant data of the metrice than its state would change to the insufficient and no alarm would be raised.

We are going to follow the following format while creating our custom cloudwatch plugins which would comprise of the metrice which provides the data to the cloudwatch about the state of the resource and configure the alarms on these metrices to form the  overall monitoring via the cloudwatch. You need to be familiar with bash scripting to use these plugins of the cloudwatch.

We are going to follow the following directory structure and would design our scripts to follow the following design patterns starting with the appcwplugins directory which will contain following directories.

1. bin:- Executable cloudwatch plugins will go in this directory, but won't be having any configuration part.
2. conf:- Alarm configuration with cloudwatch cli and Access/Secret Key for AWS CLI along with proxy details will be this directory.
3. extra:- log rotation for the cloudwatch alarms, start and stop service for the cloudwatch alarms would be in this directory.
4. logs:- Cloudwatch logs would be in this directory.
5. script:- A script to copy the application/system logs to s3 and a script to push the metrics to the cloudwatch which would be executed by cron.