Removing a file in S3 with space in bucket name

If you have created a file in S3 with Name which is having a space than you won't be able to delete it from aws console. Also due to the space it won't take the complete name and will give 404 i.e. object not found error.

For e.g Removing a file name s3://bucket/Demo Demo won't work

You can remove it from the Command line as follows

 aws s3 rm s3://bucket/Demo\ Demo/  

Verifying all running instance types in Amazon AWS

If you have reserved your instances , than you should track which all instances are running in your environment. Like if you have reserved m4.large instance so you are always billed for them whether you spawn it up or not you are charged for it.

So if you suppose reserved 5 instances of m4.large than you should make sure that 5 instances of m4.large is running at all times. But this can be difficult from checking this information from the console , you can grep all the m4.large instances from the AWS CLI.

Use the following command to see all the instance types running in your environment.

 aws ec2 describe-instances | grep InstanceType | cut -d "\"" -f4 | sort | uniq -c  

Script for checking database connections and queries taking maximum time

If you want to monitor the database connections , you can use this script. The script checks for the  connections count every minute and sends an email whenever the connections crosses the specified threshold. Same can be done via some monitoring tool(Nagios) also.

But you can customise the script to check the queries running at the time your database connections were high. This will help you to find the rogue query which might be causing the issue and will present you the better results for the troubleshooting the problem.

 #!/bin/bash  
 date=`date`  
 #####check the number of connections established  
 mysql --user="db_user" --password='db_password' --host="rds-name.ap-southeast-1.rds.amazonaws.com" --execute="show status like 'Threads_connected'" > /tmp/db.log;  
 conn_count=`cat /tmp/db.log | grep -i threads | awk '{print $2}'`  
 #####Compare the established connections with the threshold in our case its 1500  
 if [ $conn_count -gt 1500 ]  
 then  
 #### Create a file with all the queries running at that time  
 mysql --user="db_user" --password='db_password' --host="rds-name.ap-southeast-1.rds.amazonaws.com" --execute="show full processlist \G" > /tmp/dbquery.txt  
 #### Create a file isolating the queries only in descending order of there execution time with the query taking most time to execute coming first  
 mysql --user="db_user" --password='db_password' --host="rds-name.ap-southeast-1.rds.amazonaws.com" --execute="show full processlist \G" | grep -v "row" | grep -v Id | grep -v User | grep -v Host | grep -v db | grep -v Command | sed 's/  Info://g' | sed 's/  Time://g' | sed 'N;s/\n/ /' | grep -v NULL > /tmp/maxtime-db-query.txt  
 #### Sent an Alert mail with connection count and attaching the Queries with the mail for troubleshooting and establing root cause  
 echo "High DB connections($conn_count) on Database_name at $date, DB Queries attached " | /bin/mail -v -s "Alert: High DB connections" -a /tmp/dbquery.txt -a /tmp/maxtime-db-query.txt [email protected]  
 fi  
 #### Remove all the temporary files generated during the script execution  
 rm -rf /tmp/db.log  
 rm -rf /tmp/dbquery.txt  
 rm -rf /tmp/maxtime-db-query.txt  

Script to Detect Dos attacks on a Webserver

While DOS attacks are very common on the Webserver , its easy to block the ips causing the Dos attacks. But the trickier part is to detect the Dos attacks as they happen. This would cause your webservers to load if a significant attack occurs but if you move towards the cloud implementation and your environment is under the autoscaling chances are new servers would get attached and absorb that attack. But this would significantly increase up your cost.

Whether you are running your environment in cloud , VMs or physical machine its always good to automate the detection of the Dos attacks as soon as it occurs. We are going to create a Bash Script to detect the DOS attacks as it happens.