Website Availaibility with New Relic

NewRelic is a great tool to monitor the webapplications , see the insights of the various transactions occurring in the applications and track the code issues. It helps you identify the response times from the different geographical locations and helps you optimize the user experience by  reducing the load times for your applications.

If you just want to monitor the website availability you can use the ping monitor of the newrelic. Its simple and don't require any agents configuration upfront and the advantage is that though the traditional monitors monitor for the website http status i.e. 200 for the site loading but won't check if the content is actually loaded or not. This can sometimes cause an issue, while newrelic gives you a functionality where you can monitor the string on the page load and if the newrelic doesn't find that string it would raise an alarm.

You can configure the ping for a website as follows, in our example we have simply put the monitor for the google website you can replace it with your own.

Webserver/Appserver config backup with Git Script

In production environment there are always backup scheduled for the entire servers using the AMI or snapshots which take the system backup over a period of time and runs daily in the non peak hours.

But there are environments such as non-prod where many people have access to the config so that they can tune this according to there requirements. While this makes things to speed up but since no one is accountable for the changes this can simply go on breaking up the configuration due to multiple changes being performed.

Now you can always troubleshoot the changes but sometimes there are situation when the hardware or dns problems might arise which is outside your control. In those cases you can't keep your environment down since this would affect the testing in the lower environments.

AWS Cli installation from awscli.zip

You can install the AWS cli using the boto sdk as discussed in my previous post

You can also download the aws cli zip package and install the aws cli from it as

 curl "http://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"  

 unzip awscli-bundle.zip  

  sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws  

Adding a Slave Server to Jenkins

By default the jenkins initiates the builds from the jenkins master. But if you are using the jenkins in an production environment there are chances several builds for different components/ micro services would be required to triggered parallely.

In case the IO/Network/System performance for the master may degrade and UI or jenkins might start to give reduced performance , hangs up etc. which is not a desirable state. So its recommended to use the jenkins slave for the builds while master only control those nodes.

To add the jenkins slave you need to have the two server's first one is your master server and other one the slave server on which you are going to build the jobs created in the jenkins.

For this you require to have 2 servers in our case we are going to use 172.31.59.1 as the jenkins master and 172.31.62.152.

You need to install the java jdk on both the servers. For installing java jdk and setting up the environment variables follow my previous posts Java JDK Installation and jenkins installation.

Ensure the jenkins user exist on both the  servers and generate a key on the jenkins master server using the ssh-keygen command and copy this key using the ssh-copy-id command on both the servers to make them passswordless from the jenkins master server.

Once done go to the jenkins dashboard

Proxy server instance id instance of ec2-instance

If you are using the awscli to write your scripts than while checking the ec2 instance id from the instance you might be using the following command

 wget -q -O - http://169.254.169.254/latest/meta-data/instance-id  

The problem is if you are having your server on private subnet and using a proxy to connect to internet and if you use the above command it would give you the proxy server id instead of your instance-id. To overcome this issue use the following while writing your script.

 export NO_PROXY=169.254.169.254;  

The resolves the ip 169.254.169.254 from the instance and does not forward it to the proxy and you would get the instance id of the instance rather than the proxy server.

Resolved Error TCP segment of a reassembled PDU

Error code as noted in Wireshark:-
106   8.721506 0.000024 TCP 172.XX.XXX.XXX -> 172.XX.XX.XXX 368 [TCP segment of a reassembled PDU] 106

Problem statement:- Behind the ELB we were using the HAproxy and sending an options request in which the original request status was replaced by 200 status using the cors configuration. While HAproxy received the request from the ELB and responed back with 200 status ELB was not able to respond back and connection was terminated.

Resolution:- After recording the tcpdump and capturing the packets using the pcap file generated and analyzed via the wireshark we noticed the packed 106 was a [TCP segment of a reassembled PDU]. Actually the HTTP Packet is not complete, so the Wireshark is also unable to see the packet as an HTTP valid one, this is the same behavior as the ELB have.

According to the RFC-2616, section-6  After receiving and interpreting a request message, a server responds with an HTTP response message. [2]

       Response      = Status-Line               ; Section 6.1
                       *(( general-header        ; Section 4.5
                        | response-header        ; Section 6.2
                        | entity-header ) CRLF)  ; Section 7.1
                       CRLF
                       [ message-body ]          ; Section 7.2

So after the HEADER Section, it's required a CRLF (Carriage Return  + Line Feed) to complete the HEADER Section.

In our case the this was missing.

ELB needs the full request to understand that the request has been completed, so it's mandatory to be fully compliant with the RFC-2616.

In order to fix the issue, we have to add a CRLF after the Content-Lenght: 0 in the end of the file

This can be done by doing this:

# echo >> /directory/file.http

Then you will see that the file is on Unix format, Unix format does not use CRLF terminators:

# file /directory/file.http
/directory/file.http: ASCII text

So the file needs to be converted, in order to do that there is a tool called unix2dos, on Red Hat it can be installed by issuing this command:

# yum install unix2dos -y

then to convert the file:

# unix2dos /directory/file.http
unix2dos: converting file /directory/file.http to DOS format ...

You will see that the the file now will have CRLF line terminators:

# file /directory/file.http
/directory/file.http: ASCII text, with CRLF line terminators

After doing this we needed to restart HA-Proxy to use this new file that rewrites the http status from 503 to 200.

You can check the last line as

# cat -A /directory/file.http

last line should be blank (with CRLF which is the ^M$)

user-id^M$
Content-Length: 0^M$
^M$

Using pcap to analyze the network packets and troubleshooting web applications

If you are facing the problems with the webrequest and getting an error status. Than you can monitor the responses on the server side and client side by monitoring the packets sent over the network. You can use the tcpdump for it. If you want to have a greater insight into whats happening over your network than you need to capture the packets and analyze it using the network packet analyzer tool such as wireshark.

Else you can use the tcpdump also to view the packets captured. The pcap file needs to generated which captures your packet over the network and is basically a binary file. You need to query that file using the tcpdump or wireshark to see whats happening in your network.

To generated the pcap file for monitization use the following command

 tcpdump -i eth0 -s 65535 -w request.pcap  

To analyze the pcap file use the following command

 tcpdump -qns 0 -X -r request.pcap  

you should see the time of the request, Ip from which request was received, your server ip, protocol tcp or udp, packets information.

Installing Botosdk on RHEL

In my previous post i covered the installation of the python pip which can be used for the Botosdk and awscli installation.

Follow these steps to install the botosdk and awscli on RHEL in Amazon AWS

 pip install boto3  

 pip install awscli

This completes the boto3  sdk and awscli installation.

Before you can use the command line you need to connect to the aws and authenticate. use the following command for this

 aws configure  

This would in-turn ask your Access key and Secret Access key which you can generate from the AWS IAM. Also you need to enter the region endpoint to ensure you are connecting to correct region in case you are using the multiple regions in AWS and also its a good practice to be followed.

Once done you should be able to connect to your AWS environment. 

Installing pip on Redhat Linux on Amazon AWS

The RHEL doesn't come with the Awscli or botosdk preinstalled unlike the Amazon Linux. To install the Awscli or python botosdk you need to install the python pip through which you can install the aws cli and boto sdk.

But the pip is not installed by default. Follow the following steps to install the python pip on RHEL 7 in Amazon Linux

 cd /tmp  

 yum install wget -y

  wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

 rpm -ivh epel-release-latest*

 yum install python-pip

This Will complete the installation of the python pip on the RHEL7

Checkout the version of the pip using following command

 [root@ip-xxx-xx-xx-xxx tmp]# pip -V  

 pip 7.1.0 from /usr/lib/python2.7/site-packages (python 2.7)