Problem:- The network split has occured in the Rabbitmq cluster causing the cluster of node1, node2 and node3 divide in two. Also the erlang process count was continuously high and hitting the upper limit. Further on network split the main cluster node hang up.
Cause:- The network split and high erlang process count might have occured if the request are not equally split across different nodes rather application is using one server as its endpoint. Due to which the erlang process count was continuously high on the node and that node got hanged , even it was hard to restart the process again.
Resolution:-
1. As network split occurs you need to stop the rabbitmq across all the nodes using the following command.
service rabbitmq-server stop
