Enabling the JMX PORT in kafka

If you want to get the metrics for the monitoring of the kafka than you need to enable the JMX port for the kafka which is 9999 be default.

You need to configure this port in the kafka/bin/kafka-server-start.sh by exporting the JMX_PORT which can be used to get the metrics for the kafka. The same port is also used by the datadog agent for providing the metrices of the kafka cluster.

Just enter the following line in the kafka-server-start.sh

 export JMX_PORT=${JMX_PORT:-9999}  

  exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "$@"

Afterwards you will need to restart the kafka broker service to make this active.

Verify if the service is listening on the port 9999

# telnet localhost 9999
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

# netstat -tunlp | grep 9999
tcp        0      0 0.0.0.0:9999            0.0.0.0:*               LISTEN      20031/java

This confirms the jmx port was successfully configured.

Manually allocating shards when Elasticsearch cluster is red

If you have large number of shards  with replica sets with huge amount of data its possible that you  get the ES cluster as red. The ES cluster goes red due to the issues with the primary shards which gets unassigned now depending on the situation you can take number of steps to resolve this issue.

However as the last resort you might have to allocate the shard manually but its last recommendation best way it to figure out whats the issue with the cluster i.e. why its not assigning the shards.

As a pre step you need to set the replication off otherwise you would have comparatively higher number of unassigned shards and that might take lot of time so if you want to quickly recover its better to set the replicas to 0 and than you can allow them back at a later point in time

Adding logrotation in elasticsearch

Elasticsearch supports log rotation with built in functionality you just need to configure the log4j.properties for the same.

Just copy the below configuration file at the following location

vim /etc/elasticsearch/log4j2.properties

 +status = error  

 +
 +# log action execution errors for easier debugging
 +logger.action.name = org.elasticsearch.action
 +logger.action.level = debug
 +
 +appender.console.type = Console
 +appender.console.name = console
 +appender.console.layout.type = PatternLayout
 +appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %marker%m%n
 +
 +appender.rolling.type = RollingFile
 +appender.rolling.name = rolling
 +appender.rolling.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}.log
 +appender.rolling.layout.type = PatternLayout
 +appender.rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %marker%.-10000m%n
 +appender.rolling.filePattern = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}-%d{yyyy-MM-dd}.log.gz
 +appender.rolling.policies.type = Policies
 +appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
 +appender.rolling.policies.time.interval = 1
 +appender.rolling.policies.time.modulate = true
 +
 +appender.rolling.strategy.type = DefaultRolloverStrategy
 +appender.rolling.strategy.action.type = Delete
 +appender.rolling.strategy.action.basepath = ${sys:es.logs.base_path}
 +appender.rolling.strategy.action.condition.type = IfLastModified
 +appender.rolling.strategy.action.condition.age = 5D
 +appender.rolling.strategy.action.PathConditions.type = IfFileName
 +appender.rolling.strategy.action.PathConditions.glob = ${sys:es.logs.cluster_name}-*
 +
 +rootLogger.level = info
 +rootLogger.appenderRef.console.ref = console
 +rootLogger.appenderRef.rolling.ref = rolling
 +
 +appender.deprecation_rolling.type = RollingFile
 +appender.deprecation_rolling.name = deprecation_rolling
 +appender.deprecation_rolling.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_deprecation.log
 +appender.deprecation_rolling.layout.type = PatternLayout
 +appender.deprecation_rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %marker%.-10000m%n
 +appender.deprecation_rolling.filePattern = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_deprecation-%i.log.gz
 +appender.deprecation_rolling.policies.type = Policies
 +appender.deprecation_rolling.policies.size.type = SizeBasedTriggeringPolicy
 +appender.deprecation_rolling.policies.size.size = 1GB
 +appender.deprecation_rolling.strategy.type = DefaultRolloverStrategy
 +appender.deprecation_rolling.strategy.max = 4
 +
 +logger.deprecation.name = org.elasticsearch.deprecation
 +logger.deprecation.level = warn
 +logger.deprecation.appenderRef.deprecation_rolling.ref = deprecation_rolling
 +logger.deprecation.additivity = false
 +
 +appender.index_search_slowlog_rolling.type = RollingFile
 +appender.index_search_slowlog_rolling.name = index_search_slowlog_rolling
 +appender.index_search_slowlog_rolling.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_index_search_slowlog.log
 +appender.index_search_slowlog_rolling.layout.type = PatternLayout
 +appender.index_search_slowlog_rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c] %marker%.-10000m%n
 +appender.index_search_slowlog_rolling.filePattern = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_index_search_slowlog-%d{yyyy-MM-dd}.log
 +appender.index_search_slowlog_rolling.policies.type = Policies
 +appender.index_search_slowlog_rolling.policies.time.type = TimeBasedTriggeringPolicy
 +appender.index_search_slowlog_rolling.policies.time.interval = 1
 +appender.index_search_slowlog_rolling.policies.time.modulate = true
 +
 +logger.index_search_slowlog_rolling.name = index.search.slowlog
 +logger.index_search_slowlog_rolling.level = trace
 +logger.index_search_slowlog_rolling.appenderRef.index_search_slowlog_rolling.ref = index_search_slowlog_rolling
 +logger.index_search_slowlog_rolling.additivity = false
 +
 +appender.index_indexing_slowlog_rolling.type = RollingFile
 +appender.index_indexing_slowlog_rolling.name = index_indexing_slowlog_rolling
 +appender.index_indexing_slowlog_rolling.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_index_indexing_slowlog.log
 +appender.index_indexing_slowlog_rolling.layout.type = PatternLayout
 +appender.index_indexing_slowlog_rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c] %marker%.-10000m%n
 +appender.index_indexing_slowlog_rolling.filePattern = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_index_indexing_slowlog-%d{yyyy-MM-dd}.log
 +appender.index_indexing_slowlog_rolling.policies.type = Policies
 +appender.index_indexing_slowlog_rolling.policies.time.type = TimeBasedTriggeringPolicy
 +appender.index_indexing_slowlog_rolling.policies.time.interval = 1
 +appender.index_indexing_slowlog_rolling.policies.time.modulate = true
 +
 +logger.index_indexing_slowlog.name = index.indexing.slowlog.index
 +logger.index_indexing_slowlog.level = trace
 +logger.index_indexing_slowlog.appenderRef.index_indexing_slowlog_rolling.ref = index_indexing_slowlog_rolling
 +logger.index_indexing_slowlog.additivity = false