[Resolved] ERROR Uncaught exception in thread 'kafka-admin-client-thread | adminclient-1': (org.apache.kafka.common.utils.KafkaThread) java.lang.OutOfMemoryError: Java heap space

 

Issue:- 

When trying to delete the topic in the Amazon MSK kafka clusterr got the following error

Error:- 

 ERROR Uncaught exception in thread 'kafka-admin-client-thread | adminclient-1': (org.apache.kafka.common.utils.KafkaThread)

java.lang.OutOfMemoryError: Java heap space

Effect:-

Was not able to delete the Topic in the MSK kafka cluster due to the above error message.

 ERROR Uncaught exception in thread 'kafka-admin-client-thread | adminclient-1': (org.apache.kafka.common.utils.KafkaThread)
java.lang.OutOfMemoryError: Java heap space
	at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:61)
	at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:348)
	at org.apache.kafka.common.memory.MemoryPool$1.tryAllocate(MemoryPool.java:30)
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:112)
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:424)
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:385)
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:651)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:572)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:483)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1131)

Resolution:-

Follow the following steps

1. By including the truststore in the command with --command-config and the client properties was able to resolve the above issue. 

 kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml  
 

2. Now run the nslookup command on the DNS name and verify if its getting resolved or not

./kafka-topics.sh --bootstrap-server b-1.test-kafka.q15lx0.c10.kafka.us-west-2.amazonaws.com:9094,b-2.test-kafka.q15lx0.c10.kafka.us-west-2.amazonaws.com:9094,b-3.test-kafka.q15lx0.c10.kafka.us-west-2.amazonaws.com:9094 --delete --topic <topic-name>  --command-config  /Users/amittal/kafka/kafka_2.12-2.2.1/bin/client.properties

Explanation:-

You might be having the truststore in your home directory but you need to include it in your command with the --command-config otherwise it will fail to connect to the kafka cluster and eventually you wont be able to delete the topic from the Amazon Kafka cluster.

[Resolved] default.svc.cluster.local: Name or service not known

 

Issue:- 

After creating a service when I tried to verify if the DNS name for the service is getting resolved or I got the following error. 

Error:- 

 my-service.default.svc: Name or service not known

Effect:-

I was unable to confirm if the service DNS was actually resolving or not and if there was some issue as the service itself was not accessible via curl or the browser

 [centos@kubemaster service]$ nslookup my-service.default.svc  
 -bash: nslookup: command not found  
 [centos@kubemaster service]$ dig nslookup my-service.default.svc  
 -bash: dig: command not found  
 [centos@kubemaster service]$ ping nslookup my-service.default.svc  
 ping: my-service.default.svc: Name or service not known  
 [centos@kubemaster service]$ ping my-service.default.svc  
 ping: my-service.default.svc: Name or service not known  

Resolution:-

Follow the following steps

1. Create a pod with the DNS utils installed on it for making the nslookup command work inside the pod

 kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml  
 

2. Now run the nslookup command on the DNS name and verify if its getting resolved or not

[centos@kubemaster service]$ kubectl exec -it dnsutils -- nslookup my-service.default.svc  
 Server:          10.96.0.10  
 Address:     10.96.0.10#53  
 Name:     my-service.default.svc.cluster.local  
 Address: 10.111.144.147  

Explanation:-

Previously i was trying to resolve the DNS on the host network but the coredns works inside the kubernetes cluster only or pod network not the host network that is why you cannot use the traditional way of resolving the DNS using the nslookup or dig command. So we installed a pod with the dnsutils installed inside it and than we provide the nslookup command from inside the pod and directly print the result on the stdout. So you can use this way to resolve the DNS and verify if its working fine or not. Also you should put till svc as the kubernetes can take the cluster.local itself.

[Resolved] groupVersion shouldn't be empty

 

Issue:- 

When creating the simple resource like pod, replicaset, deployments etc got a groupVersion error specified below. 

Error:- 

 groupVersion shouldn't be empty

Effect:-

Not able to create the resource because of the above error

 apiversion: v1  
 kind: Pod  
 metadata:  
  name: pod2  
 spec:  
  containers:  
  - name: c1  
   image: nginx  

Resolution:-

If you look at the above configuration precisely you will find the apiversion  has been specified incorrectly. It should have been apiVersion  k.So just a difference of block letter can make that error. The same error will occur even if you forgot to mention the apiVersion in the configuration or it is misspelled. Below configuration will work fine.

 apiVersion: v1  
 kind: Pod  
 metadata:  
  name: pod2  
 spec:  
  containers:  
  - name: c1  
   image: nginx  

 

Explanation:-

apiVersion is hardcoded in the kubernetes. So if you misspell it, not use it or make a error in the Capital and small letter as well it will give the above error.