[Resolved] Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.

 

Issue:- 

Issue is with the dashboard service. When deploying the Dashboard service using the yaml in the kubernetes it gives the following error.

Error:- 

 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.

Effect:-

Because the dashboard service is not able to connect to the dashboard-metrics-scraper service the UI for the dashboard service is not loading up due to which the Dashboard is not working in the UI and timeout after some time.

Resolution:-

Instead of directly applying the yaml download it first using the wget or curl command as

1. [centos@kubemaster dashboard]$ wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml 

2. Edit the recommended.yaml and add the following arg into the yaml as

Original file:-

   spec:  
    containers:  
     - name: kubernetes-dashboard  
      image: kubernetesui/dashboard:v2.0.0  
      imagePullPolicy: Always  
      ports:  
       - containerPort: 8443  
        protocol: TCP  
      args:  
       - --auto-generate-certificates  
       - --namespace=kubernetes-dashboard  

Add the arg sidecar-host as - --sidecar-host=http://dashboard-metrics-scraper.kubernetes-dashboard.svc.cluster.local:8000 

  spec:  
    containers:  
     - name: kubernetes-dashboard  
      image: kubernetesui/dashboard:v2.0.0  
      imagePullPolicy: Always  
      ports:  
       - containerPort: 8443  
        protocol: TCP  
      args:  
       - --auto-generate-certificates  
       - --namespace=kubernetes-dashboard  
       - --sidecar-host=http://dashboard-metrics-scraper.kubernetes-dashboard.svc.cluster.local:8000 

Explanation:-

The address of the Sidecar Apiserver uses the format as protocol://address:port, e.g. http://localhost:8000. If not specified the assumption is that the binary runs inside a kubernetes cluster and service proxy will be used.

But this assumption was causing the issue so instead you can pass the sidecar-host with the DNS name of the service as - --sidecar-host=http://dashboard-metrics-scraper.kubernetes-dashboard.svc.cluster.local:8000 s If you have changed the service name than you might want to adjust the DNS name of your sidecar but this should work for most of the default settings. 

After running the sidecar-host now when you apply kubectl create command to the yaml file than it should be able to connect to the dashboard-metrics-scraper service and the Kubernetes Dashboard UI should open and you will no longer see the error in the logs.