Observability of SpringBoot Services in K8s with Prometheus and Grafana

In kubernetes (k8s) we can monitor our spring boot services and gain insights on it with Prometheus and Grafana.

Observability of SpringBoot Services in K8s with Prometheus and Grafana

In kubernetes (k8s) we can monitor our spring boot services and gain insights on it with Prometheus and Grafana.

Continuing on from our previous post, Spring Data REST on Kubernetes. We add the following dependency to the vadal-data-rest service pom.xml.

<!-- Micrometer Prometheus registry  -->
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

Build and Deploy the image to k8s

mvn spring-boot:build-image

there should be an update to the docker image

docker images
vadal-data-rest                                                  0.0.1-SNAPSHOT

Tell k8s to update the image in its pods:

kubectl patch deploy vadal-data-rest -p '{"spec":{"template":{"spec":{"terminationGracePeriodSeconds":29}}}}'

Prometheus

Delete any older installation.

helm del --purge prometheus
helm install --wait --name prometheus --set service.type=NodePort,service.nodePort=9090 stable/prometheus

Note the local cluster url, prometheus-server.default.svc.cluster.local, we will need this to connect grafana to it.


Make it available on port 30001

kubectl edit svc/prometheus-server

and change the nodePort to 30001

  ports:
  - name: http
    nodePort: 30001
    port: 80
    protocol: TCP
    targetPort: 9090
  selector:
    app: prometheus
    component: server
    release: prometheus
  sessionAffinity: None
  type: NodePort

Navigate to

http://localhost:30001/

You should see the prometheus dashboard. Add connectivity for prometheus to vadal-data-rest service:

kubectl edit svc/vadal-data-rest
 apiVersion: v1
 kind: Service
 metadata:
   annotations:
      prometheus.io/path: /actuator/prometheus
      prometheus.io/port: "7777"
      prometheus.io/scrape: "true"
   name: vadal-data-rest

Navigate to prometheus status > targets and vadal-data-rest should show up as an endpoint.

NOTE: prometheus scrapes data once every 15s by default.

Adding grafana gives you a great dashboard to observe your services.


GRAFANA

Install

helm install --name grafana stable/grafana

Make it available on port 30002

kubectl edit svc/grafana

and change the nodePort to 30002

To login to http://localhost:30002/

username is admin

For password:

kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode

Connect grafana to prometheus as a data source. Configure data source as a server and url as prometheus-server.default.svc.cluster.local (this will be output when you install prometheus).

Import spring boot 2.1 dashboard using the id 10280:

You should get something similar to this:

Conclusion

For spring boot service monitoring all that is needed is the micrometer-registry-prometheus dependency. With helm it is simple to install both prometheus and grafana. One extra step to point the rest service to prometheus, which can also be done with a service yml file. With grafana you can view all the exposed spring metrics.

Next step logs.

See also:

https://developer.ibm.com/technologies/containers/tutorials/monitoring-kubernetes-prometheus/

Related Article