Observability with Istio, Kiali and Grafana in Kubernetes and Spring Boot

Create REST Connected Services

Slightly contrived example, we have two services, vadal-users and vadal-posts. Vadal-posts contains a list of posts for users identified by their user id. Vadal-posts makes a REST call to vadal-users to find the name of the user based on the user id. This allows Kiali to gather tracing and traffic details.

Add vadal-users and vadal-posts spring-boot java services. See source code here https://gitlab.com/lightphos/spring/vadal.

In essence vpost calls vuser (we have added two posts):

    @GetMapping(value = "/posts")
    public Iterable<Post> getPosts(HttpServletRequest request) {
        log.info(LocalDateTime.now() + ", " + request.getRequestURL());
        Iterable<Post> posts = postRepo.findAll();
        posts.forEach(post -> {
            ResponseEntity<User> userResponse = restTemplate.getForEntity(userService + "/u/" + post.getUserId(), User.class);
            log.info("User returned {}", userResponse.getBody().getName());
            post.setUser(userResponse.getBody().getName());
        });

        return posts;
    }

In vadal-posts in the application.yml we define the following:

spring.application.name: Vadal Posts
freds.id: ${FREDS.ID:3}
wilmas.id: ${WILMAS.ID:4}

user.service: ${USER.SERVICE:http://localhost:7777}

As usual build the docker image using build pack.

mvn spring-boot:build-image for both.

Deploy To K8s

Help with RSI.

alias kc=`kubectl`

Deploy vusers.

kc create deployment vusers --image=vadal-users:0.0.1-SNAPSHOT -n vadal

kc expose deployment vusers --port 80 --target-port 7777 -n vadal

Deploy vposts (we need to provide the USER.SERVICE environment variable for vposts to call vuser). Run the following:

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vposts
  namespace: vadal
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vposts
  template:
    metadata:
      labels:
        app: vposts
    spec:
      containers:
        - name: vposts
          image: vadal-posts:0.0.1-SNAPSHOT
          ports:
            - containerPort: 8888
          env:
            - name: USER.SERVICE
              value: http://vusers.vadal.svc.cluster.local:80
EOF

Expose the service:

kc expose deployment vposts --type NodePort --port 8888 --target-port 8888 -n vadal

Check the logs manually:

kc get pods -n vadal | grep post
vposts-5585dfffbd-mpxkm   2/2     Running   0          3m46s

kc logs vposts -c vadal-posts -n vadal

2020-07-11T22:44:12.158095261Z Hibernate: call next value for hibernate_sequence
2020-07-11T22:44:12.166799807Z Hibernate: insert into post (details, topic, user, user_id, id) values (?, ?, ?, ?, ?)
2020-07-11T22:44:12.172370297Z Hibernate: insert into post (details, topic, user, user_id, id) values (?, ?, ?, ?, ?)
2020-07-11T22:44:12.174744664Z 2020-07-11 22:44:12.174  INFO 1 --- [           main] u.c.a.vadalposts.VadalPostApplication    : User service: http://vusers.svc.cluster.local

USER.SERVICE has been set to http://vusers.svc.cluster.local.

Also you can view the logs in Kiali (see previous post):

http://localhost:30004/kiali/console/namespaces/vadal/workloads/vposts?tab=logs

Check out the vposts service.

curl -i localhost:31436/posts
HTTP/1.1 200 OK
content-type: application/json
date: Sat, 11 Jul 2020 23:20:02 GMT
x-envoy-upstream-service-time: 519
server: istio-envoy
x-envoy-decorator-operation: vposts.vadal.svc.cluster.local
transfer-encoding: chunked

[
    {
    id: 1,
    userId: 3,
    user: "fred",
    topic: "freds topic",
    details: "rock city works"
    },
    {
    id: 2,
    userId: 4,
    user: "wilma",
    topic: "wilmas shop",
    details: "stone mall"
    }
]

Istio and Kiali Observation

Create a gateway and virtual service for vposts within Istio, this will enable the inter-service communication visualisation.

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: vadal-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
    - port:
        number: 80
        name: http
        protocol: http
      hosts:
        - vadal.local

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: posts
  namespace: vadal
spec:
  hosts:
    - vadal.local
  gateways:
    - vadal-gateway.istio-system.svc.cluster.local
  http:
    - match:
        - uri:
            exact: /posts
      route:
        - destination:
            host: vposts.vadal.svc.cluster.local
            port:
              number: 8888
EOF

Add vadal.local to your /etc/hosts with it pointing to your hosts IP.

Check it out:

curl -i http://vadal.local/posts

Increase the vusers replicas to 2.

kc scale deployment/vusers -n vadal --replicas=2

Make a request again.

The two calls to vuser from vposts are being load balanced between the two pods, nice:

Pod 1

Hibernate: select user0_.id as id1_1_0_, user0_.name as name2_1_0_, user0_.role_id as role_id3_1_0_, role1_.id as id1_0_1_, role1_.role as role2_0_1_ from user user0_ left outer join role role1_ on user0_.role_id=role1_.id where user0_.id=?

GET /u/3 HTTP/1.1" 200 - "-" "-" 0 288 12 11 "-" "Java/1.8.0_252" "91cc29b8-c3c8-9408-b3a4-c20d1fa236e0" "vusers.vadal.svc.cluster.local" "127.0.0.1:7777" inbound|80||vusers.vadal.svc.cluster.local 127.0.0.1:37020 10.1.1.43:7777 10.1.1.57:48222 outbound_.80_._.vusers.vadal.svc.cluster.local default

Pod 2

Hibernate: select user0_.id as id1_1_0_, user0_.name as name2_1_0_, user0_.role_id as role_id3_1_0_, role1_.id as id1_0_1_, role1_.role as role2_0_1_ from user user0_ left outer join role role1_ on user0_.role_id=role1_.id where user0_.id=?

GET /u/4 HTTP/1.1" 200 - "-" "-" 0 289 18 17 "-" "Java/1.8.0_252" "c235f5f2-7d54-9473-91a2-55909f1cc0c8" "vusers.vadal.svc.cluster.local" "127.0.0.1:7777" inbound|80||vusers.vadal.svc.cluster.local 127.0.0.1:37024 10.1.1.58:7777 10.1.1.57:58646 outbound_.80_._.vusers.vadal.svc.cluster.local default

Let's increase the replicas for vposts as well.

kc scale deployment/vposts -n vadal --replicas=2

Lets apply some load with this shell script:

# /bin/bash
for i in {1..10}
do
  curl -s http://vadal.local/posts &
done

You can see the services visually interacting in Kiali. Nice.

You can also check the request traces with the request-id.

Jaeger

With istioctl installed (which we did previously) we can also look at traces with Jaeger.

istioctl dashboard jaeger

Grafana

We can view volumetrics etc from Grafana, from our previous installation:

http://localhost:30003/?orgId=1

Kiali may give an error so change the Grafana details in ConfigMaps for Kiali:

  grafana:
    url: http://localhost:30003
    in_cluster_url: http://grafana.istio-system:3000

Conclusion

With Istio and Kiali we can see the  service REST communication and observe these visually with it providing request traffic rates, tracing and logs. Very useful. We can also look at service to service request tracing with Jaeger.

In addition there is a Grafana view providing details of the service cluster and the Isitio mesh as a whole.