Hazelcast Kubernetes Istio Kiali and Spring Data
Hazelcast in kubernetes and Spring Data
Quote:
Hazelcast provides central, predictable scaling of applications through in-memory access to frequently used data and across an elastically scalable data grid. These techniques reduce the query load on databases and improve speed. .
https://en.wikipedia.org/wiki/Hazelcast#:~:text=Hazelcast%20provides%20central%2C%20predictable%20scaling,on%20databases%20and%20improve%20speed.&text=The%20Hazelcast%20platform%20can%20manage%20memory%20for%20many%20different%20types%20of%20applications.
We deploy a hazelcast enabled java service that shares a data object in the memory grid between its instances. The data object is just like a normal database repository and we use the hazelcast spring data extension to read and write it. This is an interesting use case from the usual distributed caching ability hazelcast is used for. You can view and insert objects with the normal spring data methods, very useful.
We also deploy this within K8s and Istio giving us the load balancing and observability capabilities (with Kiali). Previous posts detail how this is done. We use our normal setup of macos running desktop for docker with kubernetes. This exercise can equally be done using minikube or kind.
Spring Data Application Code
Spring boot application code:
@SpringBootApplication
@RestController
@EnableHazelcastRepositories
@Slf4j
public class VadalUsersHCApplication {
public static void main(String[] args) {
SpringApplication.run(VadalUsersHCApplication.class, args);
}
@Autowired
UserRepo userRepo;
@PostConstruct
public void init() {
List<User> users = Arrays.asList(new User("fred", "boss"), new User("wilma", "director"));
users.forEach(user -> {
List<User> result = userRepo.findByName(user.getName());
if (result.isEmpty()) {
userRepo.save(user);
}
});
}
@Bean
Config config() {
Config config = new Config();
JoinConfig join = config.getNetworkConfig().getJoin();
join.getTcpIpConfig().setEnabled(false);
join.getMulticastConfig().setEnabled(false);
join.getKubernetesConfig().setEnabled(true)
.setProperty("namespace", "vadal")
.setProperty("service-name", "vhazelcast"); // endpoint name in the service port
return config;
}
@GetMapping(value = "/users")
public Iterable<User> getUsers(HttpServletRequest request) {
log.info(LocalDateTime.now() + ", " + request.getRequestURL());
return userRepo.findAll();
}
@PostMapping(value = "/user/{name}/{handle}")
public User addUser(HttpServletRequest request, @PathVariable("name") String name, @PathVariable("handle") String handle) {
User user = new User(name, handle);
log.info(LocalDateTime.now() + ", " + request.getRequestURL() + ", " + user);
return userRepo.save(user);
}
User class:
@Data
@NoArgsConstructor
@ToString
public class User implements Serializable {
@Id
private Long id;
private String name;
private String handle;
public User(String name, String handle) {
this.name = name;
this.handle = handle;
}
}UserRepo class:
public interface UserRepo extends HazelcastRepository<User, Long> {
List<User> findByName(@Param("n") String name);
The full code can be found here. The shared memory grid is initialised with objects in the PostConstruct. However we check first for existing objects with the same name as we will have multiple instances.
POM Snippet (inherited, check out the full source code, for spring dependencies etc):
<properties>
<spring.data.hazelcast.version>2.2.2</spring.data.hazelcast.version>
<kubernetes.hazelcast.version>1.5.2</kubernetes.hazelcast.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>com.hazelcast</groupId>
<artifactId>spring-data-hazelcast</artifactId>
<version>${spring.data.hazelcast.version}</version>
</dependency>
<dependency>
<groupId>com.hazelcast</groupId>
<artifactId>hazelcast-spring</artifactId>
</dependency>
<dependency>
<groupId>com.hazelcast</groupId>
<artifactId>hazelcast-client</artifactId>
</dependency>
<dependency>
<groupId>com.hazelcast</groupId>
<artifactId>hazelcast-kubernetes</artifactId>
<version>${kubernetes.hazelcast.version}</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
</dependencies>Build this in the usual manner.
mvn spring-boot:build-image
vadal-users-hazelcast:0.0.1-SNAPSHOT
Deployment, Services, Permissions
The application image exposes two ports, our user service and the embedded hazelcast code on port 5701. The vhazelcast port name in the first service is what our java application code in its config looks for (this config could also be placed in the application.yml file). We have set the replicas to two so we should expect two pods to be started and hazelcast linking the two. The cluster binding may or may not be needed depending on your K8s setup. It seemed to make no difference in desktop for docker K8s.
apiVersion: apps/v1
kind: Deployment
metadata:
name: vusershc
namespace: vadal
labels:
app: vusershc
spec:
replicas: 2
selector:
matchLabels:
app: vusershc
template:
metadata:
labels:
app: vusershc
spec:
containers:
- name: vadal-users-hazelcast
image: vadal-users-hazelcast:0.0.1-SNAPSHOT
ports:
- containerPort: 7777
- containerPort: 5701
---
apiVersion: v1
kind: Service
metadata:
name: vhazelcast
namespace: vadal
spec:
selector:
app: vusershc
ports:
- name: vhazelcast
port: 5701
type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
name: vusershc
namespace: vadal
labels:
app: vusershc
spec:
ports:
- name: http
port: 7777
protocol: TCP
selector:
app: vusershc
type: NodePort
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: default-cluster
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: view
subjects:
- kind: ServiceAccount
name: default
namespace: default
kubectl apply -f - <above>
This should deploy and set up the services.
You can check the logs in Kiali (see previous blog).
Logs from one of the workloads:
Members {size:2, ver:2} [
Member [10.1.2.83]:5701 - b85b78f8-08ac-47d9-9a33-bb6315eedc90 this
Member [10.1.2.84]:5701 - 21cd8100-10aa-47be-b236-069ec90f8478
]
Two members linked to each other, from the two replicas, nice.
Find our application services NodePort:
kc get svc -n vadal | grep hc
vusershc NodePort 10.111.27.233 7777:32334/TCP 25h
curl http://localhost:32334/users
[{"id":-7621056571992339709,"name":"fred","handle":"boss"},{"id":-4624569787408292316,"name":"wilma","handle":"director"}]
That's our IMDG acting like a database, across both instances.
Istio is also providing load balancing across the two services, which can be verified by checking the logs.
We can add a new entry:
curl http://localhost:32334/user/pebbles/baby -X POST
{"id":-5517234661065649476,"name":"pebbles","handle":"baby"}
Query:
curl http://localhost:32334/users
[{"id":-5517234661065649476,"name":"pebbles","handle":"baby"},{"id":-7621056571992339709,"name":"fred","handle":"boss"},{"id":-4624569787408292316,"name":"wilma","handle":"director"}]
We can scale the services:
kubectl scale deployment/vusershc -n vadal --replicas=3
deployment.apps/vusershc scaled
Check out hazelcast membership in the logs:
All three pods join as members.
Members {size:3, ver:3} [
Member [10.1.2.83]:5701 - b85b78f8-08ac-47d9-9a33-bb6315eedc90
Member [10.1.2.84]:5701 - 21cd8100-10aa-47be-b236-069ec90f8478 this
Member [10.1.2.85]:5701 - 8033f1d6-eb9b-408b-b97e-9b6ec2c5bc76
]
And with Istio the request is distributed across the instances.
Session Affinity
Using this technique it would be relatively easy to construct a session class and hold session data common across all the instances. This would provide a highly scalable resilient solution with very low latency by adding a few dependencies. There is no need for complicated Nginx rules, routes, header manipulation etc. If you've encountered this issue with JBoss instances and Nginx servers you know how tricky this can be.
With K8s and Istio it just scales distributing the load and the sessions are common in memory between all member instances. Auto-magical.
See the links below on session replication on why this is a big deal.
Observability
As well as logs, Kiali provides us with useful visuals.
You can see the hazelcast endpoint communicating with itself and you can gain details on the links.
Kubernetes API
The hazelcast K8s code uses K8s API to connect to the hazelcast endpoint (there is DNS way of connecting as well), and we can view that this exists in the following way.
kc proxy
http://localhost:8001/api/v1/namespaces/vadal/endpoints/vhazelcast
Conclusion
We have deployed our Spring application with Hazelcast embedded to K8s with Istio acting as the service mesh. The Spring Data code acts exactly as if we were talking to a database, quite powerful this. The same data is shared across this distributed set of instances and this is done dynamically as we use K8s power to scale the number of instances.
In order to add a gateway URL or security in Istio or Kong, check out my previous blog.
Further Reading
https://hazelcast.com/blog/spring-boot-hazelcast-session-replication/