[EXAMPLE] Kafka DevOps by Confluent

Confluent has just released its Kafka DevOps project which is a great example of using shell-operator.

This new project is a “simulated production environment running a streaming application targeting Apache Kafka on Confluent Cloud.” Applications and resources there are managed via operators and GitOps approach. More details are available in the Confluent’s blog.

Shell-operator is used to implement two operators residing inside Docker images.

Confluent Cloud Operator

The ccloud-operator utilizes the Confluent Cloud CLI and shell scripts built into a Docker image to implement a Kubernetes Operator style (Declare → Apply) solution from the command line. Using the shell-operator project enables us to build basic event-driven scripts, called hooks, which run inside the Kubernetes cluster, reacting to changes in Kubernetes resources. This allows us to declare basic Kubernetes ConfigMap resources, which defines the Confluent Cloud services we want to manage, and respond to changes to the resources automatically.

Kafka Connect Operator

Connector configurations are defined in JSON and managed via the Connect worker HTTP API, which makes them well suited to the Declare → Apply model of Kubernetes. This is accomplished in kafka-devops using another shell-operator tool called connect-operator.

I really enjoyed building the tools and the shell-operator worked perfectly for my use case. If you have any questions about the project feel free to reach out!

4 Likes

ⓒ 2020 Flant. Powered by Discourse.