Kafka REST proxy in Kubernetes

Overview

We are considering the following scenario — you have applications that run in a Kubernetes cluster that will use of Kafka as a streaming platform. This article will guide you through creating a cluster on Confluent’s platform and deploy a Kafka REST proxy inside Kubernetes for communicating with the Kafka cluster.

Confluent cloud setup

With the following steps you will create an account with Confluent’s cloud platform, create an environment and deploy a Kafka cluster in that environment.

1. Register for an account with Confluent and setup your account — https://confluent.cloud/signup
2. After logging into your account you will have to create an environment
3. The next step after creating an environment is to create a cluster. There are three parts to this:

Creating a new cluster

• Name the cluster
• Select a provider (AWS or GCP)
• Select a zone that is close to your Kubernetes cluster

With this you have created the cluster. After the creation process finishes there are only two steps left to do on Confluent’s platform:

• Generate API Key + Secret for access. In the cluster details go to Overview -> API access and select Create Key. A key + secret will be automatically created for you and you must copy them.

Creating a new API key for the cluster

• Go to Data In/Out -> Clients and copy the configuration part — we will need it for the REST proxy. Save the configuration on your local machine and name the file kafka-rest.properties

Deploying Kafka REST proxy in your Kubernetes cluster*

*This article assumes you have a Kubernetes cluster running and kubectl configured

After we have created our Kafka cluster we can continue with deploying the REST proxy in the cluster. For this scenario we have a deployment with an autoscaler and a service that exposes the deployment within the cluster. You can checkout the configuration file in this repository.

• Open the kafka-rest.properties you saved earlier. Prepend with client. the following configuration properties: sasl.mechanism, sasl.jaas.config and secuirty.protocol. Substitute with your API key and secret you generated before.

ssl.endpoint.identification.algorithm=https
client.sasl.mechanism=PLAIN
request.timeout.ms=20000
bootstrap.servers=xx.yyy.zzz.gcp.confluent.cloud:9092
retry.backoff.ms=500
client.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username={API_KEY_HERE} password={API_SECRET_HERE}
client.security.protocol=SASL_SSL

• Create a secret with the configuration you save:

kubectl create secret generic kafka-rest-config —from-file=kafka-rest.properties

• Deploy the Kafka REST Proxy:

curl -s https://gitlab.com/atkozhuharov/kafka-rest-proxy-kubernetes/raw/master/kafka-proxy-deployment.yaml | kubectl apply -f -

Verification and testing

Now that we have deployed the proxy we have to verify that it is indeed running and connected to our Kafka server.

• Go to your cluster in Confluent and create a new Topic

• Create a port forwarding to the the service

kubectl port-forward service/kafka-cloud-proxy 8082:8082

• Verify that we can see the previously created topic:

curl http://localhost:8082/topics

If you can see your topic in the list then

Troubleshooting

In case you can’t verify the proxy is working here are some pointers on what to check:

1. Verify the API Key and Secret are correct or generate a new pair
2. Verify you have
3. Check the pod logs for errors

kubectl get pod -l app=kafka-cloud-proxy
kubectl logs $POD_NAME