Apache Kafka Confluent cluster Dockerization (Part-3)

---
version: '3.2'
services:
kafka-rest:
image: <image kafka-rest>
container_name: kafka-rest
ports:
- 8082:8082
environment:
KAFKAREST_HEAP_OPTS: "-Xmx256M"
KAFKA_REST_ID: kafka-rest-server
KAFKA_REST_LISTENERS: http://0.0.0.0:8082
KAFKA_REST_SCHEMA_REGISTRY_URL: http://10.0.0.101:8081
KAFKA_REST_BOOTSTRAP_SERVERS: PLAINTEXT://10.0.0.101:9092,PLAINTEXT://10.0.0.102:9092,PLAINTEXT://10.0.0.103:9092
KAFKA_REST_ID: This is a unique ID for the Confluent REST Proxy server instance.KAFKA_REST_LISTENERS: Comma-separated list of listeners that listen for API requests over either HTTP or HTTP.KAFKA_REST_SCHEMA_REGISTRY_URL: The base URL for Schema Registry that should be used.KAFKA_REST_BOOTSTRAP_SERVERS: A list of Kafka brokers to connect to. This configuration is particularly important when Kafka security is enabled, because Kafka may expose multiple endpoints that all will be stored in ZooKeeper, but REST Proxy may need to be configured with just one of those endpoints.
---
version: '3.2'
services:
kafka-connect:
image: <kafka connect image>
container_name: kafka-connect
ports:
- 8083:8083
environment:
KAFKA_HEAP_OPTS: "-Xms256M -Xmx1024M"
CONNECT_BOOTSTRAP_SERVERS: 10.0.0.101:9092,10.0.0.102:9092,10.0.0.101:9092
CONNECT_GROUP_ID: connect-cluster
CONNECT_KEY_CONVERTER: io.confluent.connect.avro.AvroConverter
CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: http://10.0.0.101:8081
CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://10.0.0.101:8081
CONNECT_CONFIG_STORAGE_TOPIC: connect-configs
CONNECT_OFFSET_STORAGE_TOPIC: connect-offsets
CONNECT_STATUS_STORAGE_TOPIC: connect-statuses
CONNECT_PLUGIN_PATH: /usr/share/java
CONNECT_BOOTSTRAP_SERVERS: A list of Kafka broker to use for establishing the initial connection to the Kafka cluster. The client uses all servers regardless of which servers are specified for bootstrapping. The list only impacts the initial hosts used to discover the full set of servers.CONNECT_GROUP_ID: A unique string that identifies the Connect cluster group this worker belongs to. This property is important when setting up a distributed cluster. All nodes of a cluster should use the same group id.CONNECT_KEY_CONVERTER: Converter class for key Connect data. This controls the format of the data that will be written to Kafka for source connectors or read from Kafka for sink connectors.CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: This property provides the Schema Registry URL, this property is needed when key_converter is specified.CONNECT_VALUE_CONVERTER: Converter class for value Connect data. This controls the format of the data that will be written to Kafka for source connectors or read from Kafka for sink connectors.CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: This property provides the Schema Registry URL, this is needed when value_converter is specified.CONNECT_CONFIG_STORAGE_TOPIC: The name of the topic where connector and task configuration data are stored. This must be the same for all Workers with the same groupID.CONNECT_OFFSET_STORAGE_TOPIC: The name of the topic where connector and task configuration offsets are stored. This must be the same for all Workers with the same groupID.CONNECT_STATUS_STORAGE_TOPIC: The name of the topic where connector and task configuration status updates are stored. This must be the same for all Workers with the same groupID.CONNECT_PLUGIN_PATH: The comma-separated list of paths to directories that contain Kafka Connect plugins.
---
version: '3.2'
services:
kafka-connect:
image: <kafka connect image>
container_name: kafka-connect
ports:
- 8083:8083
environment:
KAFKA_HEAP_OPTS: "-Xms256M -Xmx1024M"
CONNECT_BOOTSTRAP_SERVERS: 10.0.0.101:9092,10.0.0.102:9092,10.0.0.101:9092
CONNECT_SEND_BUFFER_BYTES: 52428800
CONNECT_RECEIVE_BUFFER_BYTES: 52428800
CONNECT_GROUP_ID: connect-cluster
CONNECT_KEY_CONVERTER: io.confluent.connect.avro.AvroConverter
CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: http://10.0.0.101:8081
CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://10.0.0.101:8081
CONNECT_CONFIG_STORAGE_TOPIC: connect-configs
CONNECT_OFFSET_STORAGE_TOPIC: connect-offsets
CONNECT_STATUS_STORAGE_TOPIC: connect-statuses
CONNECT_PLUGIN_PATH: /usr/share/java
https://docs.confluent.io/current/kafka-rest/config.htmlhttps://docs.confluent.io/current/connect/references/allconfigs.html

--

--

--

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Exposing a SOAP service as a Managed REST API

IFTTT — If This Then That (The Tool To Automate The Web Application Task)

The Problem With Requirements

what i have been waiting for has come !

Unleashing the Full Potential of Software Engineering Talent

How to Export your Data Set from Power BI to SQL Server Using DAX Studio

About Tarun Telang

The Curious Case of Devise Gem

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Ankur Garg

Ankur Garg

More from Medium

CockroachDB — Postgres for the cloud

[Kafka] Running Kafka with Docker

Continuous Integration (CI)

Packaging your work in a box to ship safely anywhere aka Containers