Creating a Topic with Custom Configurations in Kafka

Use Docker Compose to build the Kafka Cluster:

cd content-kafka-deep-dive

docker-compose up -d --build

Now, let's make sure Java is installed (if you receive a dpkg frontend lock message, wait a few minutes and then try again):

sudo apt install default-jdk

Unzip and change into the Kafka binaries directory:

tar -xvf kafka_2.12-2.2.0.tgz && mv kafka_2.12-2.2.0/ kafka

Create a topic named transaction:

bin/kafka-topics.sh --zookeeper localhost:2181 
--create 
--topic transaction 
--replication-factor 3 
--partitions 3

Add the custom configuration min.insync.replicas=3 to the topic transaction:

bin/kafka-configs.sh --zookeeper localhost:2181 
--alter 
--entity-type topics 
--entity-name transaction 
--add-config min.insync.replicas=3

Verify the topic configuration applied:

bin/kafka-configs.sh --zookeeper localhost:2181 
--describe 
--entity-type topics 
--entity-name transaction

First off, let's create a JSON file named replicacount.json with these contents:

{"partitions":
 [{"topic": "transaction", "partition": 0,
 "replicas": [
 2
 ]
 }
 ],
 "version":1
}

Now, we can execute the replica count change but using that JSON file:

bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 
--execute 
--reassignment-json-file replicacount.json

Once we get a "Successfully started..." message, let's describe the topic to see the replica change:

bin/kafka-topics.sh --zookeeper localhost:2181 
--topic transaction 
--describe

Open a producer and send some messages to your topic:

bin/kafka-console-producer.sh --broker-list localhost:9092 
--topic transaction 
--producer-property acks=all