Setting Up Kafka in Docker

“Here we go, another article on setting up some thing in Docker 😒”, you might think. This time it’s Kafka. And I wouldn’t write about it if it was that trivial.

Thanks to wurstmeister, we have separate docker images for Kafka and Zookeeper. But setting this up my way without docker compose is a task in and of itself.


We’ll need two volumes for this guy. You know how it goes, go to “Volumes” side-menu and smash the “Add volume” button. Name the first one , and the second one - .

Next go to the “Containers” side menu, click the “Add container” button. Set:

  • name: zookeeper
  • image: wurstmeister/zookeeper:latest
  • publish port: 2181:2181

add volumes:

  • /opt/zookeeper-3.4.13/data to zookeeper_data
  • /opt/zookeeper-3.4.13/conf to zookeeper_conf

As always, a handy screenshot for you:

Now destroy that “Deploy the container” button and wait a little.


This guy will need just one volume. Name it . Now pay attention when creating a container. Set:

  • name: kafka
  • image: wurstmeister/kafka:latest
  • publish port: 9092:9092

add volumes:

  • /kafka to kafka_data
  • bind /var/run/docker.sock to /var/run/docker.sock


  • ZK:
  • HOST:

and envs are just for shortening commands later on.

Screenshot for sanity check:

And deploy.


There is already a guide on how to use this stuff, but there is a mistake in the consuming part.

Create topic

But let’s start with creating a topic. Go to “>_ Console” of the Kafka container. Or execute command below in your terminal session

docker exec -it kafka /bin/bash

Once inside container create topic just as in above mentioned guide: --create --topic topic \
--partitions 4 --zookeeper $ZK --replication-factor 1 --describe --topic topic --zookeeper $ZK

You’ll see messages about topic creation and it’s description:

Created topic topic.

Topic: topic PartitionCount: 4 ReplicationFactor: 1 Configs:
Topic: topic Partition: 0 Leader: 1002 Replicas: 1002 Isr: 1002
Topic: topic Partition: 1 Leader: 1002 Replicas: 1002 Isr: 1002
Topic: topic Partition: 2 Leader: 1002 Replicas: 1002 Isr: 1002
Topic: topic Partition: 3 Leader: 1002 Replicas: 1002 Isr: 1002


The guide says to run this: --topic=topic --zookeeper=$ZK

But if you try to do that you’ll get an error about .

So the correct command is this: --topic=topic --bootstrap-server=$HOST 

It didn’t freeze, it just waits for messages to come.


Open another console or terminal session and connect to the kafka container. Execute: --topic=topic \ --broker-list=`` 

Now you can type some messages here and it’ll show in another session where we have a consumer 🚀


Inside kafka_app in the project repo, you’ll find two short files with producer and consumer. This is just to show how we can use Kafka in Python:

python -m venv venv source
pip install -r requirements.txt

In another terminal session:

source venv/bin/activate

Now look at the terminal with our python consumer and you’ll see a topic, message key and value.

Also, if you didn’t close previous sessions with the console consumer, you’ll see the same message there. And publishing from the console producer will deliver a message to both console and python consumers.

Now you have one more tool in your toolbox! Go use this stuff 💪

Originally published at

переводчик-любитель | трачу два часа своего времени, чтобы тебе было чем заняться в течение пяти минут