Is there an easy way to expose the Kafka started i...
# general
Is there an easy way to expose the Kafka started in that container? I tried adding both a
-p 19092:19092
-p 9092:9092
on that command line but it didn't work. It looks like the Dockerfile doesn't expose it.
do you mean the kafka started inside the quickstart ?
the port is 19092
@Xiang Fu - yea, I tried to access it but couldn't.
If I use the
docker run ...
and then try
bin/ --create --bootstrap-server localhost:19092 --replication-factor 1 --partitions 1 --topic events-realtime
the call times out. I've tried a few combinations.
I think the kafka started inside container is not correctly exposed outside
let me check
Cool, ty. Yea, when I looked at the file, it didn't look like it was exposed.
One thin I’m trying is like this :
Copy code
docker run \
    --network=pinot-demo \
    --name pinot-quickstart \
    -v ~/software/kafka_2.11-2.4.0:/app/kafka_2.11-2.4.0 \
    -p 9000:9000 \
    -d apachepinot/pinot:latest QuickStart \
    -type stream
is where I put my local kafka bin directory
then I can do
docker exec -it pinot-quickstart bash
to enter into the container
then create topic
Copy code
root@25128d55df90:/app/kafka_2.11-2.4.0# bin/  --zookeeper localhost:2191/kafka --topic test --replication-factor 1 --partitions 1 --create
Created topic test.
use console producer
Copy code
root@25128d55df90:/app/kafka_2.11-2.4.0# bin/  --broker-list localhost:19092 --topic test
and console consumer
Copy code
root@25128d55df90:/app/kafka_2.11-2.4.0# bin/ --bootstrap-server localhost:19092 --topic test --from-beginning
you can also copy your local file to directory
then that file is visible inside the container
You can get kafka bins from:
Copy code
wget <>
tar -xzvf kafka_2.11-2.4.0.tgz -C  ~/software
then mount
Ah, true. That will work. Thanks!
@Xiang Fu - ah, I was trying to get access to Kafka's port so I could stream results to it from another job I'm running locally.
Also, when I go through the README in the Docker directory, I tried one of the commands and it's failing
docker-compose -f docker-compose.yml up
I'm seeing a lot of connection refused
Are your local app also running in the docker?
If yes, you can do docker run —network pinot-demo...
Hmm, I could try running my job in a different network.
another way is to use a standalone kafka container
and set to configure then expose those ports
also something you can try if not stick to kafka native producer is a Kafka-Rest-Proxy
which has rest api to prodcue to kafka
is easier for local testing
you can deploy a kafka rest proxy with docker network
and port forward ot your local for test producer
Interesting. Ok, I'll first try to see if changing the network works okay.
Hmm, so I got my job running on the same network but it's still not getting access to the kafka that's in the quick start docker container.
How would I do the "set to configure then expose those ports" option?
Are those command line args I can pass into a docker run?
kafka is not customizable in pinot-quickstart,
are your trying to produce to
Sorry, just got back. Yea.
Neha provided a possible solution to the local script. I'm going to try that.
Ah, sweet. The other local solution works now. I'm unblocked now. I'll probably get back to a better Docker solution when I'm focusing on deploying Pinot.