https://pinot.apache.org/ logo
Join Slack
Powered by
Is there an easy way to expose the Kafka started i...
# general
d
Is there an easy way to expose the Kafka started in that container? I tried adding both a 
-p 19092:19092
and 
-p 9092:9092
on that command line but it didn't work. It looks like the Dockerfile doesn't expose it.
x
do you mean the kafka started inside the quickstart ?
the port is 19092
d
@Xiang Fu - yea, I tried to access it but couldn't.
If I use the 
docker run ...
and then try 
bin/kafka-topics.sh --create --bootstrap-server localhost:19092 --replication-factor 1 --partitions 1 --topic events-realtime
the call times out. I've tried a few combinations.
x
I think the kafka started inside container is not correctly exposed outside
let me check
d
Cool, ty. Yea, when I looked at the file, it didn't look like it was exposed.
x
One thin I’m trying is like this :
Copy code
docker run \
    --network=pinot-demo \
    --name pinot-quickstart \
    -v ~/software/kafka_2.11-2.4.0:/app/kafka_2.11-2.4.0 \
    -p 9000:9000 \
    -d apachepinot/pinot:latest QuickStart \
    -type stream
~/software/kafka_2.11-2.4.0
is where I put my local kafka bin directory
then I can do 
docker exec -it pinot-quickstart bash
to enter into the container
then create topic
Copy code
root@25128d55df90:/app/kafka_2.11-2.4.0# bin/kafka-topics.sh  --zookeeper localhost:2191/kafka --topic test --replication-factor 1 --partitions 1 --create
Created topic test.
use console producer
Copy code
root@25128d55df90:/app/kafka_2.11-2.4.0# bin/kafka-console-producer.sh  --broker-list localhost:19092 --topic test
>a
>b
>c
>d
>e
and console consumer
Copy code
root@25128d55df90:/app/kafka_2.11-2.4.0# bin/kafka-console-consumer.sh --bootstrap-server localhost:19092 --topic test --from-beginning
a
b
c
d
e
you can also copy your local file to directory 
~/software/kafka_2.11-2.4.0
then that file is visible inside the container
You can get kafka bins from:
Copy code
wget <https://downloads.apache.org/kafka/2.4.0/kafka_2.11-2.4.0.tgz>
tar -xzvf kafka_2.11-2.4.0.tgz -C  ~/software
then mount 
~/software/kafka_2.11-2.4.0
d
Ah, true. That will work. Thanks!
@Xiang Fu - ah, I was trying to get access to Kafka's port so I could stream results to it from another job I'm running locally.
Also, when I go through the README in the Docker directory, I tried one of the commands and it's failing 
docker-compose -f docker-compose.yml up
I'm seeing a lot of connection refused
x
Are your local app also running in the docker?
d
No
x
If yes, you can do docker run —network pinot-demo...
Hmm
d
Hmm, I could try running my job in a different network.
x
another way is to use a standalone kafka container
and set server.property to configure advertize.broker then expose those ports
also something you can try if not stick to kafka native producer is a Kafka-Rest-Proxy
which has rest api to prodcue to kafka
is easier for local testing
you can deploy a kafka rest proxy with docker network
and port forward ot your local for test producer
d
Interesting. Ok, I'll first try to see if changing the network works okay.
Hmm, so I got my job running on the same network but it's still not getting access to the kafka that's in the quick start docker container.
How would I do the "set server.property to configure advertize.broker then expose those ports" option?
Are those command line args I can pass into a docker run?
x
kafka is not customizable in pinot-quickstart,
are your trying to produce to 
pinot-quickstart:19092
?
d
Sorry, just got back. Yea.
Neha provided a possible solution to the local script. I'm going to try that.
Ah, sweet. The other local solution works now. I'm unblocked now. I'll probably get back to a better Docker solution when I'm focusing on deploying Pinot.
x
cool！
Open in Slack
PreviousNext
Powered by