https://pinot.apache.org/ logo
Join Slack
Powered by
Hey everyone! I’m trying to build a streaming arch...
# troubleshooting
f
Hey everyone! I’m trying to build a streaming architecture that consists of kafka and Pinot. Here’s my `docker-compose`:
Copy code
services:
  web:
    image: apachepinot/pinot:latest
    command: QuickStart -type hybrid
    container_name: "pinot"
    volumes:
      - ./config:/config
    ports:
      - "9000:9000"  # Controller
      - "9001:8000"  # Broker
      - "9002:7000"  # Server
Is 
kafka
included as a part of this image? If so, how can I reach it (which port)? Finally, how can I create a topic and publish/subscribe to it? thanks for your help!
d
A while ago I blogged about a monitoring setup for Apache Pinot. The provided example contains a fully executable docker compose setup with Kafka and Pinot and even a smoke test app
f
@Daniel Lavoie Thanks! Looking at the 
docker-compose
file in the repo you shared, I can see that you separate all Pinot pieces + zookeeper + kafka. Hovewer, in the compose file I shared, they are a part of one image. My question is: which solution is officially suggested by Pinot devs?
d
Kafka is not packaged in the Pinot image
Quickstart mode is good for local testing. Standalone component is more scalable and a better production practice.
f
So do I understand correctly that the best practice for testing locally (kafka + pinot) would be not to use Quickstart mode and follow your tutorial?
d
Just use whatever works for you 🙂
The example I shared has details to setup kafka in your docker compose
You can take the kafka stuff and integrate it in your quickstart docker compose
Kafka will be recheable locally from 
localhost:9092
, and from 
kafka:19092
within the pinot container.
f
Great, thanks a lot for your help!
Hey, just a follow up on our yesterday’s conversation. Here’s how ports in the image are exposed:
Copy code
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:2123            0.0.0.0:*               LISTEN      1/java              
tcp        0      0 0.0.0.0:7500            0.0.0.0:*               LISTEN      1/java              
tcp        0      0 0.0.0.0:2191            0.0.0.0:*               LISTEN      1/java              
tcp        0      0 127.0.0.1:19092         0.0.0.0:*               LISTEN      1/java              
tcp        0      0 0.0.0.0:7000            0.0.0.0:*               LISTEN      1/java              
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      1/java              
tcp        0      0 127.0.0.11:43079        0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:9000            0.0.0.0:*               LISTEN      1/java              
udp        0      0 127.0.0.11:49880        0.0.0.0:*                           -
It seems (confirmed) that 
19092
port is visible only internally. So there’s no way to access it from the outside. Is there a reason why this was designed this way?
The consequence is that I can’t interact with kafka (create a new topic and push events) from outside of the container.
d
Copy code
kafka:
    image: confluentinc/cp-kafka:5.5.0
    hostname: kafka
    restart: unless-stopped
    depends_on:
    - zookeeper
    ports:
    - 9092:9092
    - 19092:19092
    environment:
      KAFKA_ADVERTISED_LISTENERS: <LOCALHOST://localhost:9092>,<DOCKER://kafka:19092>
It’s suposed to be exposed with 9092 with localhost so you can access it from your local machine.
With kafka, there’s a one on one mapping between hostname and ports.
if you want to have it accept localhost, it needs to be specified explicitly with its own port.
f
Yeah, but I mean when I use only this image:
Copy code
services:
  web:
    image: apachepinot/pinot:latest
    command: QuickStart -type hybrid
    container_name: "pinot"
    volumes:
      - ./config:/config
    ports:
      - "9000:9000"  # Controller
      - "9001:8000"  # Broker
      - "9002:7000"  # Server
d
You have to use 2 images if you want kafka
pinot + kafka
when you will create your pinot table, you will only need to specify kafka:19092 as the boostrap server in your table stream config
f
Ok, let me test this config. I misunderstood your previous message and thought that kafka would be available externally on 19092 port by only using 
pinot:latest
image. Now it’s crystal clear that I must add another image for kafka. Let me test this. Once again, thanks for your help!:)
d
Yes, as I mentioned before, Kafka is not packaged inside the apache pinot docker image.
f
What about zookeeper? I see it mentioned as a dependency inside 
kafka
service.
d
Is the ZK port exposed on your quickstart container? If so, you can configure kafka to use the 
pinot:2181
f
Hey, I decided to use compose file from your tutorial + this one: https://docs.pinot.apache.org/basics/getting-started/advanced-pinot-setup#start-pinot-components-using-docker So finally I can access kafka etc. One thing I found out that this setup is not working with 
apachepinot/pinot:latest
image. 
0.6.0
is working perfectly fine. Just wanted to let you know about this.
Open in Slack
PreviousNext
Powered by