https://pinot.apache.org/ logo
r

Raúl G.

07/13/2020, 12:27 PM
Hello, I'm newbie with Pinot and I need some help. I have deployed Pinot into a swarm cluster with docker. There is a controller, a broker and a server. I defined an schema and a realtime table. Zookeeper and Kafka were running before than Pinot, and are shared with other applications. I can see the table into the Data Explorer, but when I run a query ( select * from dmaTestData limit 10 ), only get an timeout error. Can someone help me? What's wrong in my configuration? -------------- docker-compose -------------- version: '3.7' services: pinot-controller: image: apachepinot/pinot:latest ports: - "9091:9000" command: ["StartController", "-zkAddress", "zookeeper:2181"] volumes: - ./data/controller_data:/opt/pinot/data - ./data/controller_configs:/opt/pinot/configs networks: - public pinot-broker: image: apachepinot/pinot:latest command: ["StartBroker", "-zkAddress", "zookeeper:2181"] volumes: - ./data/broker_data:/opt/pinot/data - ./data/broker_configs:/opt/pinot/configs networks: - public pinot-server: image: apachepinot/pinot:latest command: ["StartServer", "-zkAddress", "zookeeper:2181"] volumes: - ./data/server_data:/opt/pinot/data - ./data/server_configs:/opt/pinot/configs networks: - public networks: public: external: true ------ Schema ------ { "schemaName": "dmaTestSchema", "dimensionFieldSpecs": [ { "name": "dma_name", "dataType": "STRING" }, { "name": "dma_id", "dataType": "INT" } ], "metricFieldSpecs": [ { "name": "value", "dataType": "FLOAT" }, { "name": "value_old", "dataType": "FLOAT" } ], "dateTimeFieldSpecs": [ { "name": "time_stamp", "dataType": "STRING", "format" : "1DAYSSIMPLE_DATE_FORMAT:yyyy-MM-dd", "granularity": "1:DAYS" } ] } ----- Table ----- { "tableName": "dmaTestData", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "time_stamp", "timeType": "DAYS", "schemaName": "dmaTestSchema", "replication": "1", "replicasPerPartition": "1" }, "tenants": { "broker": "pinot-broker", "server": "pinot-server" }, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.consumer.type": "simple", "stream.kafka.topic.name": "transcript-topic", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.broker.list": "kafka:9092", "stream.kafka.hlc.zk.connect.string": "zookeeper:2181/kafka", "stream.kafka.zk.broker.url": "zookeeper:2181/kafka", "realtime.segment.flush.threshold.time": "12h", "realtime.segment.flush.threshold.size": "100000", "stream.kafka.consumer.prop.auto.offset.reset": "smallest" } }, "metadata": { "customConfigs": {} } }
message has been deleted
In the Kafka transcript-topic there are 1000 messages
x

Xiang Fu

07/14/2020, 12:54 AM
could you check pinot controller log
feeling could be some dns issue
I see you put controller port as : ports:       - “9091:9000”
then for the ui, does it send request to localhost:9091 or localhost:9000
r

Raúl G.

07/14/2020, 10:28 PM
Thank you for your reply, Xiang Fu. The ui is working on a custom url on 9091 port, that is redirected by docker swarm to the 9000 port in the container. It works, I think the problem is other ting. I attach the controller log. I can see is triying to connect to http://172.18.0.18:8099/query/sql , but I don't know where this IP address is from , because the subnet 172.18.255.255 is managed by docker bridge, but the container has an IP from 10.0.X.X network. From host can connect to 172.18.0.18:8099 and 172.18.0.17:9000 (Helix leader), but from pinot-controller docker only can reach 172.18.0.17:9000.
k

Kishore G

07/17/2020, 5:02 AM
@Xiang Fu ^^
x

Xiang Fu

07/17/2020, 5:15 AM
this comes from the container, you can enter into the container and check the network setting
can you try to start pinot using config file and put this line in the config
Copy code
pinot.set.instance.id.to.hostname=true
this will make pinot instance infer itself with name of hostname instead of ip
r

Raúl G.

07/17/2020, 5:07 PM
Thank you, Xiang Fu, now its working, we can see Kafka data