Hi Team, I'm trying to load data from Minio to pin...
# troubleshootingm
Mahesh babu
11/24/2021, 5:17 AMHi Team, I'm trying to load data from Minio to pinot but facing issues while running yaml files. ERROR:expected '<document start>', but found BlockMappingStart
in 'string', line 6, column 1:
jobType: SegmentCreationAndMetad ...
x
Xiang Fu
11/24/2021, 5:18 AM
seems yaml issue
Xiang Fu
11/24/2021, 5:19 AM
should be no space before jobType
Xiang Fu
11/24/2021, 5:19 AM
check indentation
m
Mahesh babu
11/24/2021, 5:25 AMNow i'm getting this error
Mahesh babu
11/24/2021, 5:25 AMjava.lang.IllegalStateException: PinotFS for scheme: http has not been initialized
Mahesh babu
11/24/2021, 5:27 AMi'm trying this is in docker trying to read data from minio bucket used s3 as the scheme pinotFSSpecs:
- scheme: s3
className: org.apache.pinot.plugin.filesystem.S3PinotFS
Mahesh babu
11/24/2021, 6:26 AMCould not instantiate file system for class org.apache.pinot.plugin.filesystem.S3PinotFS with scheme s3
java.lang.IllegalArgumentException: null
x
Xiang Fu
11/24/2021, 7:32 AM
can you add a region into the config?
Xiang Fu
11/24/2021, 7:32 AM
Copy code
- scheme: s3
className: org.apache.pinot.plugin.filesystem.S3PinotFS
configs:
region: 'us-west-2'm
Mahesh babu
11/24/2021, 11:39 AMno use still i'm facing java.lang.IllegalStateException: PinotFS for scheme: https has not been initialized
Mahesh babu
11/24/2021, 4:07 PMFacing {"code":500, "error":"org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata"}
Mahesh babu
11/24/2021, 4:09 PMwhen creating realtime table in pinot ,i'm using{
"tableName": "transcript",
"tableType": "REALTIME",
"segmentsConfig": {
"timeColumnName": "timestampInEpoch",
"timeType": "MILLISECONDS",
"schemaName": "transcript",
"replicasPerPartition": "1"
},
"tenants": {},
"tableIndexConfig": {
"loadMode": "MMAP",
"streamConfigs": {
"streamType": "kafka",
"stream.kafka.consumer.type": "lowlevel",
"stream.kafka.topic.name": "transcript-topic",
"stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
"stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
"stream.kafka.broker.list": "localhost:9876",
"realtime.segment.flush.threshold.size": "0",
"realtime.segment.flush.threshold.time": "24h",
"realtime.segment.flush.desired.size": "50M",
"stream.kafka.consumer.prop.auto.offset.reset": "smallest"
}
},
"metadata": {
"customConfigs": {}
}
}
x
Xiang Fu
11/24/2021, 5:50 PM
Hmm, how do you start Pinot ?
Xiang Fu
11/24/2021, 5:51 PM
Docker or k8s or script?
Xiang Fu
11/24/2021, 5:51 PM
I guess your Kafka is local but in docker?
m
Mahesh babu
11/25/2021, 2:36 AMKafka and pinot both are in docker only
x
Xiang Fu
11/25/2021, 4:38 AM
are kafka and pinot in same docker network? https://docs.pinot.apache.org/basics/components/table#streaming-table-creation
Xiang Fu
11/25/2021, 4:39 AM
localhost:9876 seems to be wrong
Xiang Fu
11/25/2021, 4:40 AM
docker containers cannot talk to each other with localhost:<port>
m
Mahesh babu
11/25/2021, 4:40 AMi tried with 9092 also but no use.
x
Xiang Fu
11/25/2021, 4:42 AM
do you mean you have two docker containers one for pinot and one for kafka ?
m
Mahesh babu
11/25/2021, 4:42 AMyes
x
Xiang Fu
11/25/2021, 4:42 AM
then they cannot talk to each other using your local network
Xiang Fu
11/25/2021, 4:43 AM
you need to create a docker network and give each container a name when you start it
Xiang Fu
11/25/2021, 4:44 AM
see the doc I send you above, when you start a kafka using cmd like:
Copy code
docker run \
--network pinot-demo --name=kafka \
-e KAFKA_ZOOKEEPER_CONNECT=pinot-zookeeper:2181/kafka \
-e KAFKA_BROKER_ID=0 \
-e KAFKA_ADVERTISED_HOST_NAME=kafka \
-d wurstmeister/kafka:latestXiang Fu
11/25/2021, 4:44 AM
then you can use
kafka:9092m
Mahesh babu
11/25/2021, 4:45 AMhere network name is kafka or pinot-demo
x
Xiang Fu
11/25/2021, 4:45 AM
kafka
m
Mahesh babu
11/25/2021, 4:45 AMok let me try
x
Xiang Fu
11/25/2021, 4:45 AM
pinot-demo is the network name, which you need to create upfront
Xiang Fu
11/25/2021, 4:46 AM
also read this: https://docs.docker.com/network/
Xiang Fu
11/25/2021, 4:46 AM
running docker in local is far from running bash script.
m
Mahesh babu
11/25/2021, 4:48 AMsure thank you.
3 Views