Amit Chopra
12/11/2020, 4:50 PMTing Chen
12/11/2020, 8:00 PMMahesh Yeole
12/14/2020, 9:04 PMAmit Chopra
01/13/2021, 6:46 PMJackie
01/13/2021, 6:53 PMpinot.server.query.executor.num.groups.limit
Zac Farrell
01/20/2021, 8:12 PMjava.lang.NoClassDefFoundError: org/apache/pinot/client/JsonAsyncHttpPinotClientTransportFactory
i've tried running both v0.6.0 (latest) and 0.5.0 (version called out in docs) but both produce the same error. I've also tried compiling the jar from source, as well as including it as an explicit dependency in maven.
Any help is appreciated, thanks!Mohit Singh
05/23/2021, 2:42 PM{
"schemaName": "test_schema",
"dimensionFieldSpecs": [
{
"name": "client_id",
"dataType": "STRING"
},
{
"name": "master_property_id",
"dataType": "INT"
},
{
"name": "business_unit",
"dataType": "STRING"
},
{
"name": "error_info_str",
"dataType": "STRING"
}
],
"dateTimeFieldSpecs": [
{
"name": "timestamp",
"dataType": "LONG",
"format": "1:MILLISECONDS:EPOCH",
"granularity": "1:MILLISECONDS"
}
]
}
Table:
{
"REALTIME": {
"tableName": "test_schema_REALTIME",
"tableType": "REALTIME",
"segmentsConfig": {
"schemaName": "test_schema",
"replication": "1",
"replicasPerPartition": "1",
"timeColumnName": "timestamp"
},
"tenants": {
"broker": "DefaultTenant",
"server": "DefaultTenant",
"tagOverrideConfig": {}
},
"tableIndexConfig": {
"bloomFilterColumns": [],
"noDictionaryColumns": [],
"onHeapDictionaryColumns": [],
"varLengthDictionaryColumns": [],
"enableDefaultStarTree": false,
"enableDynamicStarTreeCreation": false,
"aggregateMetrics": false,
"nullHandlingEnabled": false,
"invertedIndexColumns": [],
"rangeIndexColumns": [],
"autoGeneratedInvertedIndex": false,
"createInvertedIndexDuringSegmentGeneration": false,
"sortedColumn": [],
"loadMode": "MMAP",
"streamConfigs": {
"streamType": "kafka",
"stream.kafka.topic.name": "TestTopic",
"stream.kafka.broker.list": "localhost:9092",
"stream.kafka.consumer.type": "lowlevel",
"stream.kafka.consumer.prop.auto.offset.reset": "smallest",
"stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder",
"stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
"schema.registry.url": "<http://localhost:8081>",
"realtime.segment.flush.threshold.rows": "0",
"realtime.segment.flush.threshold.time": "24h",
"realtime.segment.flush.segment.size": "100M"
}
},
"metadata": {},
"quota": {},
"routing": {},
"query": {},
"ingestionConfig": {
"transformConfigs": [
{
"columnName": "error_info_str",
"transformFunction": "json_format(error_info)"
}
]
},
"isDimTable": false
}
}
Kafka Avro Schema:
{
"type": "record",
"name": "TestRecord",
"namespace": "com.test.ns",
"fields": [
{
"name": "client_id",
"type": [
"null",
"string"
]
},
{
"name": "master_property_id",
"type": "int"
},
{
"name": "business_unit",
"type": [
"null",
"string"
]
},
{
"name": "error_info",
"type": {
"type": "record",
"name": "ErrorInfo",
"fields": [
{
"name": "code",
"type": [
"null",
"string"
]
},
{
"name": "description",
"type": [
"null",
"string"
]
}
]
}
},
{
"name": "timestamp",
"type": [
"null",
"long"
],
"default": null
}
]
}
Kaushik Ranganath
06/07/2021, 4:01 AMKamal Chavda
07/09/2021, 3:24 PMBruce Ritchie
07/09/2021, 6:27 PMBruce Ritchie
07/09/2021, 6:56 PMBruce Ritchie
08/01/2021, 6:31 PMxtrntr
08/10/2021, 2:30 AMProcessed requestId=34,table=sorted_events_OFFLINE,segments(queried/processed/matched/consuming)=198/198/198/-1,schedulerWaitMs=0,reqDeserMs=0,totalExecMs=426,resSerMs=0,totalTimeMs=426,minConsumingFreshnessMs=-1,broker=Broker_172.26.0.4_8099,numDocsScanned=259467,scanInFilter=619119085,scanPostFilter=259467,sched=fcfs
Slow query: request handler processing time: 427, send response latency: 3, total time to handle request: 430
Processed requestId=35,table=events_OFFLINE,segments(queried/processed/matched/consuming)=198/198/118/-1,schedulerWaitMs=0,reqDeserMs=5,totalExecMs=221,resSerMs=0,totalTimeMs=226,minConsumingFreshnessMs=-1,broker=Broker_172.26.0.4_8099,numDocsScanned=657,scanInFilter=346815,scanPostFilter=657,sched=fcfs
also, i’m wondering what is considered prompts "Slow query: …"
to show up in logs? does this mean pinot is suggesting that some optimization is possible to speed up my queries?Tiger Zhao
08/16/2021, 3:19 PMxtrntr
08/20/2021, 5:14 AM# table1
<s3://bucket/pinot-segments/table1/>
# table2
<s3://bucket/pinot-segments/table2/>
do you need to tell controller where segments for each table? i only see controller.data.dir
Tiger Zhao
08/20/2021, 2:50 PMcontroller.data.dir
(from the controller conf) be the same as the outputDirURI
(from the ingestion jobspec) ?Tiger Zhao
08/24/2021, 9:26 PMTiger Zhao
08/25/2021, 2:18 PMTiger Zhao
08/26/2021, 7:16 PMThiago Pereira
08/28/2021, 12:51 PMJ K
08/31/2021, 2:04 PMLuis Fernandez
08/31/2021, 3:02 PMTiger Zhao
09/01/2021, 5:39 PMTiger Zhao
09/02/2021, 9:49 PMLuis Fernandez
09/07/2021, 4:38 PMxtrntr
09/08/2021, 2:52 AMThis should only be used in standalone setups or for POC.
Tiger Zhao
09/08/2021, 2:53 PMenableDefaultStarTree=True
, is it possible to also specify extra aggregations in functionColumnPairs or change the maxLeafRecords (or any other config)? I think having it automatically generate and sort the dimensionSplitOrder list is very helpful but I also want to add more aggregations on top of the default.Tiger Zhao
09/08/2021, 10:17 PMsina
09/10/2021, 6:51 AMTiger Zhao
09/10/2021, 8:24 PM