Hi! Pinot Team. Is there a guide to connecting to a cluster from SQL Tools (dbeaver, mster, power bi etc) using pinot jdbc driver? without going through trino.

Hello all, just wanted to see whether you had anything planned to celebrate your first year? According to I work on ASF press team and always looking for things for social etc. Assuming this is accurate: https://projects.apache.org/committees.html?date. Thanks!

Hi Quick question regarding timestamps : if we have the sample value as epoch: "timestamp":1571900400000} and specify the schema as mentioned in the docs : "dateTimeFieldSpecs": [{ "name": "timestamp", "dataType": "LONG", "format" : "1MILLISECONDSEPOCH", "granularity": "1:MILLISECONDS" }]

Will it convert to timestamp format or will it still be epoch values ?

Also can i use data type as JSON for these kind of values: "address": { "city": "City_71", "state": "State_33", "zipcode": 58184 }

is the deep store intended purely for backup/recovery? specifically i'm wondering: • is there any concept of cold storage where data is only persisted in the deep store and read as needed for queries? • the docs say the deep store doesn't typically (?) store indexes, and that zk stores segment metadata. are indexes stored in zk? or they're stored locally and rebuilt on restore?

Hey folks, I was curious about the flink segment writer[1][2]. Is column sorting supported when using this method of segment creation? Is that handled internally by IngestionUtils.buildSegment i.e. SegmentIndexCreationDriverImpl#build ? Or is sorting not supported when building segments with Flink? Thanks! [1] https://github.com/apache/pinot/blob/e41d86b4b1225a4857767960ce595d735d5ec620/pinot-connectors/pinot-flink-connector/src/main/java/org/apache/pinot/connector/flink/sink/FlinkSegmentWriter.java [2] https://github.com/apache/pinot/blob/e41d86b4b1225a4857767960ce595d735d5ec620/pinot-connectors/pinot-flink-connector/src/main/java/org/apache/pinot/connector/flink/sink/PinotSinkFunction.java

Hi Team , I have a doubt in Star Tree I have a case in which it checks for nested columns in a table What I am trying to do is , apply these nested columns in a CASE as A In Star Tree I need to apply that A --> count__A Can I apply like these in my star tree generator or only columns respective to schema? Or there are any other support available?

Hi Team, Pinot Realtime tables connected to Kafka topics stops updating after sometime even though Kafka topics gets populated

Hello Hello Swag lovers! 🎩 🎒 🕶️ A stylish opportunity to tell our new joiners about our Swag store! We want to reward you for your Pinot contribution, whether for speaking at conferences, documenting your journey with Pinot, writing a blog post, organizing meetups, or anything else that brings value to the entire community ❤️ So- check out the table scoring (in the comment), share it with us via email, and choose your unique items!

Hi fellows, I'm trying to build a schema to accommodate a new table using the following structure

"event_time": "2021-07-30 12:45:16.668010"

trying to issue the following pattern

"dateTimeFieldSpecs": [{
    "name": "event_time",
    "dataType": "STRING",
    "format": "1:DAYS:SIMPLE_DATE_FORMAT:yyyy-MM-ddTHH:mm:ss.SSS-SS:SS",
    "granularity": "1:DAYS"
  }]

however getting the following issue

{
  "code": 400,
  "error": "Invalid schema: sch_output_faust_enriched_rides. Reason: invalid datetime format: 1:DAYS:SIMPLE_DATE_FORMAT:yyyy-MM-ddTHH:mm:ss.SSS-SS:SS"
}

is there someone here available to shed some lights?

Hi Folks , I read in the slack group that Connection Pooling is not possible in Pinot . That reduces the capability of my multithreaded low latency java application which I use to read the count of documents and guarantee the arrival . Are there any alternatives . If no , then who can help in suggesting why a reader connection pool is not an option ? @Mayank

In case someone is interesd, MindsDB has now integeation with pinot https://github.com/mindsdb/mindsdb/issues/1886

Hi Question on segments. Is there a way to uniquely identify a segment? Looking through a bunch of classes/interfaces in pinot-segment-spi. Don’t see anything. Hopefully just missing it. Looking at the tests (ConcurrentMapPartitionUpsertMetadataManagerTest for example) I can see segments being replaced by segments with the same name. So the segment’s name does not identify it uniquely. Is there anything that would uniquely identify a segment? And be stable? So I could refer to a segment through a property(s), not just the pointer to the segment implementation instance.

Hi. Is there a way to apply partial case statement ? Use case : case when followed by else scans all the documents whereas the same condition when applied in where doesn't do full scan.

sum(case when column_a = 1 then 1 else 0 end)

vs

count(*) where column_a = 1

Follow-up : Is there a way to apply coalesce ?

Hi all, we are starting to build our production dashboard for pinot and I'm curious if anyone wants to share their favorite/most important metrics. I'm going through them all and picking mine, but want to see what the community has to say.

Hey, what does this Pinot broker metric represents:

pinot_broker_queries_Count

? We have a table that is not receiving any queries but this metric value is still very high. Any other broker metric that shows the actual number of received queries? Something similar to:

pinot_broker_queries_meanRate

or

pinot_broker_queries_oneMinuteRate

but shows absolute query numbers.

Hello I am trying to query my pinot table by a BYTE type field. I am setting the query parameter in a PreparedStatement . I know internally Pinot store bytes as hex string but when I try to query it using the absolute value via code I get the following error

Caused by: org.apache.pinot.client.PinotClientException: Query had processing exceptions: 
[{"message":"QueryExecutionError:\norg.apache.pinot.spi.exception.BadQueryRequestException: java.lang.IllegalArgumentException: Cannot convert value: '0xac384c53f03855fa1b3616052f8ba32c6c2a2fec' to type: BYTES\n\tat org.apache.pinot.core.query.pruner.ColumnValueSegmentPruner.convertValue(ColumnValueSegmentPruner.java:469)\n\tat org.apache.pinot.core.query.pruner.ColumnValueSegmentPruner$ValueCache$CachedValue.ensureDataType(ColumnValueSegmentPruner.java:526)\n\tat org.apache.pinot.core.query.pruner.ColumnValueSegmentPruner$ValueCache.get(ColumnValueSegmentPruner.java:493)\n\tat org.apache.pinot.core.query.pruner.ColumnValueSegmentPruner.pruneEqPredicate(ColumnValueSegmentPruner.java:287)\n...\nCaused by: java.lang.IllegalArgumentException: Cannot convert value: 'HIQQEXWKW53RKN4W6XXC4Q232SYNZ3SZANVZZSUME5B5PRGXL663UAQA' to type: BYTES\n\tat org.apache.pinot.spi.data.FieldSpec$DataType.convertInternal(FieldSpec.java:522)\n\tat org.apache.pinot.core.query.pruner.ColumnValueSegmentPruner.convertValue(ColumnValueSegmentPruner.java:467)\n\t... 20 more","errorCode":200},{"message":"QueryExecutionError:\norg.apache.pinot.spi.exception.BadQueryRequestException: java.lang.IllegalArgumentException: Cannot convert value: 'HIQQEXWKW53RKN4W6XXC4Q232SYNZ3SZANVZZSUME5B5PRGXL663UAQA' to type: BYTES\n\tat org.apache.pinot.core.query.pruner.ColumnValueSegmentPruner.convertValue(ColumnValueSegmentPruner.java:469)\n\tat org.apache.pinot.core.query.pruner.ColumnValueSegmentPruner$ValueCache$CachedValue.ensureDataType(ColumnValueSegmentPruner.java:526)\n\tat org.apache.pinot.core.query.pruner.ColumnValueSegmentPruner$ValueCache.get(ColumnValueSegmentPruner.java:493)\n\tat org.apache.pinot.core.query.pruner.ColumnValueSegmentPruner.pruneEqPredicate(ColumnValueSegmentPruner.java:287)\n...\nCaused by: java.lang.IllegalArgumentException: Cannot convert value: 'HIQQEXWKW53RKN4W6XXC4Q232SYNZ3SZANVZZSUME5B5PRGXL663UAQA' to type: BYTES\n\tat org.apache.pinot.spi.data.FieldSpec$DataType.convertInternal(FieldSpec.java:522)\n\tat org.apache.pinot.core.query.pruner.ColumnValueSegmentPruner.convertValue(ColumnValueSegmentPruner.java:467)\n\t... 20 more","errorCode":200},{"message":"QueryExecutionError:\norg.apache.pinot.spi.exception.BadQueryRequestException: java.lang.IllegalArgumentException: Cannot convert value: 'HIQQEXWKW53RKN4W6XXC4Q232SYNZ3SZANVZZSUME5B5PRGXL663UAQA' to type: BYTES\n\tat org.apache.pinot.core.query.pruner.ColumnValueSegmentPruner.convertValue(ColumnValueSegmentPruner.java:469)\n\tat org.apache.pinot.core.query.pruner.ColumnValueSegmentPruner$ValueCache$CachedValue.ensureDataType(ColumnValueSegmentPruner.java:526)\n\tat org.apache.pinot.core.query.pruner.ColumnValueSegmentPruner$ValueCache.get(ColumnValueSegmentPruner.java:493)\n\tat org.apache.pinot.core.query.pruner.ColumnValueSegmentPruner.pruneEqPredicate(ColumnValueSegmentPruner.java:287)\n...\nCaused by: java.lang.IllegalArgumentException: Cannot convert value: 'HIQQEXWKW53RKN4W6XXC4Q232SYNZ3SZANVZZSUME5B5PRGXL663UAQA' to type: BYTES\n\tat org.apache.pinot.spi.data.FieldSpec$DataType.convertInternal(FieldSpec.java:522)\n\tat org.apache.pinot.core.query.pruner.ColumnValueSegmentPruner.convertValue(ColumnValueSegmentPruner.java:467)\n\t... 20 more","errorCode":200},{"message":"QueryExecutionError:\norg.apache.pinot.spi.exception.BadQueryRequestException: java.lang.IllegalArgumentException: Cannot convert value: 'HIQQEXWKW53RKN4W6XXC4Q232SYNZ3SZANVZZSUME5B5PRGXL663UAQA' to type: BYTES\n\tat org.apache.pinot.core.query.pruner.ColumnValueSegmentPruner.convertValue(ColumnValueSegmentPruner.java:469)\n\tat org.apache.pinot.core.query.pruner.ColumnValueSegmentPruner$ValueCache$CachedValue.ensureDataType(ColumnValueSegmentPruner.java:526)\n\tat org.apache.pinot.core.query.pruner.ColumnValueSegmentPruner$ValueCache.get(ColumnValueSegmentPruner.java:493)\n\tat org.apache.pinot.core.query.pruner.ColumnValueSegmentPruner.pruneEqPredicate(ColumnValueSegmentPruner.java:287)\n...\nCaused by: java.lang.IllegalArgumentException: Cannot convert value: 'HIQQEXWKW53RKN4W6XXC4Q232SYNZ3SZANVZZSUME5B5PRGXL663UAQA' to type: BYTES\n\tat org.apache.pinot.spi.data.FieldSpec$DataType.convertInternal(FieldSpec.java:522)\n\tat org.apache.pinot.core.query.pruner.ColumnValueSegmentPruner.convertValue(ColumnValueSegmentPruner.java:467)\n\t... 20 more","errorCode":200}]

I don't see a way to set the value as a byte in the prepared statement.Any thoughts?

Hi, Did anyone know How to add new Json column name to the existing table

Just posted a job in #C034Z8216JJ if anyone is interested! https://apache-pinot.slack.com/archives/C034Z8216JJ/p1661188057990719

Hi Team , Is there a way for calling a query inside UDF and get result of that query? Any Options with ScalarFunction Of UDF to call the query manually using UDF() function?

Hi @Jeyaraj Vellaisamy, we want to restrict this workspace to apache pinot related questions.. We have a separate workspace for StarTree products you can sign up here: https://join.slack.com/t/startreecommunity/shared_invite/zt-1e86u78x3-~Mtt8DBCjq_ePjFZLfNzig

We will be happy to help you there

{
  "code": 400,
  "error": "Groovy filter functions are disabled for table config. Found 'Groovy({eventnumber ==  1}, eventnumber)'"
}

Hi team , I have done all the controller/broker conf in https://github.com/apache/pinot/pull/8711 and tried this and getting the same above error Can Anyone help me in this?

Hi, I am trying to grab JVM related metrics for Pinot components using helm. https://github.com/apache/pinot/blob/master/kubernetes/helm/pinot/values.yaml#L102 Can anyone point me to as to what params I need to add so that GC related metrics are available on K8s metrics. Thanks !

Did anyone worked with managing Pinot via Terraform? Not deployment of infra, but deployment of pinot resources like TableSchemas and configs?

No pagination for group by queries yet https://apache-pinot.slack.com/archives/CDRCA57FC/p1661454626801039

Hello! Did anyone had segment issues ingesting realtime? something like this: "ingestionStatus": { "ingestionState": "UNHEALTHY", "errorMessage": "Did not get any response from servers for segment: student__0__0__20220823T0146Z" }

Hey! How does host name resolution happen for the ingested records from kafka? Edit: I want to know on which servers does the records go based on the configs of the table. Based on what table config params data is distributed among servers.

Is it possible to manually add segments to a table that have already been uploaded to the deep store without uploading each segment to the controller? Basically I want to send just the metadata to the controller to add the segments to the table