It does not seem that concatenation is supported for multi-valued column. Is there good reason not supporting it? If not, I will create the jira item for it. Let me know if there has already been one.

Hi Team, can some one help me on some configuration in UPSERT.

APPEND

and

UNION

strategies. how to configure it?

Can anyone tell me for the hosted option of Apache Pinot what is "StarTree Cloud Credits" and how it is calculated ?

based on what I can see in

AvgAggregationFunction.java

it looks as though null handling is implemented, but it seems that

avg

is still including nulls in the count (where avg is implemented as sum/count). I confirmed this using filtering, and test data where there are only 2 rows like this:

myCol  
 ------- 
     1      
  null

No filtering:

SELECT avg(myCol) from myTable

results in

0.5

With filtering:

SELECT avg(myCol) from myTableWHERE myCol is not null

results in

1

Is this expected? Looks like a bug to me.

looks like a small change, I’ll propose a PR now

Hello, is there is any way to subscribe to table or export table data from pinot to kafka ?

Hey, quick question regarding storage expansion. We have a pinot cluster on EC2 instance(each component on a different instance). Lets say we have provisioned X GB EBS on the server instances. If a situation arises where 90% of X gets utilised, instead of adding more EC2 instances as servers, can one possibly extend the EBS alone- probably having a lambda to run an ebs expansion on that particular instance rather than adding more compute? Is this possible in pinot without any downtime?

Is there anyway to clean up old segments files from the server file system. Do all servers need all segments locally or should we look to enable tiered storage.

We also have copies in s3

Qn on schema, how can I have a datetimefield as a metric I.e. published time. It looks like RealTimeToOffline roll up will think all date time columns are dimensions and wont aggregate across digfferent received timestamps. Would be better if rollup/dedup just used the tables primaryKey field list.

general q: is there more pinot activity here or in https://startree.ai/blog/introducing-startree-community-slack?

Hi Team, I using startree cloud. I tried to connect with snowflake to sync the data and connected successfully. The table has a column

CREATED

with epoch timestamp in seconds. But Pinot gives only Epoch datatype in Milliseconds. Can someone help me to move forward?

are there docker images for recent master builds?

General question: is there any way to handle deduplication in Pinot like InfluxDB does here. Basically, I want to specify some columns in a table which need dedup. If there are multiple rows which have same values for those columns, Pinot will only keep 1 row even though the other columns have different values. Is there any way to do that in Pinot? Thanks.

I am observing inconsistent results in my Pinot select Query.We are using upsert.Can someone help me with this

@channel I am getting this error after building pinot docker image and running it:

All illegal access operations will be denied in a future release
2023/01/19 12:25:41.149 INFO [PinotAdministrator] [main] Usage: pinot-admin.sh <subCommand>

Here is the dockerFile link: https://github.com/apache/pinot/blob/master/docker/images/pinot/Dockerfile This error is because of line 66. can somebody please help me with this?

if I wanna submit a PR to the docs, should I just edit the markdown or is there some

GitBook

stuff I should do?

Happy lunar new year of rabbit 🐰 !

hi my table is in status bad.Can someone help ?

"segmentsConfig": {
 "timeType": "MILLISECONDS",
      "schemaName": "workflowEvents",
      "retentionTimeUnit": "DAYS",
      "retentionTimeValue": "1826",
      "timeColumnName": "eventTimestamp",
      "allowNullTimeValue": false,
      "replicasPerPartition": "3",
      "segmentPushType": "APPEND"
    }

I am not seeing records in the consuming segments for 1 shard of kinesis workflowEvents__1__444__20230123T2054Z

{
  "segment.creation.time": "1674507284747",
  "segment.flush.threshold.size": "5000000",
  "segment.name": "workflowEvents__1__444__20230123T2054Z",
  "segment.realtime.numReplicas": "3",
  "segment.realtime.startOffset": "{\"shardId-000000000001\":\"49632676210804516725069199056584947625655896387241377810\"}",
  "segment.realtime.status": "IN_PROGRESS",
  "segment.table.name": "workflowEvents",
  "segment.type": "REALTIME"
}

Hello, Can someone please confirm if we can apply star-tree indexing to the realtime tables? Below article says "star-tree indexing can't be applied due to the nature for data for realtime tables" https://medium.com/@surajkmth29/apache-pinot-tables-and-segments-a72dc5854876

I ingested close to 200Gb of data without providing any indexing and im assuming by default pinot used dictionary encoding to all the columns. My questions are 1. can i change the index of a couple of columns to sorted index by updating the table config and reloading the segments? 2. If yes, will it take time to reload all the segments- ingestion took around 15hrs. Does this cause any downtime to the cluster? 3. Is there any documentation on which indexing should be applied to different types of columns- like based on cardinality or filtering conditions?

👋 Wanted to check in about best-practices/configurations when using a deep store with Pinot. I'm interested to see what the best practice is around server storage setup (EBS vs. EC2 local volume) when an S3 deep store is configured for a cluster.

I see the Pinot tag being created for 0.12.0.Wanted to confirm if it is officially released ? https://github.com/apache/pinot/releases/tag/release-0.12.0

Hello Pinot friends, has anyone done some analysis on the penalty of adding "nullHandlingEnabled": true to a pretty wide and large hyrbid table? I'm doing some now and curious to see if others have found it worth setting to true.

Hello Community, We are pleased to announce that Apache Pinot 0.12.0 is released! Apache Pinot is a realtime distributed OLAP datastore, designed to answer OLAP queries with low latency use-cases. The release can be downloaded at https://pinot.apache.org/download The release note is available at https://docs.pinot.apache.org/basics/releases/0.12.0 Additional resources - Project website: https://pinot.apache.org Getting started: https://docs.pinot.apache.org/getting-started Pinot developer blogs: https://medium.com/apache-pinot-developer-blog What is Apache Pinot? (and User-Facing Analytics) Video: Intro to Pinot Video: Join Pinot Community - Twitter: https://twitter.com/ApachePinot Meetup: https://www.meetup.com/apache-pinot/ Slack channel: https://communityinviter.com/apps/apache-pinot/apache-pinot Best Regards, Apache Pinot Team

Hello, Can someone please share the maven dependency for PinotSinkFunction in apache flink? I'm unable to find it in maven repository. https://docs.pinot.apache.org/basics/data-import/batch-ingestion/flink#flink-application

Hi team, trying to start kafka using docker as per https://docs.pinot.apache.org/basics/getting-started/running-pinot-in-docker this doc.

docker run --rm -ti \
    --network pinot-demo --name=kafka \
    -e KAFKA_ZOOKEEPER_CONNECT=pinot-zookeeper:2181/kafka \
    -e KAFKA_BROKER_ID=0 \
    -e KAFKA_ADVERTISED_HOST_NAME=kafka \
    -p 9092:9092 \
    -d bitnami/kafka:latest

but can not see docker container.

Hello to all, I am trying to create a table with a startree index and a timestamp index, is it possible to use the virtual column generated by the timestamp index ($ts$DAY) in the startree index?

Hello team, quick doubt regarding data retention. I have ingested batch data into pinot. And it has data from 2022 Jan 1st till Dec 2022 31st. The retention was set to 365 days. I ingested the data last week( around 18th Jan 2023). I can still see data beyond 18th Jan 2022 in the pinot table. How does retention period works- is it from the date of ingestion or the time column in the data? If it had been the latter, i should not possibly be seeing data lesser than 18th Jan 2022. Can someone help me understand how the retention period works?