Do record message keys need to be strings for pinot?

Hello everyone , i am working on managing offline flows and found that if some time windows has no data ,it doesn’t get eligible segment.that is true .but it stuck into the same time window.So, my concern is to if some time windows doesn’t found any eligible segment , it should move to the next time window.

Hi, Is there a configuration attribute to write the controller/broker/server logs to a specific directory other than the default ?

Hello everyone, I have been trying schema evolution for real tables and then updating the new column with real values through Upsert. If segment is already created and is in consuming state before schema evolution and we try to update the default column value with the real values or try to insert whole new record with new column added, then the new column values are not getting reflected even after segment reload is done. Once the segment state changes from consuming to done and new segment gets created for consuming the new records ,then the real values get reflected. If the old segment is able to consume all the new records, then the real values don't get reflected, default value remains and real values for that column are lost.

Screenshot 1943-05-18 at 16.26.54.png

{"code":400,"error":"Invalid transform function

facing above error while adding groovy function

can someone help?

response: {"code":400,"error":"Invalid transform function 'Groovy((CASE WHEN visits = 1 and (first_tab = 'true' or first_tab = 'null') then 1 else 0 end), visitscase)' for column 'visitscase'"}

Hi Team, I got to know that Pinot provides support for encryption of data stored in deepstore. A crypter needs to be provided for this. But I am not able to figure out where to provide the crypter details , can anyone help ?

i have an issue with queries not being deterministic. my table schema looks like this:

"tableIndexConfig": {
        "invertedIndexColumns": [
            "user",
            "ts",
            "cell"
        ],
        "sortedColumn": [
            "user"
        ],
        "loadMode": "MMAP"
    }

query looks like this:

select user, count(*) from events where ts between {start_ts} and {end_ts} and cell between 500 and 550 group by user having count(user) >= 24 limit 10000

logs look like this:

Processed requestId=51,table=events_OFFLINE,segments(queried/processed/matched/consuming)=66/66/66/-1,schedulerWaitMs=1,reqDeserMs=0,totalExecMs=388,resSerMs=0,totalTimeMs=390,minConsumingFreshnessMs=-1,broker=Broker_172.20.0.4_8099,numDocsScanned=1039299,scanInFilter=206881916,scanPostFilter=1039299,sched=fcfs

Processed requestId=52,table=events_OFFLINE,segments(queried/processed/matched/consuming)=66/66/66/-1,schedulerWaitMs=0,reqDeserMs=1,totalExecMs=424,resSerMs=2,totalTimeMs=427,minConsumingFreshnessMs=-1,broker=Broker_172.20.0.4_8099,numDocsScanned=1039299,scanInFilter=206881916,scanPostFilter=1039299,sched=fcfs

Processed requestId=53,table=events_OFFLINE,segments(queried/processed/matched/consuming)=66/66/66/-1,schedulerWaitMs=0,reqDeserMs=0,totalExecMs=407,resSerMs=0,totalTimeMs=408,minConsumingFreshnessMs=-1,broker=Broker_172.20.0.4_8099,numDocsScanned=1039299,scanInFilter=206881916,scanPostFilter=1039299,sched=fcfs

Processed requestId=54,table=events_OFFLINE,segments(queried/processed/matched/consuming)=66/66/66/-1,schedulerWaitMs=0,reqDeserMs=0,totalExecMs=416,resSerMs=0,totalTimeMs=418,minConsumingFreshnessMs=-1,broker=Broker_172.20.0.4_8099,numDocsScanned=1039299,scanInFilter=206881916,scanPostFilter=1039299,sched=fcfs

is this expected or am i doing something wrong here?

Sorry, which part is inconsistent?

I've table created with 1 segment locally configured controller... after that i configured deepstore as HDFS and moved controller data of that table 'T' to HDFS deepstore and restarted controller & server... after that query is not serving for that table.

Table status is Good

only while querying for the migrated controller data from local to hdfs has a problem...

can someone help?

Hi All I have built pinot0.8.0 release package and deploy in our env , but I found zookeeper page can’t be open

Hello All! Is there anything special about extracting values from nested JSON object? My (stream) ingestion looks like

{
    "metadata": 
    {
        "queryId": "20210818_091933_00000_j2ma7", 
        "transactionId": "89328f28-146d-46c7-b562-f9a436e78bac", 
        "query": "select * from postgresql.bnef_930.weather limit 10", 
    }
}

My schema has the fields

{
            "name": "metadata.queryId",
            "dataType": "STRING"
        },
        {
            "name": "metadata.query",
            "dataType": "STRING"
        },

But those don't get populated. Do I need to explicitly flatten metadata? Also kind of related to this, is unnesting data suported in realease 0.7.1, or is it 0.8 only?

Hi, Iam trying to migrate existing controller data from local disk to HDFS deepstore.. reference: https://github.com/apache/pinot/issues/7275 i changed required segment metadata to respective hdfs path after migrating this /tmp/data/PinotController/{Table}/{segment} to hdfs ... and restarted server , controller as well.

while querying, its getting timedout and not working

@Xiang Fu tried to help me, but didnt worked out, can someone help migrating local data to deep store... ??

@Neha Pawar @Mayank

https://docs.pinot.apache.org/basics/getting-started/hdfs-as-deepstorage used to configure HDFS as deepstore...

like no. of threads (num of segments)

image.png

Hey, Getting this sometimes while making connection from local

sometimes its works, but most of time it doesn't and throws above error

Hey , was trying to ingest s3 hosted json file using rest api, but getting some exception in controller node