@User i am probably missing something. Want to validate my understanding.. if we have x pinot servers running, since threshold has met a new pinot server has to be added, are you saying we should not use HPA file ( horizontal auto scaling) or v will have hpa and using lifecycle hook of a container we identify the trigger and automate running of the rebalance script? Or is there any effective way to get this done ?

you can use definitely use HPA (make sure you have a good handle on the metrics based on which you are setting the auto-scaler and avoid thrashing). Rebalance is just a rest api call, you invoke this using lifecycle hook or contribute the feature to Pinot to trigger rebalance on node additions. Helix provides all the callbacks you need to implement this and can be enabled based on a cluster config

Thanks @User for clarifying. Will reflect and get back

Hello all. Last meetup was my first one. There were few roadmap items towards the end and one of it was Kinesis connector. I have some experience with AWS Kinesis Client Library. I started exploring the src code. If anyone has any pointers and/or high-level requirements, please share.

@User is working on it

Hi we realized that we didnt configure column size and it seems to be using default of 512. How can i update it ?

Column size for ?

seems like String

There's a

maxLength

field in the FieldSpec

Yes string type

Can i update config via REST api ?

Can you elaborate the use case?

@User basically, we didn't specify any size for dimensions and that ended up with default size of

512

for its length. Now, we're wondering if we can increase the max size for a dimension and if so what config?

You can upload the schema again with the maxLength via rest api.

Is there a limit on how much we can set ?

For example string ~ 128KB

Technically you are overwriting that limit. You definitely want to remove dictionary for this column

Could you elaborate a bit on what you are storing on this column and how you plan to query it?

@User right now the requirement is more like putting a blob, which will only be used in selection for display purpose. We think Pinot isn't probably optimized for this and could potentially move that out but until then, we want to have the blob not truncated to 512 chars

One other thing to note is computing the space overhead. If it is standard english ASCII, then size of each value in bytes is equal to number of characters since we use UTF-8. But if the blob data has heterogeneous characters going into the higher ASCII range then each character in UTF-8 is likely to take 2 or even 3 bytes

<!here> The recording of the 5/5 meetup with Uber is now available, enjoy! (recorded as separate videos)

Added this to FAQs : https://docs.pinot.apache.org/basics/getting-started/frequent-questions#how-to-set-inverted-indexes

Is there any way to create sql/pql query string from broker request?

Need to store this JSON post flattening as separate column in table

{

  "payload" : {     "tenantId" : "0",     "attributes" : {       "attr1" : "0",       "attr2" : "0"     },     "verticalId" : "",     "orderNo" : "",     "orderDate" : "",     "entityErrors" : [ ],     "orderLines" : [ {       "ft" : "",       "qty" : {         "uom" : ""       },       "lineNo" : 2,       "orderLineQuantityInfo" : [ {         "status" : "",         "statusChangeDate" : ""       } ]     } ]   } } (edited)

Any suggestion will be appreciated.. columns in tables are tenantId, attr1, attr2, orderNo,statusChangeDate etc

I think TransformFunction can be used here

this entire json as one column?

This is probably incoming event

Not in one column , each field should go to seperate columns as mentioned above. We have to flaten this before ingestion.. This Json is coming from Kafka