set the channel topic: Community-wide announcements. Please use #C011C9JHN7R for work-based matters.

I created #C013WKLT5T7 to discuss Pinot PR and Issues, please join the channel if you are committer/contributor or want to learn more about Pinot

<!here>: I am quite new to Pinot, Have couple of questions - 1. While query data, I could see that those are case sensitive.. is it by product or I am missing anything here. Most of the query language are case insensitive. 2. While defining schema I was getting error when using STRING under metricFieldSpecs - documentation says that STRING is supported.

@User 1. table name, column names are case sensitive in Pinot. This decision made far in past so I don’t have a full context on why we chose to support case sensitive over case sensitive. Maybe @User @User has the context on this decision. 2. Why do you want to use

STRING

for metric column? There’s an exceptional case where we use STRING for metric column (I can think of storing serialized hyperloglog object, which is used for computing approximate distinct count)

Thanks for prompt reply @User: As I mentioned I am quite new and was trying to explore few things, here I did define metric field as STRING but that was not working thought posting here. So far I dont have usecase for STRING. Another question: In query any option available for not in/not exists kind of option ?

For Ex: I am trying following - select ORDER_NO from my_test where ORDER_NO not in (SELECT ORDER_NO FROM my_test where status = '1100')

we do have a support for

NOT IN

but only for the list of simple values

e.g. ORDER_NO NOT IN (111,222,333)

we don’t support nested query

@User case insensitive is supported if you have a schema and you set the cluster config enable.case.insensitive.pql=true

For nested queries and joins, use presto Pinot connector https://docs.pinot.apache.org/integrations/presto

Thanks @User

Question about deploying pinot: if we use the gcs filesystem for the controller deep storage is there any need to have persistence enabled and attach a pvc to the controller?

No

Also we noticed with realtime ingestion that if we set

realtime.segment.flush.threshold.time

to 24h and

realtime.segment.flush.desired.size

to 150M, for small data sets it never seems to flush, just gets the data from kafka again. We didn't see this behavior in pinot-0.2.0. Did something change? Are the 2 options AND'd together or OR'd together? We're using pinot-0.3.0 vanilla

OR'd

Is there any way to create pinot cluster in memory for testing purposes?

As part of junit/integration test or standalone

See what we do in ClusterTest

Does pinot have a now() or current_timestamp function?

What's the best way with Pinot to make sure my offline segment writes are idempotent? E.g. If I have a job that writes old data, how do I make sure I replace the old segments? I see

jobType

has upload only. What's the identity for a segment? How should I split my input data?

if the segment name is the same, it will replace the old data

split your input data by time

Got it. I see

segmentNameGeneratorSpec

in the docs.

yep

Cool, thanks!

default is startDate_endDate_<sequence_id>

we derive startDate and endDate automatically from the timecolumn in the data

What's

sequence_id

?

it allows you to have multiple segments per day