Hey folks! <@U0840FMQY7P> just wrote up an *eight ...
# general
k
@Fasih Khatib I want to understand the created_at part a bit more and its impact we use debezium but it is a zoned timestamp so we configured the datatype as STRING with simpledateformat
Would like to know more on that part and using it with segment pruner
f
Hey @Kasinath Reddy, in my example table,
created_at
is a
TIMESTAMP
column without timezone. In the JSON that's generated by Debezium, it's emitted as milliseconds since epoch. I use this to specify the
timeColumnName
for my table's configuration. Additionally, I use it to create a range index which can be used to efficiently filter slices of time. For example, when using the
ago
function.
As for pruning,
time
-based pruning can happen using the column specified in the
timeColumnName
. Relevant docs here. https://docs.pinot.apache.org/configuration-reference/table#routing
Does this answer your question?
k
Nope @Fasih Khatib my question is I am using debezium zoned timestamp i.e it is a SIMPLE DATE FORMAT and how timeColumnName work interms of partitioning
I understand that by specifying time in pruning but can you specify to partition on basis of Date and etc ? If possible my time column can be simple date format or it has to be epoch ?
f
In case of timestamp with timezone, Debezium emits that as a string. I believe that you can convert it to an epoch again using one of the field-level transforms.
k
Transformation is not a issue want to understand how the pruning would work so that i can take a call whether to store the created_at as epoch or keep it string with simple date format
f
The documentation for the table's configuration show that
timeType
is a number so my understanding is that you'd need it as an epoch. https://docs.pinot.apache.org/configuration-reference/table#real-time-table
k
Can't find that part in the documentation
Copy code
The name of the time column for this table. This must match with the time column name in the schema. This is mandatory for tables with push type APPEND, optional for REFRESH. timeColumnName along with timeColumnType is used to manage segment retention and time boundary for offline vs real-time.
All I see is this alone
f
If you look at the config of the example listed above, you'd find the following section in the configuration.
Copy code
"segmentsConfig": {
    "timeColumnName": "daysSinceEpoch",
    "timeType": "DAYS",
This indicates that time column
daysSinceEpoch
is numeric.