https://pinot.apache.org/ logo
Join Slack
Powered by
Seems like a new check added: <@UK6JD1E85> ^^
# troubleshooting
m
Seems like a new check added: @Rong R ^^
👀 1
s
what to do since we are facing a delay in adding schema in our live app
r
This is actually something that's going to break so we added a check
s
should we change the date format conversion logic in our streaming app logic to make it ingested in pinot
conclusion : yyyy-MM-dd'T'HHmmss'Z'" works while :*dd/MM/yyyy HHmmss" fails* even "format": "1HOURSSIMPLE_DATE_FORMAT:dd/MM/yyyy'T'HHmmss'Z'", fails
so do i need to change to a working format @Rong R @Mayank?
r
correct
s
is there no other workaround?
before older helm version .. it was getting ingested without 'T' or 'Z' like :*dd/MM/yyyy HHmmss"*
r
no. this was actually a bug, pinot index the content for the column in one ordering, but if the column is a string type then it needs to be both lexicagraphically and in time order
so those 2 ordering needs to produce the same result. otherwise your query results will be silently wrong
it is the dd/MM/yyyy that’s the problem, not the 
T
because dd/MM/yyyy produces a different ordering comparing with its underlying timestamp
✔️ 1
s
ohh I see. . we will update the logic of date time conversion .. thanks for the clarification .. will dd-mm-yyyy work fine? @Rong R
r
nope. dd-mm-yyyy produces the same error. for example 01-03-2022 comes before 03-12-2021 lex-order but in fact it is later in epoch
s
ok so what format should i use with a slash separator or a hyphen separator .. means we need to add 'T' or 'Z' ??
without 'T' or 'Z' .. it wont work?
lex-order is something I was not aware of before
r
no. it has to be 
yyyy
follow by 
MM
follow by 
dd
.
any formatting in between is fine
has nothing to do with 
-
or 
_
, or 
/
or 
T
or 
Z
s
got it ..thanks @Rong R for the help
👍 1
r
so one non-breaking way to solve your issue is 1. keep this current column 
time
unchanged but dont indexing it as a time column, in addition 2. create another index column and pass in a transform function that transforms the time string into epoch long. then use that as your time index/ordering field
btw are you using master branch? i am not sure this change has been released yet
s
Latest Deployed Chart Version. @Rong R CC: @Sadim Nadeem
Older Chart Version.
s
also share the github branch @Shailesh Jha since @Rong R needs confirmation if its from master branch
s
It's Master Branch. Git Hub Link: https://github.com/apache/pinot
✔️ 2
Open in Slack
PreviousNext
Powered by