This message was deleted.
# troubleshootings
Slackbot
03/02/2021, 10:25 PMThis message was deleted.
x
Xiang Fu
03/02/2021, 10:27 PM
what’s the select * query results?
Xiang Fu
03/02/2021, 10:28 PM
there should be one doc per kafka event
Xiang Fu
03/02/2021, 10:28 PM
if there is only one doc, then it means pinot only consumed 1 record from Kafka
Xiang Fu
03/02/2021, 10:29 PM
also is there any exceptions from pinot server log?
j
Josh Highley
03/02/2021, 10:31 PMselect * has no query result but query response stats has totalDocs = 1, so it seems the data is there, just not being returned
Josh Highley
03/02/2021, 10:32 PMand, yes, I only sent 1 record to Kafka for the table
Josh Highley
03/02/2021, 10:58 PMno errors in the server or broker logs
x
Xiang Fu
03/02/2021, 11:05 PM
Then I somehow feel the groovy function might not be correct. @Neha Pawar is there a way we can ingest using consumption/index time as the time value?
n
Neha Pawar
03/02/2021, 11:12 PM
you could use
now()x
Xiang Fu
03/02/2021, 11:21 PM
true, can you try this function `now()`: https://docs.pinot.apache.org/users/user-guide-query/supported-transformations#datetime-functions
j
Josh Highley
03/02/2021, 11:34 PMthose transforms are on the Select. I'm running the transform on the ingestion
Josh Highley
03/02/2021, 11:35 PMmy actual transform is more complex, but I've boiled it down to the test case. putting just a long in the Groovy script doesn't work, but putting System.currentTimeMillis() does work. When it doesn't work, the record still shows up in the query stats but not in the results
Josh Highley
03/02/2021, 11:36 PMif I use a different date field, defined the same way but not acting as the table time column, then its value is populated by the transform as I expect
Josh Highley
03/02/2021, 11:45 PMI've found that when I destroy the table, Pinot still seems to remember the most recent table date-time. If I create the table again, same name and schema, then even though the table is starts empty, Pinot queries won't return records with a date-time prior to those when the table was destroyed.
Josh Highley
03/02/2021, 11:49 PMif I create the table with a different table name but same transformation, then the records will be returned by the query
Josh Highley
03/03/2021, 1:01 AMI've been able to simplify the issue somewhat, I'm going to start a new message thread
x
Xiang Fu
03/03/2021, 1:04 AM
Sure, I think that transform function can also be used in ingestion transform function
Xiang Fu
03/03/2021, 1:04 AM
for table destroy, did you also delete the schema ?
Xiang Fu
03/03/2021, 1:05 AM
then it seems indicate that Pinot caches some transform functions per table name
j
Josh Highley
03/03/2021, 1:08 AMI found that the transform doesn't matter. I can specify the column time in the data and have the same issue
Josh Highley
03/03/2021, 1:08 AMwhen I re-create the table, if I insert the same primary key with an earlier column time, the record won't be returned by a query
Josh Highley
03/03/2021, 1:09 AMit's still using the most recent column time for the record that existed in the deleted table
x
Xiang Fu
03/03/2021, 1:11 AM
Is this for upsert?
j
Josh Highley
03/03/2021, 1:11 AMyes
x
Xiang Fu
03/03/2021, 1:11 AM
i think upsert uses timestamp to figure out the latest record
Xiang Fu
03/03/2021, 1:12 AM
so only newest version of the record is counted
j
Josh Highley
03/03/2021, 1:12 AMright, but it's using the timestamp from records in the deleted table when I re-create the table with the same name
x
Xiang Fu
03/03/2021, 1:13 AM
I see, how long did you wait for the table to be deleted then recreated it? We observed some issue when recreate table in a very short time
Xiang Fu
03/03/2021, 1:14 AM
if not, then it’s very likely that some intermediate status is not cleaned up
j
Josh Highley
03/03/2021, 1:17 AMseveral seconds between deleting and re-creating. There's only a few rows in the tables (testing)
n
Neha Pawar
03/03/2021, 2:41 AM
now