hello team we have 10 millions of data and want to transfer that data to other offline table. we are facing data loss. we create segment on hourly basis and push to offline table. whenn there is 10K records all data went correct. sample code in python cursor.execute(queryEventime, params) rows = cursor.fetchmany(batch_size) we are fetching by batch .. issue is that if there is 10k row its fetch all the row .. when 10 milion then we get few of rows but no exception. is it any memory setting in pinot?

Hi San, Pinot itself does not impose a low row limit when you run a SELECT without a LIMIT clause. The broker’s query response limit defaults to Integer.MAX_VALUE, and the limit-override option is disabled by default. That means the server will return all matching rows unless you configure a limit. In your Python snippet the call

cursor.fetchmany(batch_size)

will only return the next

batch_size

rows from the result set. If you only call

fetchmany()

once you will get at most one page of results (10 k rows is the default page size if you are using the Pinot cursor API). To retrieve all rows you need to loop over

fetchmany()

until it returns an empty list, or just call

fetchall()

. Hope that helps!