https://pinot.apache.org/ logo
#general
Title
# general
s

Sheetal

01/06/2022, 10:52 PM
Hello, I am using Pinot tables for an app which cannot have a down time. The tables are REFRESH tables. I am thinking of maintaining 2 table versions of current and old for each table. So that at any given time data is available and to add some logic to switch the tables being referenced at the API side once data gets refreshed. Is this is the best way to go about it? Or can I use segments with date and try to query the correct segment- something along those lines. Am I missing something here?
m

Mayank

01/07/2022, 3:31 AM
Do you mean you want to get consistent results during the time you are refreshing the segments?
@User do we have versioned upload support now?
s

Seunghyun

01/07/2022, 3:50 AM
We currently have the building blocks but we don’t have the OSS client implementation yet
s

Sheetal

01/07/2022, 4:02 AM
While updating its fine if I get the stale copy. Availability is important here. For example if the data refreshing takes an hour, I don’t want the application to be down in that refresh hour. Its fine to just show the older version
m

Mayank

01/07/2022, 4:04 AM
when data is refreshing, Pinot can still serve queries,
what do you mean by app being down? That shouldn’t happen
s

Seunghyun

01/07/2022, 5:47 AM
@User Yeah as @User mentioned, Pinot will still be available for queries but the data result will be computed partially from one version and the other partially from another version.
s

Sheetal

01/10/2022, 4:59 PM
@User but there would be a point when the segment is being downloaded that there would be a chunk of data missing right? For example, if there are on an avg say 1B records in a table at any given time- when the load is happening would total rows still be 1B?