Hi all, I've set up daily batch ingestion for a ta...
# troubleshooting
j
Hi all, I've set up daily batch ingestion for a table that I have configured with
"segmentIngestionType": "REFRESH",
, but as I'm generating the data with BigQuery, the number of output partfiles may vary occasionally, and I'm seeing that if we export 16 files on day 1, it'll load 16 segments.... but if we export 15 files on day 2, it'll only replace 15 segments and leave the 16th segment from day 1, leaving duplicate/bad data in the table. are there nice ways of handling this problem? My plan was to set up a task that reads the existing number of segments and manually deletes any additional segments, but there would still be some small amount of time where there would be duplicate data so if there are better ways, I'm all ears
j
thank you -- this looks like it'll do the trick. About to try it
it looks like it may have succeeded, but is it normal to have both old and new segments listed in the table view like this? Based on the zk lineage, it seems like it knows which are old and which are new
and the old segments have a
"segment.refresh.time"
prop in their metadata
m
It needs to know that to ensure that it doesn’t route the query to both sets
j
okay, so these are all signals it was successful. cool!
I was worried it might be reading both old and new segments (and return duplicate data), but that doesn't seem to be the case