Jonathan Baxter
08/21/2025, 5:08 PM"segmentIngestionType": "REFRESH",, but as I'm generating the data with BigQuery, the number of output partfiles may vary occasionally, and I'm seeing that if we export 16 files on day 1, it'll load 16 segments.... but if we export 15 files on day 2, it'll only replace 15 segments and leave the 16th segment from day 1, leaving duplicate/bad data in the table.
are there nice ways of handling this problem? My plan was to set up a task that reads the existing number of segments and manually deletes any additional segments, but there would still be some small amount of time where there would be duplicate data so if there are better ways, I'm all earsMayank
Jonathan Baxter
08/21/2025, 6:20 PMJonathan Baxter
08/21/2025, 8:13 PMJonathan Baxter
08/21/2025, 8:14 PM"segment.refresh.time" prop in their metadataMayank
Jonathan Baxter
08/21/2025, 8:21 PMJonathan Baxter
08/21/2025, 8:22 PM