This message was deleted.

I think the easier is to use the SQL api to import using append mode https://docs.imply.io/latest/druid/multi-stage-query/reference/#insert if you are sure that all data is, in fact, a backfill, then you can do then in parallel

Appending would not duplicate data if backfill overlaps realtime for some time?

yes, it will! you can discard (at least on kafka) all record prior to a epoch[ at the real-time ingestion], and add a where condition on the select for the append [batch backfill]

whats the difference from sql api and index_parallel?

the sql ingestion is a new (somewhat easier) way to import data from multiple sources, but nothing prevents it from running just from one s3 it has a nice UI for the progress, but in the background, it's managing and issuing index_parallel tasks

so if the realtime has a daily query granularity and monthly segment granularity, the rejection period might be tricky

oh, you can also include a filter in the kinesis ingestion, let the where in the definition until the overload is synced (I don't know how it works, it like kafka, with offsets?)

I would need to check the filter in the kinesis ingestion

what's the format? csv, json ?

it is available in json and parquet, I have found parquet to be better in performance for ingesting

seems like mine would fall into Insert with Rollup section but with an interval I guess

you want to rollup? eg: group all records within some some period into just (hopefully) one row ?

yeah my current index_parallel spec might be a bit complex to translate into sql since I also have some transform specs, but for the granularity spec:

"granularitySpec": {
                "type": "uniform",
                "segmentGranularity": "MONTH",
                "queryGranularity": "DAY",
                "rollup": true,
                "intervals": [
                    "2023-02-15T00:00:00.000Z/2023-02-22T00:00:00.000Z"
                ]
            },

if is a lot of data, and do not use rollup, maybe you be interested in the 'Range' compaction, it's a 2 step compaction, but it's the best of all worlds

Also new to me but I think given we use rollup that might no apply

it only makes sense if your segments are so big that in just one segmentGranularity you have more than one segment file

but also in the realtime ingestion we have for that particular datasource, we also have rollup and same segmentGranularity and queryGranularity