Does Druid has ‘alias’ support for data source name? We are facing something very similar to this issue https://groups.google.com/g/druid-user/c/_rKQofO4QK8 AFAIK, Druid doesn’t support renaming table. Is it possible to create/update alias name for Druid datasources? Also, the link above mention view support in Druid SQL (e.g: create view data_src as select * from data_src_v1). Does this exist in Druid? I never heard of this….

I recall a "view" construct for Hadoop related ingestion, but not necessarily within core Druid. I only use table name aliases in my SQL queries directly. If this is just a naming convention/readability issue, you could always add a CTE at the top of each of your queries to rename tables and columns for easier reference throughout the rest of the query. if these suggestions don't help, then please provide more detail on the issue, and the query you want to write. Thanks. John

I only use table name aliases in my SQL queries directly. If this is just a naming convention/readability issue, you could always add a CTE at the top of each of your queries to rename tables and columns for easier reference throughout the rest of the query.

This won’t work. The goal is that we want to restate (reingest) the whole datasource. This is because some existing value changed. Because Druid will load (segment onto historical) for each time partition (per the segment granularity) independently, our new data will show up at different time. Since the client will continue to make queries to our table, there will be period where data keeps changing (until all new time partitions are loaded). We want to be able to switch to the new ingested data only after all new time partitions are loaded. One idea is to ingest to a new datasource (and hence not overwrite the existing datasource). After the new datasource is ingested and all new time partitions are loaded, we would update the alias to point to the new datasource. The client query will only use the alias (and not the actual datasource names as that will change after each reingestion)

The datasource alias concept has been discussed in the past but to my knowledge never implemented. In the meantime the only method I can think of to atomically switch old/new data is to reingest with segmentGranularity=ALL, which might be doable if your datasource isn't too big.

I have similar requirements which we call backfilling the datasource. Generally speaking our process looks like this: 1. Create new datasource named

tmp-datasource

and ingest the data into it 2. Check that everything looks good in

tmp-datasource

3. Create a new datasource named

backup-datasource

and use a Druid Re-Index task to make a copy of the original datasource 4. Check that everything looks good in

backup-datasource

5. Use a Druid Re-Index task to overwrite

datasource

with

tmp-datasource

6. Check that everything looks good in

datasource

7. After some period of time drop both

tmp-datasource

and

backup-datasource

If Druid supported a rename feature, this process could be simplified (less error prone and use less disk space): 1. Create new datasource named

tmp-datasource

and ingest the data into it 2. Check that everything looks good in

tmp-datasource

3. Rename

datasource

to

backup-datasource

4. Rename

tmp-datasource

to

datasource

5. After some period of time drop

backup-datasource

I am thinking of more of a alias or a view function

I suggest you post a request on the GitHub issues list. I did a quick check and didn’t see anything there related to aliases.

Yes, agreed, there is a window during the re-index where segments are loaded for some, but not all, time windows. A rename or alias feature would solve that.

Just wanted to reiterate ... if you use ALL segment granularity, then all segments in the datasource are included in the same core partition set, which means the queries won't use the new set of data until all segments (in that core partition set) are loaded and announced by the Historicals.

Yep. But if the datasource is large and/or contians a large period of time, the query performance will be impacted

You will lose the time-based pruning ... but if you range partition you can still get significant filter-based range pruning. It's just an option, won't be beneficial in all cases, but may in some ...

I concur. I have datasources with TBs of data and a simple rename or alias would be much preferred to copying all of the data over the network hogging disk and bandwidth.