A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hello guys,
We have `redshift-usage` and `redshift`. Can we use both when ingesting in order to have stats and tables metadata ? Or it's only one or other ?
Thank you in advance !

redshift-usage adds usage stats on the top of redshift ingestion

<@UV14447EU>  If `redshift-usage` adds usage stats on the top of `redshift` ingestion, what is the difference between setting `profiling.enabled: true` in `redshift-usage` and `redshift` ? Should I, therefore, keep all the transformers and profiling options only on `redshift`, and keep `redshift-usage` yaml the most simple possible simply to add usage stats ?

with `redshift-usage`  you can’t do profiling

<@UV14447EU> But we have that option on the docs: <https://datahubproject.io/docs/generated/ingestion/sources/redshift#config-details-1>

Thanks, I think this is something we should clean from the doc

<@UV14447EU> You are very welcome ! :sunglasses:

<@UV14447EU> And for all the other options ? `stateful_ingestion`, `profile_pattern`, `schema_pattern`, `transformers`, `options.connect_args`... I've used them and the CLI accepts these keys on the yaml, but I don't know if it really works. Do they work or is just another "doc cleaning" problem ?
Thank you a lot for all your help. :)

I’m only not sure in `stateful_ingestion` the others should work I think

<@UV14447EU> But since `redshift-usage` simply adds usage stats on the top of `redshift` ingestion, what's the meaning of using all those other yaml options ?

it can filter out some usage info if you want, so for example to not collect usage for a specific schema