https://datahubproject.io logo
Join Slack
Powered by
Hi team. I’ve added a REST API call for my Redsh...
# troubleshoot
p
Hi team. I’ve added a REST API call for my Redshift cluster, but seems that data set records do not appear - see img below I’ve set in the REST API. the following lines:
Copy code
include_tables: true
        include_views: true
        profiling:
            enabled: true
So all dataset data will be available Any idea? Cc: @modern-garden-35830
h
Hey Aviv ! Which source type are you using 
redshift
or 
redshift-usage
? While I am not sure what you mean by REST API, I am assuming you are running an ingestion recipe.
p
@hundreds-photographer-13496 hi 🙂
Yes ingestion recipe, not explicit DS from the uI
I’m using 
redshift-usage
Why?
When using 
type: redshift
it fails @hundreds-photographer-13496
Ok I see this I need to update permission, from the prerequisites 1st, before executing ingestion
h
right.
p
@hundreds-photographer-13496 but I'm not using 
datahub_user
but my user, which is admin
h
does your user have necessary permissions already ? If not, you'll need to grant them to your user first.
p
Checking in front of the db
@hundreds-photographer-13496 it is ok now. Ingestion completed BUT - I still don’t know if profiling is required Another Q - for every ingestion running - it takes only the delta from the exist datasets?
h
Enabling Profiling allows one to see metadata like number of rows, columns and column-level statistics in DataHub, as shown here - https://demo.datahubproject.io/dataset/urn:li:dataset:(urn:li:dataPlatform:snowflake,dem[…]ine.public.all_entities,PROD)/Stats?is_lineage_mode=false
However it may be expensive to compute and hence not enabled by default.
When ingestion recipe runs (pull-based), it reads all of existing datasets every time. Some metadata aspects such as usage history are processed for particular time window in the past (e.g. last 1 day which is default). At the same time, I would like to mention that, the stateful ingestion feature enables us to store state of past ingestion run and we are slowly moving towards using this state and the delta to make some features smarter. e,g, profiling, however this is not supported in redshift yet.
2 Views
Open in Slack
PreviousNext
Powered by