https://datahubproject.io logo
Join Slack
Powered by
Hi Team, we're trying to get our Redshift database...
# troubleshoot
c
Hi Team, we're trying to get our Redshift database visible with the following columns (see attached pic from the demo environment). Can you give us a hint on how to make this work?
g
@cuddly-ram-44320 those stats columns are populated by turning on the sql profiling options in the recipe config. After that’s done go ahead and re-ingest the database and that should do the trick
m
Hi @gentle-lifeguard-88494 do you know if we can remove a specific column from this model? I dont want to display the Sample Values @cuddly-ram-44320
g
@modern-garden-35830 You can turn off the sample profiling. For the postgres page (https://datahubproject.io/docs/generated/ingestion/sources/postgres/) for example it is the 
profiling.include_field_sample_values
option. Just set that to false. I don't think that excludes that column entirely - I think it will just be blank, but you can run it and verify
m
Thanks @gentle-lifeguard-88494! sounds good enough to have it empty. can we also choose in which tables we would like to apply it? or it will impact all tables? @cuddly-ram-44320
g
No problem @modern-garden-35830! I don't see that option, but you can double-check here: https://datahubproject.io/docs/generated/ingestion/sources/postgres/
@modern-garden-35830 It does look like a function exists internally in the open source code to profile a single table: _generate_single_profile - could be worth checking out. It's in ge_data_profiler.py
m
@gentle-lifeguard-88494 thanks for assistance! highly appreciated!
c
Hi @gentle-lifeguard-88494 is this relevant only for postgress or for redshift as well?
currently the ingestions are getting hanged on Pending state and not get completed, please see our XML: 
source:
type: redshift config: host_port: 'vpce-************' database: ******* username: '${j************_user_name}' password: '${**************_password}' include_tables: true include_views: true options: connect_args: sslmode: prefer sslrootcert: null profiling: enabled: true stateful_ingestion: enabled: false include_field_sample_values: enabled: false sink: type: datahub-rest config: server: 'http://***********************.cluster.local:8080'
also tried with : 
include_field_sample_values: true
g
@modern-garden-35830 I actually found an option for redshift to use regex to filter to certain tables: profile_pattern - just set in the recipe @cuddly-ram-44320 Here is the page for the redshift integration: https://datahubproject.io/docs/generated/ingestion/sources/redshift/ It looks like is 
include_field_sample_values:
is not nested under the profiling keyword. It should be below enabled:true in your example up above. Also stateful_ingestion should be 
stateful_ingestion.enabled: true
instead of what you have there. So something like this:
Copy code
source:
    type: redshift
    config:
        host_port: 'vpce-************'
        database: *******
        username: '${j************_user_name}'
        password: '${**************_password}'
        include_tables: true
        include_views: true
        options:
            connect_args:
                sslmode: prefer
                sslrootcert: null
        profiling:
            enabled: true
            include_field_sample_values: false
        stateful_ingestion.enabled: false

sink:
    type: datahub-rest
    config:
        server: 'http://***********************.cluster.local:8080'
c
Thanks @gentle-lifeguard-88494
I will try and let you know if it solved the issue 🙏
Hi @gentle-lifeguard-88494 I fixed the syntax but the ingestion still doesn't finish and getting stuck on Pending state.
is there a log page so i can see why its getting stuck?
g
I think if you go to your 8080 docker image it should have some logs, but the only other ideas for fixes I have is when I compare my recipe to yours I see that my server points to the front-end 
"<http://localhost:9002/api/gms>"
Also, idk if you enabled authentication, but you would need a token after server if you did.
Copy code
token: ${DATAHUB_TOKEN}
Also my datahub-rest type is in quotes 
"datahub-rest"
If those things don't work, I think you should repost in troubleshoot with the updated issues or tag an acryl team member since I'm just a community member who started using datahub this month so there are definitely plenty of ppl who know more than I do. Hopefully you can get it fixed soon
2 Views
Open in Slack
PreviousNext
Powered by