Hi All 👋. New member here and new to DataHub. This looks like an amazing community. I have DataHub deployed in AWS EKS following the Quickstart Guide. It all works great in my dev environment, however I can’t get Redshift metadata ingestion to work in production. After consuming metadata from one table, it just hangs without any errors in the UI or in any of the pod logs (including prerequisites). Can this happen if it is lacking sufficient resources? I’m not certain how to debug this but I have ruled out MySQL and Elasticsearch by moving both to AWS managed services

Hey @prehistoric-knife-90526, How do you run the ingestion? Do you run from the UI or from the command line? Please, can you share your recipe as well? Normally, if you only do ingestion we only need permission for the system tables, so if it can ingest one table then it should ingest the others as well.

Hey Tamas! Thanks so much for the response. I should have updated here but I am getting help from Pedro Silva after joining office hours. I am ingesting from the UI. I think the issue is with the actions pod as it keeps restarting. My ingestion recipe is below. I will update here for future reference once I get it resolved!

source:
    type: redshift
    config:
        host_port: '${redshift_host_port}'
        database: <database>
        username: <username>
        password: '${redshift_pw}'
        include_views: true
        include_tables: true
        include_table_lineage: true
        profiling:
            enabled: false
        options:
            connect_args:
                sslmode: prefer
sink:
    type: datahub-rest
    config:
        server: '<http://datahub-datahub-gms.data-services.svc.cluster.local:8080>'