Hi! I'm trying to ingest some metadata fro redshift to datahub. But 2 things are happening: 1. When I try to allow 55 tables and profilings from my db, the task (on airflow) take several minutes and i receive different erro logs like:

Pipeline finished with failures
'downstream_total_latency_in_seconds': 9783.199817}
'info': {'message': "('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))"}}],
failures': [{'error': 'Unable to emit metadata to DataHub GMS',
'warnings': [],
{'records_written': 106,
Sink (datahub-rest) report:
                     'query_exceptions': 3}}
                     'queries_combined': 54,
                     'combined_queries_issued': 54,
                     'uncombined_queries_issued': 57,
  'query_combiner': {'total_queries': 108,
  'soft_deleted_stale_entities': [],
...

2. I can't find a way to ingest exclusively 1 table's profiling. That's why: Allowing like:

profile_pattern:
      allow:
        - "^db.schema.table\$"

Column level profiling is filtered Allowing like:

profile_pattern:
      allow:
        - "db.schema.table"

Other tables with names like "table" are profiled and they're not in datahub. Someone can save me?

1. To get around the latency issue, you can start by setting

profiling.turn_off_expensive_profiling_metrics

as documented here. 2. This needs a minor code change here to also check against the

table_pattern

. Please let me know if you will be able to make this change. Otherwise, please open a github issue and we can get to it as soon as we can.

Thanks @helpful-optician-78938!!! And now how do I use profiling allow to uso for? Should I just use it like:

profile_pattern:
      allow:
        - "^db.schema.table\$"

?

Now you will just set the table_pattern to the specific table you are interested in. If you want to choose specific columns of the table, then you can use the profile_pattern to do so.