Hi, I want to load metadata (through datahub ingestion UI) from certain schemas and for tables with specific pattern in their names (Postgresql source). My yaml looks similar to this: schema_pattern: allow: - abc - pqr - test table_pattern: allow: - test_base_tbl - check_validations - user_info I assumed it to follow logical AND of these schemas and tables but I see that all objects from the given schemas (irrespective of the object name pattern) are being ingested. Also, is there a way to restrict the views that we ingest. (I am using include_views as True which is pulling all the views) Any sample on how to provide schema and table regex pattern that can be used in conjunction will help. Thanks

Ther is a

view_pattern

property where you can add allow/deny patterns the same way as with tables.

Thank you for your response. How do I load metadata of tables with a specific pattern in their names, say I want to ingest tables from 5 schemas with "test" in their names, expected this to work but it is loading all tables: schema_pattern: allow: - abc - pqr - xyz table_pattern: allow: - test

ideally it shouldn’t load tables which does not have test in their name

no, the schema name doesn't have test string, just the table names has

Does datahub ingest tables which does not have the

test

string in the table name?

My source is posgres and I have partitioned tables. Each partition is being treated as a separate object and I want to load only the table metadata and ignore it's partition's metadata. (as the columns are same for table and it's partitions) All the partitions have the pattern "<table_name>_1_prt_<partition_date>" in the name. Below is my yaml with table_pattern.deny as "1_prt" to exclude them. -------------- source: type: postgres config: host_port: 'xxx.xx.xxxx.net:5432' database: vds_prd username: my_user_name password: '${db_pwd}' schema_pattern: allow: - my_schema table_pattern: deny: - 1_prt include_tables: true include_views: true profiling: enabled: false sink: type: datahub-rest config: server: 'http://datahub-gms:8080' ---------------- From the ingestion log, I see tables with string "1_prt" getting ingested, excerpt below: --------------------------------------- (urnlidataPlatform:postgres,vds_prd.my_schema.xxx_1_prt_232,PROD)\n' '[2022-04-04 104641,942] INFO {datahub.ingestion.run.pipeline:84} - sink wrote workunit ' 'vds_prd.my_schema.xxx_xxx_1_prt_232\n' '[2022-04-04 104644,019] INFO {datahub.ingestion.run.pipeline:84} - sink wrote workunit ' 'container-urnlicontainer5a495cd278d6547e15bf9bd46197c09d to urnlidataset(urnlidataPlatform:postgres,vds_prd.my_schema.xxx_xxx_1_prt_20150701,PROD)\n' '[2022-04-04 104644,037] INFO {datahub.ingestion.run.pipeline:84} - sink wrote workunit ' 'vds_prd.my_schema.xxx_xxx_1_prt_20150701\n' '[2022-04-04 104646,050] INFO {datahub.ingestion.run.pipeline:84} - sink wrote workunit ' 'container-urnlicontainer5a495cd278d6547e15bf9bd46197c09d to urnli:dataset

Please, can you try

.*1_prt.*

?