Hello, We are doing redshift ingestion with below source:
----------------
source:
type: "redshift"
config:
platform_instance: "dataland"
env: "DEV"
username: "${REDSHIFT_USER}"
password: "${REDSHIFT_PASSWORD}"
host_port: "
dataland-internal.big.dev.scmspain.io:5439"
database: "dwh_sch_sp_db"
schema_pattern:
deny:
- .*_mgmt$
table_pattern:
deny:
- .*dim_ad_normalization$
- .*dim_api_client$
include_table_lineage: False
include_views: False
stateful_ingestion:
enabled: True
remove_stale_metadata: True
profiling:
enabled: true
limit: 1000
turn_off_expensive_profiling_metrics: True
profile_pattern:
allow:
- ^dwh_sch_sp_db.motos
deny:
- .
tmp.
- .
temp.
options:
connect_args:
sslmode: prefer
-------------
after our ingestion ran successfully we can see in logs that tables defined with source.deny are soft deleted
--------------LOGS-----------------------
'soft_deleted_stale_entities': ['urn
lidataset:(urn
lidataPlatform:redshift,dataland.dwh_sch_sp_db.pro_infojobs_es.dim_api_client,DEV)',
'urn
lidataset:(urn
lidataPlatform:redshift,dataland.dwh_sch_sp_db.infojobs_es.dim_api_client,DEV)'],
'query_combiner': {'total_queries': 676,
'uncombined_queries_issued': 379,
'combined_queries_issued': 82,
'queries_combined': 297,
'query_exceptions': 0},
'saas_version': 'PostgreSQL 8.0.2 on i686-pc-linux-gnu, compiled by GCC gcc (GCC) 3.4.2 20041017 (Red Hat 3.4.2-6.fc3), Redshift 1.0.38698',
'upstream_lineage': {}}
Sink (datahub-kafka) report:
{'records_written': 8252,
'warnings': [],
'failures': [],
'downstream_start_time': None,
'downstream_end_time': None,
'downstream_total_latency_in_seconds': None}
----------------------------------------
but the tables are still visible from UI and the mysql table has 2 entry first with {"removed":true} and again with {"removed":false}, can you please explain what is happening wrong here.