We are stateful ingestion enabled. However, we would like to not show the

DEV

database on datahub but it is still showing:

pipeline_name: "snowflake_platform"
source:
  type: snowflake
  config:
    # Coordinates
    host_port: ${SNOWFLAKE_ACCOUNT}
    warehouse: "AGGREGATION_COMPUTE"

    # Credentials
    username: ${SNOWFLAKE_USERNAME}
    password: ${SNOWFLAKE_PASSWORD}
    role: "ACCOUNTADMIN"

    env: "PROD"

    profiling:
      enabled: False

    database_pattern:
      allow:
        - "SENNDERDWH"
        - "VISIBILITY"
        - "CARRIER_STRATEGY_AND_PLANNING"
        - "SHIPPER_STRATEGY_AND_PLANNING"
        - "NETSUITE"
        - "MARKETING"
        - "GLOBAL_OPERATIONS"
        - "CENTRAL_STRATEGY_AND_PLANNING"
        - "FINANCE"
      deny:
        - "DEV"
        - "ANALYST_DEV"

    table_pattern:
      ignoreCase: False

    include_tables: True
    include_views: True
    include_table_lineage: False
    stateful_ingestion:
      enabled: True
      remove_stale_metadata: True

sink:
  type: "datahub-rest"
  config:
    server: ${DATAHUB_GMS_HOST}

Are we doing something wrong?

Hi @kind-psychiatrist-76973, do you still have access to the ingestion log right after

DEV

was added to the deny list (if not, you can remove it once, run ingestion, add it back to the deny list and rerun ingestion to get the logs). In the source report of the log summary, what do you see in the

filtered

list? Also grep for

Soft-deleting stale entity

and let us know what you see.