I have a problem with stateful ingestion. It wasn’t enabled when we initially ingested the datasourse (Hive), I enabled it now, but Datahub still displays tables that are long gone. It says ‘Last synchronized 4 months ago” next to it, so we now that’s when they last existed, but it still doesn’t soft-delete them:( What can I do to clean-up all old deleted tables? I’m on 0.9.6.1 and my ingest recipe looks like this:

sink:
    type: datahub-rest
    config:
        server: '***'
source:
    type: hive
    config:
        host_port: '***:10000'
        env: PROD
        username: ***
        include_tables: true
        include_views: true
        stateful_ingestion:
            enabled: true
            remove_stale_metadata: true
transformers:
    -
        type: set_dataset_browse_path
        config:
            replace_existing: true
            path_templates:
                - /ENV/PLATFORM/DATASET_PARTS
pipeline_name: 'urn:li:dataHubIngestionSource:***'

Hm this recipe looks slightly off to me

The recipe looks off but I think the main issue is that pyhive runs the same show tables command when it lists views and tables. This might be why it doesn't return views. Presto-on-hive can be a workaround for this as it returns views properly.

Hmm what else seems off to you in the recipe? I'd appreciate tips

Stateful ingestion configuration block^

Ahh, sorry, I mixed up the threads :(

ok I also made a little mistake while copying recipe for the message, but now I corrected it and it's the way it actually is

@dazzling-judge-80093, We are also facing same issue where in tables are not getting soft deleted even though the 'stateful ingestion' was enabled for a while in recipe as well as in gms. currently we are on datahub v0.10.0 and cli version on 0.9.6.3

@fresh-cricket-75926 can you bump the cli version? 0.9.6.3 is quite old

@dazzling-judge-80093, thanks for the reply , i just upgraded cli to v0.10.0 just to rule out version issue and still facing same issue .

which source are you using?

we are using redshift