A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hi guys, its when i do the ingestion, if a table its delete, to remove it from datahub?

i create a view in bigquery, run the ingestion, after that  i deleted the view and re run the igestion. The view is still in datahub :disappointed:

<https://datahubproject.io/docs/metadata-ingestion/source_docs/stateful_ingestion/>
are the settings applied for stateful ingest?

&gt; are the settings applied for stateful ingest?
<@U01TCN40JKV> can you elaborate? don't understand the question

only when stateful ingestion is set in the yaml, then entities will be removed when they're removed right

<@U027B7R23J4>
```DataHub CLI version: 0.8.24.0
Python version: 3.8.2 (default, Nov  4 2020, 21:23:28)
[Clang 12.0.0 (clang-1200.0.32.28)]```

<@U01TCN40JKV>
``` curl <http://localhost:8080/config>
{
  "models" : { },
  "versions" : {
    "linkedin/datahub" : {
      "version" : "v0.8.26",
      "commit" : "3668de85e041836d743121701cd63a8504fa327a"
    }
  },
  "statefulIngestionCapable" : true,
  "retention" : "true",
  "noCode" : "true"
}```

the strange think its that i dont see the lineage also
This is the recepie
```
source:
    type: bigquery
    config:
        project_id: &lt;project&gt;
        include_table_lineage: true
sink:
    type: datahub-rest
    config:
        server: '<http://localhost:8080>'```

<@U02QVUKE9N1> As <@U01TCN40JKV> mentioned you need to enable stateful ingestion as described in <https://datahubproject.io/docs/metadata-ingestion/source_docs/stateful_ingestion>

its true
```curl <http://localhost:8080/config>
{
  "models" : { },
  "versions" : {
    "linkedin/datahub" : {
      "version" : "v0.8.26",
      "commit" : "3668de85e041836d743121701cd63a8504fa327a"
    }
  },
  "statefulIngestionCapable" : true,
  "retention" : "true",
  "noCode" : "true"
}```

statefulIngestionCapable, this one , right?

there is some settings in the yaml to add

like this?
```source:
    type: bigquery
    config:
        project_id: &lt;project&gt;
        include_table_lineage: true
        stateful_ingestion.enabled: true```

i have not used it before:sweat_smile:, just pointing you to the doc

seems like it means you need a previous run with it enabled first, though