Hey everyone, after every round of ingestion, I see extra rows in the backend with urns like 'urnlidataJob:(urnlidataFlow:(datahub,stateful_ingestion_pipeline_name%' I assume that they are technical items because they have 'datahub' as the Pipeline name in the 'urn' structure. But looks strange that these items are available via UI (search query: 'NOT name: *' and then check Tasks) Is it a bug that should be reported or it is expected behavior?

This is actually a new feature that we added. That data lets us show all of your CLI-based ingestion runs in the UI, alongside the datahub-managed ingestion

Hi @gray-shoe-75895, is it normal that we can see these stateful pipeline in the search result? They appear but they are not clickable. For my example, I just run a snowflake ingestion recipe:

source:
  type: snowflake
  config:
  ....
  stateful_ingestion:
    enabled: true
    remove_stale_metadata: true
pipeline_name: "snowflake_test_pipeline"

So, I should only see Snowflake containers/datasets.

Hm those are supposed to be internal-only - we should be hiding them

So fast! Thanks a lot @gray-shoe-75895 !