A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hello,

Is there Stateful Ingestion sor S3 sources?

Thank you in advance!

Unfortunately we don’t currently support stateful ingestion for s3

However, it wouldn’t be too complex to add - would you mind filing a feature request here <https://feature-requests.datahubproject.io/>

<@U01GZEETMEZ> Thanks!
<https://feature-requests.datahubproject.io/p/stateful-ingestion-for-s3>

<@U01GZEETMEZ> While S3 Stateful Ingestion isn't available, could I delete all S3 data using hard delete and then reingest everything? Would you say it's a good practice ? Would you have other solutions ?
Thank you so much !

That would be a decent workaround for the time being - the `datahub delete` cli command should work well, and you probably just want to “soft-delete” entities

Hi <@U01GZEETMEZ> do you have a plan to support stateful ingestion for s3?

Yes we definitely do - we have an internal tracking issue for adding stateful ingestion across the board, but I don’t have an ETA for that right now