Hi guys, Can I skip to make containers (or folders) when I run ingestions by recipe? For examples, When I set like

include: ‘<s3://bucket_name/service/name={name}/date_id={date_id}/{table}/*.parquet>’
table_name: ‘bucket_name/service/name={name}/{table}

Ingestion creates containers of each date_id so that multiple datasets are created. I’d like to create only one dataset. Is this possible?

@gray-shoe-75895 could you confirm if this is possible?

Yup it’s possible - you’ll need to use the partition_key pattern in the path spec. We have some examples here https://datahubproject.io/docs/generated/ingestion/sources/s3/#example-3---folder-of-files-as-dataset-with-partitions