Is it possible to ingest data from the latest partition of a S3 bucket instead of from all the partitions?

With which source do you want to do this?

S3 data lake

ideally, we don’t ingest all the partitions if you specify {table} in the path spec, we only get a few files for sampling

My s3 bucket looks something like that: s3:// foo/bar/sh_date={partition[0]}/*.parquet which looks like that: Foo: -Bar: - sh_date=2022-06-23/ - a1.parquet - a2.parquet - a3.parquet ….. - sh_date=2022-06-24/ - sh_date=2022-06-25/ - sh_date=2022-06-26/ … How should the path_spec_include look if I want to ingest only a few of the .parquet files from the latest partition?

Try this one: s3://foo/{table}/sh_date={partition[0]}/*.parquet

thanks