Hi, team. Is there a way to ingest data from all databases in a S3 data lake with one recipe, instead of making a recipe for each db ?

Hi @bumpy-journalist-41369, have you tried using

path_specs

array from S3 data lake config ?

I cannot find a proper example in the docs. Would you mind sharing a sample recipe using path_specs array to specify all dbs in a s3 data lake?

Sure, here is an example -https://datahubspace.slack.com/archives/CUMUWQU66/p1660639723829529?thread_ts=1660630882.336099&cid=CUMUWQU66

Thanks

Hi @hundreds-photographer-13496, I tried to ingest multiple path specs of different S3 buckets but there's a check to make sure that all path specs belong to the same S3 bucket. What's the reasons for not supporting multiple S3 buckets in path specs?