Hi there, i'm working on a new deployment, working on my first ingestion.... we have a

kafka

to

spark/scala

to

s3 parquet

datalake pipeline. I'm considering either the

s3-datalake

source or the

spark

source. My data engineer would prefer I use s3, and I tend to agree, but wondering if there are any benefits to using a spark push mechanism over the s3 pull mechanism

@gray-shoe-75895 might be able to speak to this!