Hello, I was wondering if it's possible to configure spark (for profiling) to point to a hosted spark server (i.e. hosted on kubernetes) as opposed to requiring spark to be installed locally? Maybe creating a spark session prior to the ingestion script being run?

hi @delightful-barista-90363 which connector are you trying to use?

s3_data_lake

no that’s perfect! @dazzling-judge-80093 would you mind chiming in on this?