Hello Louis, Apologies for the late response.
I haven’t run this exact setup end to end yet, but the pattern makes sense.
For HTML pages, I’d usually do both if the data is important: store the raw HTML in S3 as a bronze/raw layer for replay, debugging, and audit, then extract structured JSON or CSV fields for the clean layer that Databricks actually consumes. If cost or compliance is a concern, then I’d only keep the extracted structured data.
For Apify to S3, the easiest route is either using Apify’s S3 Uploader Actor after a run succeeds, or triggering a webhook that moves the default dataset into S3. From there, Databricks Autoloader can pick it up.