Hi Team, I have a quick question regarding external storage options in Lightdash — specifically S3, GCS, and MinIO. Based on the documentation, it seems these storage options are used to save results such as images or SQL query outputs. Are these results used for caching purposes? For example, when accessing the same dashboard or chart multiple times, does Lightdash retrieve the results from external storage, or does it regenerate them each time? Our team is planning to deploy Lightdash on a Google Compute Engine (GCE) instance. Since it's already a dedicated instance, we’re considering using MinIO instead of GCS, as it would avoid the need to configure S3 compatibility. However, I’m concerned about one thing: if the GCE instance is recreated, the MinIO deployment on it would also be lost. Would that cause any issues for Lightdash, or is that an acceptable setup? Thanks in advance for your help!

👋 Thanks for your message - someone from the Lightdash support team will get back to you asap. Feel free to add any additional context to the thread here in the meantime (screenshots, app version if you're self-hosting etc.).

@Nguyễn Mạnh I was successful with launching Lightdash with GCP buckets with the new Interoperatibility feature which lets you generate access token and secret for the bucket to be used as a S3 service. It was quick to implement so I very much recommend. Let me know if it helps.

@Kabir Gaire Thanks I have achieved something similar, but due to security concerns, I'm thinking of using MinIO. Do you believe external storage can be used for caching, or is it only suitable for storing temporary query results?

@Nguyễn Mạnh I just checked again, apparently lightdash uses postgres for caching and s3 for exports. So your caches are going to be in sql databases, not object storage like s3 or minio

@Kabir Gaire Thanks, I've never heard about that. Is that different from the caching mentioned in this documentation?https://docs.lightdash.com/references/caching#chart-and-dashboard-results-caching

Hello @Nguyễn Mạnh 👋, For results on charts and dashboards, Lightdash uses S3 to stream query results through but not as a cache by default. So pages of query data are stored there temporarily as they are sent to the frontend. This enables running large queries where it's not possible to load all data into memory at once. If you run another query, it will fetch the results from the warehouse again, streaming through S3, but not using the same files. The caching described on the page above is only available in Lightdash cloud, where the stored s3 results can be used as a cache and refetched from there without running the query again. For fitler values, the cache is in s3. And for those, caching is available in all deployments. Nothing is cached in postgres. I hope that helps.

@Marshall Agnew You saved me, thanks

Great, happy to help!