Hi guys, I have deployed Airbyte to Kubernetes to collect data from some sources to my local volume. However, Local Csv and Json Connectors are not working as stated in their description pages. Can someone guide me to do this? Is the Minio a safe solution?

@[DEPRECATED] Marcos Marx turned this message into Zendesk ticket 2522 to ensure timely resolution!

Hey there, yes Minio is the closest alternative to a local storage destination available for Airbyte in this case. It's compatible with the S3 destination connector: https://docs.airbyte.com/integrations/destinations/s3 So it should be technically possible to use the airbyte deploy minio, however since this instance will be storing app data there might be performance concerns especially around storage capacity. I would recommend using a separate Minio instance if you can.

The project still in the mvp stage. For now the current Minio will be enough for our concerns. Thanks a lot!

Pods do use local volumes to store secrets, but I spent a bit more time looking into Minio and wanted to give some clarification. We don't seem to have docs explicitly saying what is stored in Minio, but I did find this PR: https://github.com/airbytehq/airbyte/pull/17322 It looks like Minio is only used for default log storage, so if you have other logging configured you can actually disable it with the environment value

global.logs.minio.enabled: false

. Looks like it is way less necessary than I originally thought. These config values can be found in our helm chart readme: https://github.com/airbytehq/airbyte/tree/master/charts/airbyte Seems like good news for your use case! Please let us know if you're successful, your idea is really interesting and it would be great to know if it works well or if there is anything more to iron out.