Is the S3 dependency a hard requirement for the Kubernetes deployment at this point or is it only needed for some kind of supplemental logging?

@Joe I can test this here in a little bit and see if it complains at me for having no S3 env details set. I know when I set them incorrectly, it erred and did not start several pods.

Before 0.26.4 it has the behavior bradfair describes. In 0.26.4 we made it temporarily optional so it doesn’t fail. However, we will switch to requiring configuring cloud storage once we have s3, gcs, and minio support. This requirement will allow us to scale up our logging system and break apart some more internal components for even more horizontal scalability.

^^ i’d absolutely love if airbyte could accept a pvc as an option for a logging destination, or implement something like argo workflow’s ArtifactRepository model

We did think about a logging server where containers send their logs over the network to the server and it stores it on a large individual PVC. We can’t have all of the individual containers write directly to a mounted PVC unless it’s an NFS-style mount, which runs into many problems of its own.

From your use standpoint, is a PVC that we use directly much more useful than if we provide manifests configured to interact with a local minio instance that is using a PVC under the hood?

Nope, the ability to have a minio instance that’s backed by a PVC would cover my use cases

Ok cool. We’re planning on that being the default option, but allow you to configure a cloud storage of your choice if you want something that scales “infinitely”