https://linen.dev logo
j

Joe

06/25/2021, 7:49 PM
Is the S3 dependency a hard requirement for the Kubernetes deployment at this point or is it only needed for some kind of supplemental logging?
u

user

06/25/2021, 8:01 PM
@Joe I can test this here in a little bit and see if it complains at me for having no S3 env details set. I know when I set them incorrectly, it erred and did not start several pods.
u

user

06/25/2021, 8:18 PM
Before 0.26.4 it has the behavior bradfair describes. In 0.26.4 we made it temporarily optional so it doesn’t fail. However, we will switch to requiring configuring cloud storage once we have s3, gcs, and minio support. This requirement will allow us to scale up our logging system and break apart some more internal components for even more horizontal scalability.
u

user

06/25/2021, 8:25 PM
^^ i’d absolutely love if airbyte could accept a pvc as an option for a logging destination, or implement something like argo workflow’s ArtifactRepository model
u

user

06/25/2021, 8:28 PM
We did think about a logging server where containers send their logs over the network to the server and it stores it on a large individual PVC. We can’t have all of the individual containers write directly to a mounted PVC unless it’s an NFS-style mount, which runs into many problems of its own.
u

user

06/25/2021, 8:29 PM
From your use standpoint, is a PVC that we use directly much more useful than if we provide manifests configured to interact with a local minio instance that is using a PVC under the hood?
j

Joe

06/25/2021, 8:32 PM
Nope, the ability to have a minio instance that’s backed by a PVC would cover my use cases
u

user

06/25/2021, 8:35 PM
Ok cool. We’re planning on that being the default option, but allow you to configure a cloud storage of your choice if you want something that scales “infinitely”
2 Views