Airbyte server is failing when S3 usage is enabled...
# ask-ai
h
Airbyte server is failing when S3 usage is enabled. I annotated the service account with the IAM role. IAM role has policy that can get/write object. IAM role has trust relationship with the relevant oidc. What am I missing here?
Copy code
values.yaml
============
global.state.storage.type: "S3"
global.logs.storage.type: "S3"
global.logs.minio.enabled: false
global.logs.s3.enabled: true
global.logs.s3.bucket: "XXXXXX"
global.logs.s3.bucketRegion: "XXXXXX"

error
=======
2023-05-22 13:49:49 ERROR i.m.r.Micronaut(handleStartupException):338 - Error starting Micronaut server: null
java.lang.IllegalArgumentException: null
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:131) ~[guava-31.1-jre.jar:?]
	at io.airbyte.config.storage.DefaultS3ClientFactory.validateBase(DefaultS3ClientFactory.java:36) ~[io.airbyte.airbyte-config-config-models-0.44.4.jar:?]
	at io.airbyte.config.storage.DefaultS3ClientFactory.validate(DefaultS3ClientFactory.java:31) ~[io.airbyte.airbyte-config-config-models-0.44.4.jar:?]
	at io.airbyte.config.storage.DefaultS3ClientFactory.<init>(DefaultS3ClientFactory.java:24) ~[io.airbyte.airbyte-config-config-models-0.44.4.jar:?]
	at io.airbyte.config.helpers.CloudLogs.createCloudLogClient(CloudLogs.java:51) ~[io.airbyte.airbyte-config-config-models-0.44.4.jar:?]
	at io.airbyte.config.helpers.LogClientSingleton.createCloudClientIfNull(LogClientSingleton.java:226) ~[io.airbyte.airbyte-config-config-models-0.44.4.jar:?]
	at io.airbyte.config.helpers.LogClientSingleton.setWorkspaceMdc(LogClientSingleton.java:213) ~[io.airbyte.airbyte-config-config-models-0.44.4.jar:?]
	at io.airbyte.server.LoggingEventListener.onApplicationEvent(LoggingEventListener.java:34) ~[io.airbyte-airbyte-server-0.44.4.jar:?]
	at io.airbyte.server.LoggingEventListener.onApplicationEvent(LoggingEventListener.java:21) ~[io.airbyte-airbyte-server-0.44.4.jar:?]
	at io.micronaut.context.event.ApplicationEventPublisherFactory.notifyEventListeners(ApplicationEventPublisherFactory.java:262) ~[micronaut-inject-3.9.0.jar:3.9.0]
	at io.micronaut.context.event.ApplicationEventPublisherFactory.access$200(ApplicationEventPublisherFactory.java:60) ~[micronaut-inject-3.9.0.jar:3.9.0]
	at io.micronaut.context.event.ApplicationEventPublisherFactory$2.publishEvent(ApplicationEventPublisherFactory.java:229) ~[micronaut-inject-3.9.0.jar:3.9.0]
	at io.micronaut.http.server.netty.NettyHttpServer.lambda$fireStartupEvents$15(NettyHttpServer.java:587) ~[micronaut-http-server-netty-3.9.0.jar:3.9.0]
	at java.util.Optional.ifPresent(Optional.java:178) ~[?:?]
	at io.micronaut.http.server.netty.NettyHttpServer.fireStartupEvents(NettyHttpServer.java:581) ~[micronaut-http-server-netty-3.9.0.jar:3.9.0]
	at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:298) ~[micronaut-http-server-netty-3.9.0.jar:3.9.0]
	at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:104) ~[micronaut-http-server-netty-3.9.0.jar:3.9.0]
	at io.micronaut.runtime.Micronaut.lambda$start$2(Micronaut.java:81) ~[micronaut-context-3.9.0.jar:3.9.0]
	at java.util.Optional.ifPresent(Optional.java:178) ~[?:?]
	at io.micronaut.runtime.Micronaut.start(Micronaut.java:79) ~[micronaut-context-3.9.0.jar:3.9.0]
	at io.micronaut.runtime.Micronaut.run(Micronaut.java:323) ~[micronaut-context-3.9.0.jar:3.9.0]
	at io.micronaut.runtime.Micronaut.run(Micronaut.java:309) ~[micronaut-context-3.9.0.jar:3.9.0]
	at io.airbyte.server.Application.main(Application.java:15) ~[io.airbyte-airbyte-server-0.44.4.jar:?]
1
k
A message from kapa.ai
Hmm, I don't know enough to give you a confident answer yet. However, you can refer to the [Airbyte documentation](https://docs.airbyte.com) for more information on setting up destinations and troubleshooting common issues.
a
@Haim Beyhan I am also facing the same issue, Please update here if you find any workaround or solution for the issue.
plus1 1
j
same here, but im using docker-compose...micronaut just craps out
i tested on three different images, so i'm thinking this is an issue with
.env
a
@Sunny Hashmi @John Wasserman A lot of us are running into the same issue, any help or workaround will be really appreciated
l
hi I’m also trying to use S3 state storage because the minio solution is very unscalable and leads to 30-40% sync failure due to connection refused / restarts / timeouts but looks like S3 state storage is not working at all (I’m using 0.40.32). is S3 State storage supposed to be supported?
d
Just wanted to add to this, I faced this issue and was only able to resolve it using an IAM user. I was able to solve this issue by storing an IAM User Key Pair as a EKS Secret and then referencing it in the values.yaml for Helm. There is an active issue for Airbyte to add support for IAM Role Logging to S3. IAM Role for Dest/Sources were added but S3 Logging will come later (as of two weeks ago from this message).
🙏 1
e
@Drew Ipson do you know if this includes state storage in S3? I'm assuming there is also no support for IAM roles for that?
d
That is correct. To be honest, I haven't yet figured out the IAM role for source or destination. It is not yet well documented and I haven't had a chance to dive into it. My recommendation is sticking with the IAM user for now.