Hi all, is it possible set up airbyte to send logs...
# replication-troubleshooting
j
Hi all, is it possible set up airbyte to send logs to s3 using service account role in eks, instead of
AWS_ACCESS_KEY_ID
and
AWS_SECRET_ACCESS_KEY
?
n
Hi, Jesus! Unfortunately that's not currently possible.
j
Got it.. Thanks. 🙂
j
@Jesus Rivero I’m actually in the process of doing the very same thing and our infra team wanted to use a service account as well. 😔 Is the S3 config with keys working for you? I’m using kustimize and have been running into issues with
airbyte-worker
pod crashing - I found a PR related to the S3 “state” env var but wondering if anyone else resolved it using another solution.
j
@Jason Carter We tested the configuration with s3 keys a few weeks ago and we had the same error that you mention. We tried this configuration and I kept getting the same error, but it was due to an error in the keys I was using, after that I didn't keep trying. To avoid use keys we are going to mount efs volume with minio.
j
Hmmm, interesting. I’d rather not use keys either but our infra team doesn’t support persistent volumes for Minio so I’m stuck trying to hammer out a solution.
e
@Jason Carter @Jesus Rivero did you ever figure out a way to use the service account? running into the same issue
j
i have not tried again. We set up logs on minio, and set up minio with pv/pvc aiming to efs.
With this config we faced with s3 destination works fine wihtout keys if instance role have policies that allows access to s3, but s3 source does not work. It does not work if you dont set the
AWS KEYS