Hi All We have problem deploying airbyte on EKS cluster usin Airbyte #ask-ai

Hi All, We have problem deploying airbyte on EKS c...

Aman Kesharwani

04/27/2023, 8:06 AM

Hi All, We have problem deploying airbyte on EKS cluster using helm chart, We are using external minio and postgress in our setup, after deploying when I am trying to create any source connection in UI I get the following error on checking the worker and server pod logs I could see the following detailed error message, even though I am specifying minio bucket to use why is it connecting to S3, Any help will be really appreciated, let me know if I need to share values.yaml file used for deployment

Copy code

Collecting content into /tmp/toBePublished6209292906792156831.tmp before uploading.
Cannot start publish with com.van.logging.aws.S3PublishHelper@703961dc due to error: Cannot start publishing: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: null; S3 Extended Request ID: null; Proxy: null)
Publishing to S3 (bucket=sample-bucket; key=job-logging/workspace/66cd15e7-0f0c-4a7c-89be-79863ba48b6b/0/logs.log/20230427075426_int-qa-airbyte-worker-59ff685c8f-s76xt_dcce21db15244bf8a191652ceb5710f5):
java.lang.RuntimeException: Cannot publish to S3: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: null; S3 Extended Request ID: null; Proxy: null)

Aman Kesharwani

04/28/2023, 6:17 AM

Can someone please help me on this ?

Shashank Singh

04/30/2023, 1:10 AM

@kapa.ai

kapa.ai

04/30/2023, 1:10 AM

A message from kapa.ai

kapa.ai

04/30/2023, 1:10 AM

Sorry @Shashank Singh, I seem to be having a bit of trouble. Don't worry though, I've pinged my support folks and they will try to fix me up! 🧑‍💻 Perhaps try again in 5mins?

Aman Kesharwani

05/02/2023, 7:44 AM

Any support will be appreciated ?

Sunny Hashmi

05/02/2023, 6:02 PM

Hey @Aman Kesharwani minio uses the S3 api and S3 libraries so that might be why the errors are printed this way, especially for external minio. can you share the values.yaml?

Aman Kesharwani

05/02/2023, 6:31 PM

values.yaml_shared

Aman Kesharwani

05/02/2023, 6:32 PM

@Sunny Hashmi

Sunny Hashmi

05/03/2023, 4:39 PM

Hey @Aman Kesharwani could you try switching logs.s3.enabled to false and see if it helps?

Copy code

##  logs.s3.enabled Switch to enable or disable custom S3 Log location
    ##  logs.s3.bucket Bucket name where logs should be stored
    ##  logs.s3.bucketRegion Region of the bucket (must be empty if using minio)
    s3:
      enabled: true
      bucket: sample-bucket
      bucketRegion: ""

Aman Kesharwani

05/03/2023, 6:49 PM

Hi @Sunny Hashmi, I tried changing the value to false, but still the same issue

Aman Kesharwani

05/03/2023, 7:26 PM

@Sunny Hashmi when I try switching logs.s3.enabled to false I get the same as pasted above, but when I leave the bucket value to empty instead of sample-bucket I see the below error msg in worker and server pod logs

Copy code

2023-05-03 19:22:36 INFO i.a.d.c.DatabaseMigrationCheck(check):62 - Verified that database has been migrated to the required minimum version 0.29.15.001.
2023-05-03 19:22:36 ERROR i.m.r.Micronaut(handleStartupException):338 - Error starting Micronaut server: null
java.lang.IllegalArgumentException: null
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:131) ~[guava-31.1-jre.jar:?]
	at io.airbyte.config.storage.DefaultS3ClientFactory.validateBase(DefaultS3ClientFactory.java:38) ~[io.airbyte.airbyte-config-config-models-0.44.0.jar:?]
	at io.airbyte.config.storage.MinioS3ClientFactory.validate(MinioS3ClientFactory.java:33) ~[io.airbyte.airbyte-config-config-models-0.44.0.jar:?]
	at io.airbyte.config.storage.MinioS3ClientFactory.<init>(MinioS3ClientFactory.java:27) ~[io.airbyte.airbyte-config-config-models-0.44.0.jar:?]
	at io.airbyte.config.helpers.CloudLogs.createCloudLogClient(CloudLogs.java:54) ~[io.airbyte.airbyte-config-config-models-0.44.0.jar:?]
	at io.airbyte.config.helpers.LogClientSingleton.createCloudClientIfNull(LogClientSingleton.java:226) ~[io.airbyte.airbyte-config-config-models-0.44.0.jar:?]
	at io.airbyte.config.helpers.LogClientSingleton.setWorkspaceMdc(LogClientSingleton.java:213) ~[io.airbyte.airbyte-config-config-models-0.44.0.jar:?]
	at io.airbyte.server.LoggingEventListener.onApplicationEvent(LoggingEventListener.java:34) ~[io.airbyte-airbyte-server-0.44.0.jar:?]

Aman Kesharwani

05/04/2023, 2:45 PM

@Sunny Hashmi Any suggestions ?

Aman Kesharwani

05/04/2023, 5:31 PM

@Sunny Hashmi As discussed over office-hour today sharing the issue I am facing when I try to use S3 bucket instead of external minio in value.yaml configuration

Copy code

2023-05-04 17:27:39 ERROR i.m.r.Micronaut(handleStartupException):338 - Error starting Micronaut server: null
java.lang.IllegalArgumentException: null
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:131) ~[guava-31.1-jre.jar:?]
	at io.airbyte.config.storage.DefaultS3ClientFactory.validateBase(DefaultS3ClientFactory.java:36) ~[io.airbyte.airbyte-config-config-models-0.44.0.jar:?]
	at io.airbyte.config.storage.DefaultS3ClientFactory.validate(DefaultS3ClientFactory.java:31) ~[io.airbyte.airbyte-config-config-models-0.44.0.jar:?]
	at io.airbyte.config.storage.DefaultS3ClientFactory.<init>(DefaultS3ClientFactory.java:24) ~[io.airbyte.airbyte-config-config-models-0.44.0.jar:?]
	at io.airbyte.config.helpers.CloudLogs.createCloudLogClient(CloudLogs.java:51) ~[io.airbyte.airbyte-config-config-models-0.44.0.jar:?]
	at io.airbyte.config.helpers.LogClientSingleton.createCloudClientIfNull(LogClientSingleton.java:226) ~[io.airbyte.airbyte-config-config-models-0.44.0.jar:?]
	at io.airbyte.config.helpers.LogClientSingleton.setWorkspaceMdc(LogClientSingleton.java:213) ~[io.airbyte.airbyte-config-config-models-0.44.0.jar:?]
	at io.airbyte.workers.ApplicationInitializer.initializeCommonDependencies(ApplicationInitializer.java:189) ~[io.airbyte-airbyte-workers-0.44.0.jar:?]
	at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:156) ~[io.airbyte-airbyte-workers-0.44.0.jar:?]
	at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:62) ~[io.airbyte-airbyte-workers-0.44.0.jar:?]
	at io.micronaut.context.event.ApplicationEventPublisherFactory.notifyEventListeners(ApplicationEventPublisherFactory.java:262) ~[micronaut-inject-3.8.8.jar:3.8.8]
	at io.micronaut.context.event.ApplicationEventPublisherFactory.access$200(ApplicationEventPublisherFactory.java:60) ~[micronaut-inject-3.8.8.jar:3.8.8]
	at io.micronaut.context.event.ApplicationEventPublisherFactory$2.publishEvent(ApplicationEventPublisherFactory.java:229) ~[micronaut-inject-3.8.8.jar:3.8.8]
	at io.micronaut.http.server.netty.NettyHttpServer.lambda$fireStartupEvents$15(NettyHttpServer.java:587) ~[micronaut-http-server-netty-3.8.8.jar:3.8.8]
	at java.util.Optional.ifPresent(Optional.java:178) ~[?:?]
	at io.micronaut.http.server.netty.NettyHttpServer.fireStartupEvents(NettyHttpServer.java:581) ~[micronaut-http-server-netty-3.8.8.jar:3.8.8]
	at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:298) ~[micronaut-http-server-netty-3.8.8.jar:3.8.8]
	at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:104) ~[micronaut-http-server-netty-3.8.8.jar:3.8.8]
	at io.micronaut.runtime.Micronaut.lambda$start$2(Micronaut.java:81) ~[micronaut-context-3.8.8.jar:3.8.8]
	at java.util.Optional.ifPresent(Optional.java:178) ~[?:?]
	at io.micronaut.runtime.Micronaut.start(Micronaut.java:79) ~[micronaut-context-3.8.8.jar:3.8.8]
	at io.micronaut.runtime.Micronaut.run(Micronaut.java:323) ~[micronaut-context-3.8.8.jar:3.8.8]
	at io.micronaut.runtime.Micronaut.run(Micronaut.java:309) ~[micronaut-context-3.8.8.jar:3.8.8]
	at io.airbyte.workers.Application.main(Application.java:15) ~[io.airbyte-airbyte-workers-0.44.0.jar:?]

Sunny Hashmi

05/04/2023, 8:24 PM

Hey @Aman Kesharwani are there any other errors in the log? can you share the full file or a snippet with all errors?

Aman Kesharwani

05/05/2023, 5:11 AM

Hi @Sunny Hashmi No this is the only error I get in both worker and server pods when I use S3 instad of Minio and pod is not coming up

Sunny Hashmi

05/05/2023, 5:18 PM

Hey @Aman Kesharwani, I took a closer look at your values.yaml above and it looks like you have both s3 and minio enabled:

Copy code

externalMinio:
      enabled: true
      host: <http://ml-qa-minio-s3.in|ml-qa-minio-s3.in>
      port: 443
    ##  logs.s3.enabled Switch to enable or disable custom S3 Log location
    ##  logs.s3.bucket Bucket name where logs should be stored
    ##  logs.s3.bucketRegion Region of the bucket (must be empty if using minio)
    s3:
      enabled: true
      bucket: sample-bucket
      bucketRegion: ""

The

Bad Request

error is because it's trying to connect to S3 with no region specified. You need to disable s3 if you want to connect to your external minio.

Sunny Hashmi

05/05/2023, 5:46 PM

You should also add your minio creds in the logs parameters overwrites below:

Copy code

## @section Logs parameters

## @section Minio chart overwrites
##  minio.accessKey.password Minio Access Key
##  minio.secretKey.password Minio Secret Key
minio:
  enabled: false
  auth:
    rootUser: <your-minio-user>
    rootPassword: <your-minio-pw>

You can leave

minio.enabled

false

Aman Kesharwani

05/06/2023, 2:02 PM

Hi @Sunny Hashmi If I try changing s3 to

false

, then worker and server pods are not even starting and in the logs I see below error on worker and server pods,

Copy code

2023-05-06 14:00:46 ERROR i.m.r.Micronaut(handleStartupException):338 - Error starting Micronaut server: null
java.lang.IllegalArgumentException: null
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:131) ~[guava-31.1-jre.jar:?]
	at io.airbyte.config.storage.DefaultS3ClientFactory.validateBase(DefaultS3ClientFactory.java:38) ~[io.airbyte.airbyte-config-config-models-0.44.0.jar:?]
	at io.airbyte.config.storage.MinioS3ClientFactory.validate(MinioS3ClientFactory.java:33) ~[io.airbyte.airbyte-config-config-models-0.44.0.jar:?]
	at io.airbyte.config.storage.MinioS3ClientFactory.<init>(MinioS3ClientFactory.java:27) ~[io.airbyte.airbyte-config-config-models-0.44.0.jar:?]
	at io.airbyte.config.helpers.CloudLogs.createCloudLogClient(CloudLogs.java:54) ~[io.airbyte.airbyte-config-config-models-0.44.0.jar:?]
	at io.airbyte.config.helpers.LogClientSingleton.createCloudClientIfNull(LogClientSingleton.java:226) ~[io.airbyte.airbyte-config-config-models-0.44.0.jar:?]
	at io.airbyte.config.helpers.LogClientSingleton.setWorkspaceMdc(LogClientSingleton.java:213) ~[io.airbyte.airbyte-config-config-models-0.44.0.jar:?]
	at io.airbyte.workers.ApplicationInitializer.initializeCommonDependencies(ApplicationInitializer.java:189) ~[io.airbyte-airbyte-workers-0.44.0.jar:?]
	at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:156) ~[io.airbyte-airbyte-workers-0.44.0.jar:?]
	at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:62) ~[io.airbyte-airbyte-workers-0.44.0.jar:?]
	at io.micronaut.context.event.ApplicationEventPublisherFactory.notifyEventListeners(ApplicationEventPublisherFactory.java:262) ~[micronaut-inject-3.8.8.jar:3.8.8]
	at io.micronaut.context.event.ApplicationEventPublisherFactory.access$200(ApplicationEventPublisherFactory.java:60) ~[micronaut-inject-3.8.8.jar:3.8.8]
	at io.micronaut.context.event.ApplicationEventPublisherFactory$2.publishEvent(ApplicationEventPublisherFactory.java:229) ~[micronaut-inject-3.8.8.jar:3.8.8]
	at io.micronaut.http.server.netty.NettyHttpServer.lambda$fireStartupEvents$15(NettyHttpServer.java:587) ~[micronaut-http-server-netty-3.8.8.jar:3.8.8]
	at java.util.Optional.ifPresent(Optional.java:178) ~[?:?]
	at io.micronaut.http.server.netty.NettyHttpServer.fireStartupEvents(NettyHttpServer.java:581) ~[micronaut-http-server-netty-3.8.8.jar:3.8.8]
	at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:298) ~[micronaut-http-server-netty-3.8.8.jar:3.8.8]
	at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:104) ~[micronaut-http-server-netty-3.8.8.jar:3.8.8]
	at io.micronaut.runtime.Micronaut.lambda$start$2(Micronaut.java:81) ~[micronaut-context-3.8.8.jar:3.8.8]
	at java.util.Optional.ifPresent(Optional.java:178) ~[?:?]
	at io.micronaut.runtime.Micronaut.start(Micronaut.java:79) ~[micronaut-context-3.8.8.jar:3.8.8]
	at io.micronaut.runtime.Micronaut.run(Micronaut.java:323) ~[micronaut-context-3.8.8.jar:3.8.8]
	at io.micronaut.runtime.Micronaut.run(Micronaut.java:309) ~[micronaut-context-3.8.8.jar:3.8.8]
	at io.airbyte.workers.Application.main(Application.java:15) ~[io.airbyte-airbyte-workers-0.44.0.jar:?]

Also I have noticed that there is no other placeholder to specify minio bucket name, my understanding was that if I keep the bucketRegion value to null and s3 to

true

the bucket would be considered as minio bucket as mentioned in the values.yaml file as well

##  logs.s3.bucketRegion Region of the bucket (must be empty if using minio)

Aman Kesharwani

05/06/2023, 2:05 PM

On the second point which you mentioned to specify minio credentials, I do not have access to minio root credentials instead I have access to minio access key and secret key for a specific minio bucket which I have specified under the logs section

Sunny Hashmi

05/08/2023, 1:28 PM

You currently have both minio and s3 enabled, you need to pick one. I agree it's likely possible to use the s3 parameters with minio, i'm not sure if this would impact anything (maybe metrics? not sure) Here is documentation for why I recommended adding your credentials under

minio.auth.rootUser

and `minio.auth.rootPassword`: https://github.com/bitnami/charts/tree/main/bitnami/minio#to-900

Aman Kesharwani

05/08/2023, 1:32 PM

Okay @Sunny Hashmi let me get minio root user and password and disable S3 Could you please let me know where do I need to specify Minio bucket name in the values.yaml file ?

Sunny Hashmi

05/08/2023, 1:32 PM

For the record i dont think it haaas to be a root password.. i don't know why it is labelled that way

Aman Kesharwani

05/08/2023, 1:34 PM

understood, if you could clarify on Minio bucket where it needs to be added it would be great

Aman Kesharwani

05/08/2023, 1:34 PM

As I do not see any section to add minio bucket name

Aman Kesharwani

05/08/2023, 1:34 PM

@Sunny Hashmi

Sunny Hashmi

05/08/2023, 1:41 PM

Good question. Checking

🙏 1

Sunny Hashmi

05/08/2023, 7:42 PM

Hey @Aman Kesharwani, i was able to get external minio working by adding it in the s3 config but leaving it

enabled: false

. should look something like this:

Copy code

externalMinio:
      enabled: true
      host: <http://ml-qa-minio-s3.in|ml-qa-minio-s3.in>
      port: 443
    ##  logs.s3.enabled Switch to enable or disable custom S3 Log location
    ##  logs.s3.bucket Bucket name where logs should be stored
    ##  logs.s3.bucketRegion Region of the bucket (must be empty if using minio)
    s3:
      enabled: false
      bucket: "bucket-name"
      bucketRegion: ""

Aman Kesharwani

05/09/2023, 4:06 AM

@Sunny Hashmi Thanks, Let me test and get back incase of any issue

👍🏽 1

Aman Kesharwani

05/16/2023, 5:35 AM

Hi @Sunny Hashmi, I was on leave could not test and respond back, I have updated the values.yaml file as per your suggestion but I am still getting the same issue, sharing the updated

values.yaml

file with you

values.yaml_shared

Aman Kesharwani

05/18/2023, 10:31 AM

@Sunny Hashmi Any help will be really appreciated, if using external minio is an issue I am okay with switching to S3 for storage and logging but in that case I am getting different issue the worker and server pods are not even coming up ``ERROR i.m.r.Micronaut(handleStartupException):338 - Error starting Micronaut server: null java.lang.IllegalArgumentException: null at com.google.common.base.Preconditions.checkArgument(Preconditions.java:131) ~[guava-31.1-jre.jar:?] at io.airbyte.config.storage.DefaultS3ClientFactory.validateBase(DefaultS3ClientFactory.java:36) ~[io.airbyte.airbyte-config-config-models-0.44.0.jar:?] at io.airbyte.config.storage.DefaultS3ClientFactory.validate(DefaultS3ClientFactory.java:31) ~[io.airbyte.airbyte-config-config-models-0.44.0.jar:?] at io.airbyte.config.storage.DefaultS3ClientFactory.<init>(DefaultS3ClientFactory.java:24) ~[io.airbyte.airbyte-config-config-models-0.44.0.jar:?] at io.airbyte.config.helpers.CloudLogs.createCloudLogClient(CloudLogs.java:51) ~[io.airbyte.airbyte-config-config-models-0.44.0.jar:?] at io.airbyte.config.helpers.LogClientSingleton.createCloudClientIfNull(LogClientSingleton.java:226) ~[io.airbyte.airbyte-config-config-models-0.44.0.jar:?] at io.airbyte.config.helpers.LogClientSingleton.setWorkspaceMdc(LogClientSingleton.java:213) ~[io.airbyte.airbyte-config-config-models-0.44.0.jar:?] at io.airbyte.server.LoggingEventListener.onApplicationEvent(LoggingEventListener.java:34) ~[io.airbyte-airbyte-server-0.44.0.jar:?] at io.airbyte.server.LoggingEventListener.onApplicationEvent(LoggingEventListener.java:21) ~[io.airbyte-airbyte-server-0.44.0.jar:?] at io.micronaut.context.event.ApplicationEventPublisherFactory.notifyEventListeners(ApplicationEventPublisherFactory.java:262) ~[micronaut-inject-3.8.8.jar:3.8.8] at io.micronaut.context.event.ApplicationEventPublisherFactory.access$200(ApplicationEventPublisherFactory.java:60) ~[micronaut-inject-3.8.8.jar:3.8.8] at io.micronaut.context.event.ApplicationEventPublisherFactory$2.publishEvent(ApplicationEventPublisherFactory.java:229) ~[micronaut-inject-3.8.8.jar:3.8.8] at io.micronaut.http.server.netty.NettyHttpServer.lambda$fireStartupEvents$15(NettyHttpServer.java:587) ~[micronaut-http-server-netty-3.8.8.jar:3.8.8] at java.util.Optional.ifPresent(Optional.java:178) ~[?:?] at io.micronaut.http.server.netty.NettyHttpServer.fireStartupEvents(NettyHttpServer.java:581) ~[micronaut-http-server-netty-3.8.8.jar:3.8.8] at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:298) ~[micronaut-http-server-netty-3.8.8.jar:3.8.8] at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:104) ~[micronaut-http-server-netty-3.8.8.jar:3.8.8] at io.micronaut.runtime.Micronaut.lambda$start$2(Micronaut.java:81) ~[micronaut-context-3.8.8.jar:3.8.8] at java.util.Optional.ifPresent(Optional.java:178) ~[?:?] at io.micronaut.runtime.Micronaut.start(Micronaut.java:79) ~[micronaut-context-3.8.8.jar:3.8.8] at io.micronaut.runtime.Micronaut.run(Micronaut.java:323) ~[micronaut-context-3.8.8.jar:3.8.8] at io.micronaut.runtime.Micronaut.run(Micronaut.java:309) ~[micronaut-context-3.8.8.jar:3.8.8] at io.airbyte.server.Application.main(Application.java:15) ~[io.airbyte-airbyte-server-0.44.0.jar:?]``

Sunny Hashmi

05/19/2023, 4:27 PM

hey @Aman Kesharwani, please update

logs.s3.bucket

to include your bucket name and try again. If it is still not working, attach the full log and we'll take a look

Copy code

s3:
      enabled: false
      bucket: "bucket-name"
      bucketRegion: ""

Aman Kesharwani

05/22/2023, 11:31 AM

@Sunny Hashmi Attaching the server and worker pods full logs for your reference when using external minio, Please let me know if any further logs are required for debugging

server_logs.txt worker_logs.txt

Aman Kesharwani

05/24/2023, 6:15 AM

@Sunny Hashmi Kindly check and let me know if any further details are required

Aman Kesharwani

05/29/2023, 12:26 PM

@Sunny Hashmi Did you had opportunity to check and test it from your end

Open in Slack

Previous Next