https://linen.dev logo
Join Slack
Powered by
Hello, I have deployed airflow using official helm...
# replication-ask-ai
s
Hello, I have deployed airflow using official helm chart and now i am trying to setup Google search console as source but i am getting this error:
Copy code
Internal Server Error: The specified bucket does not exist (Service: S3, Status Code: 404, Request ID: 176D792AC02C652A, Extended Request ID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8)
I have S3 disabled and i am using minio.. Can someone tell me which bucket is it trying to look for? This is the log. i see in airbyte-server pod:
Copy code
at io.airbyte.commons.server.handlers.SchedulerHandler.checkSourceConnectionFromSourceCreate(SchedulerHandler.java:225) ~[io.airbyte-airbyte-commons-server-0.50.1.jar:?]
        at io.airbyte.server.apis.SchedulerApiController.lambda$executeSourceCheckConnection$1(SchedulerApiController.java:48) ~[io.airbyte-airbyte-server-0.50.1.jar:?]
        at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.java:26) ~[io.airbyte-airbyte-server-0.50.1.jar:?]
        at io.airbyte.server.apis.SchedulerApiController.executeSourceCheckConnection(SchedulerApiController.java:48) ~[io.airbyte-airbyte-server-0.50.1.jar:?]
        at io.airbyte.server.apis.$SchedulerApiController$Definition$Exec.dispatch(Unknown Source) ~[io.airbyte-airbyte-server-0.50.1.jar:?]
        at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:371) ~[micronaut-inject-3.9.2.jar:3.9.2]
        at io.micronaut.context.DefaultBeanContext$4.invoke(DefaultBeanContext.java:594) ~[micronaut-inject-3.9.2.jar:3.9.2]
        at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:303) ~[micronaut-router-3.9.2.jar:3.9.2]
        at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:111) ~[micronaut-router-3.9.2.jar:3.9.2]
        at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:103) ~[micronaut-http-3.9.2.jar:3.9.2]
        at io.micronaut.http.server.RouteExecutor.lambda$executeRoute$14(RouteExecutor.java:659) ~[micronaut-http-server-3.9.2.jar:3.9.2]
        at reactor.core.publisher.FluxDeferContextual.subscribe(FluxDeferContextual.java:49) ~[reactor-core-3.5.0.jar:3.5.0]
        at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62) ~[reactor-core-3.5.0.jar:3.5.0]
        at reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194) ~[reactor-core-3.5.0.jar:3.5.0]
        at io.micronaut.reactive.reactor.instrument.ReactorInstrumentation.lambda$init$0(ReactorInstrumentation.java:62) ~[micronaut-runtime-3.9.2.jar:3.9.2]
        at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84) ~[reactor-core-3.5.0.jar:3.5.0]
        at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37) ~[reactor-core-3.5.0.jar:3.5.0]
        at io.micronaut.scheduling.instrument.InvocationInstrumenterWrappedCallable.call(InvocationInstrumenterWrappedCallable.java:53) ~[micronaut-context-3.9.2.jar:3.9.2]
        at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1589) ~[?:?]
2023-06-30 15:24:19 ESC[1;31mERRORESC[m i.a.s.a.ApiHelper(execute):40 - Unexpected Exception
software.amazon.awssdk.services.s3.model.NoSuchBucketException: The specified bucket does not exist (Service: S3, Status Code: 404, Request ID: 176D792AC02C652A, Extended Request ID: dd9025bab4ad464b04917
7c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8)
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156) ~[aws-xml-protocol-2.20.65.jar:?]
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108) ~[aws-xml-protocol-2.20.65.jar:?]
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85) ~[aws-xml-protocol-2.20.65.jar:?]
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43) ~[aws-xml-protocol-2.20.65.jar:?]
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:95) ~[aws-core-2.20.65.jar:?]
        at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:270) ~[sdk-core-2.20.65.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) ~[sdk-core-2.20.65.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) ~[sdk-core-2.20.65.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.20.65.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[sdk-core-2.20.65.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[sdk-core-2.20.65.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78) ~[sdk-core-2.20.65.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40) ~[sdk-core-2.20.65.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50) ~[sdk-core-2.20.65.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36) ~[sdk-core-2.20.65.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81) ~[sdk-core-2.20.65.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36) ~[sdk-core-2.20.65.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.20.65.jar:?]
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[sdk-core-2.20.65.jar:?]
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[sdk-core-2.20.65.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[sdk-core-2.20.65.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[sdk-core-2.20.65.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[sdk-core-2.20.65.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48) ~[sdk-core-2.20.65.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31) ~[sdk-core-2.20.65.jar:?]
k
A message from kapa.ai
@Shubham Singh That's a great question, but unfortunately it is too long. Can you ask me again but make it shorter? I'm here to help!
s
Hello, I have deployed airflow using official helm chart and now i am trying to setup Google search console as source but i am getting this error:
Copy code
Internal Server Error: The specified bucket does not exist (Service: S3, Status Code: 404, Request ID: 176D792AC02C652A, Extended Request ID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8)
I have S3 disabled and i am using minio.. Can someone tell me which bucket is it trying to look for?
@kapa.ai Hello, I have deployed airflow using official helm chart and now i am trying to setup Google search console as source but i am getting this error:
Copy code
Internal Server Error: The specified bucket does not exist (Service: S3, Status Code: 404, Request ID: 176D792AC02C652A, Extended Request ID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8)
I have S3 disabled and i am using minio.. Can someone tell me which bucket is it trying to look for?
k
A message from kapa.ai
Based on the error message you provided, it seems like Airbyte is trying to access an S3 bucket that doesn't exist. This could be due to a misconfiguration in your environment variables. Airbyte supports logging to the 
Minio
layer, 
S3
bucket, and 
GCS
bucket. If you're using Minio for logging, you need to set the following environment variables:
Copy code
bash S3_LOG_BUCKET=<your_minio_bucket_to_write_logs_in>   AWS_ACCESS_KEY_ID=<your_minio_access_key>   AWS_SECRET_ACCESS_KEY=<your_minio_secret_key>   S3_MINIO_ENDPOINT=<endpoint_where_minio_is_deployed_at>   S3_LOG_BUCKET_REGION=
You also need to set the 
S3_PATH_STYLE_ACCESS
variable to 
true
and let the 
S3_LOG_BUCKET_REGION
variable remain empty. If you're using the Helm chart for deployment, these variables should be set in your values file. Please double-check these configurations and ensure that the specified Minio bucket exists. If you're still facing issues, please provide more details about your setup, including the version of Airbyte you're using and the contents of your values file (with sensitive information redacted). For more information, you can refer to the [Airbyte documentation on Kubernetes deployment](https://docs.airbyte.com/deploying-airbyte/on-kubernetes).
s
@kapa.ai i could not find S3_PATH_STYLE_ACCESS and S3_LOG_BUCKET_REGION variable in the official helm chart
k
A message from kapa.ai
New message text here
t
@Shubham Singh i dont know if youre still having this issue, but this is actually a bug in the helm chart. even though youre setting s3 as enabled: false, as long as the bucket name is filled with a non-empty string this error will stay happening. i was only able to store logs on my gcs bucket when removing the S3 bucket name from values.yaml
🙏 1
Open in Slack
PreviousNext
Powered by