Hey guys, how are you?
I'm trying to create a red...
# ask-community-for-troubleshootingr
Rogerio Ferreira Dos Santos
09/20/2022, 2:35 PMHey guys, how are you?
I'm trying to create a redshift destination with S3 Staging uploading method, but the set up crash and returns a log with the error 403 access denied to bucket. I have checked the credentials and the permissions to s3, and everything is ok, i don't know what is happen...someone can help me?
s
Saj Dider (Airbyte)
09/20/2022, 3:24 PMcan you post the log?
r
Rogerio Ferreira Dos Santos
09/20/2022, 8:39 PM Copy code
2022-09-20 20:21:03 [32mINFO[m i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):593 - Setting stdout...
2022-09-20 20:21:04 [32mINFO[m i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):605 - Setting stderr...
2022-09-20 20:21:11 [32mINFO[m i.a.w.p.KubePodProcess(<init>):565 - Reading pod IP...
2022-09-20 20:21:11 [32mINFO[m i.a.w.p.KubePodProcess(<init>):567 - Pod IP: 10.193.1.5
2022-09-20 20:21:11 [32mINFO[m i.a.w.p.KubePodProcess(<init>):574 - Using null stdin output stream...
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-09-20 20:21:06 [32mINFO[m i.a.i.d.r.RedshiftDestination(main):63 - starting destination: class io.airbyte.integrations.destination.redshift.RedshiftDestination
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-09-20 20:21:06 [32mINFO[m i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {check=null, config=source_config.json}
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-09-20 20:21:06 [32mINFO[m i.a.i.b.IntegrationRunner(runInternal):104 - Running integration: io.airbyte.integrations.destination.redshift.RedshiftDestination
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-09-20 20:21:06 [32mINFO[m i.a.i.b.IntegrationRunner(runInternal):105 - Command: CHECK
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-09-20 20:21:06 [32mINFO[m i.a.i.b.IntegrationRunner(runInternal):106 - Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-09-20 20:21:07 [33mWARN[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-09-20 20:21:07 [33mWARN[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-09-20 20:21:07 [33mWARN[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-09-20 20:21:07 [32mINFO[m i.a.i.d.j.c.SwitchingDestination(check):55 - Using destination type: COPY_S3
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-09-20 20:21:07 [32mINFO[m i.a.i.d.s.S3DestinationConfig(createS3Client):190 - Creating S3 client...
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-09-20 20:21:09 [32mINFO[m i.a.i.d.s.S3StorageOperations(createBucketObjectIfNotExists):102 - Storage Object datalake-sa-east-1-staging/airbyte/redshift does not exist in bucket; creating...
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-09-20 20:21:09 [32mINFO[m i.a.i.d.s.S3StorageOperations(createBucketObjectIfNotExists):104 - Storage Object datalake-sa-east-1-staging/airbyte/redshift has been created in bucket.
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-09-20 20:21:09 [32mINFO[m i.a.i.d.s.S3Destination(testIAMUserHasListObjectPermission):156 - Started testing if IAM user can call listObjects on the destination bucket
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-09-20 20:21:10 [32mINFO[m i.a.i.d.s.S3Destination(testIAMUserHasListObjectPermission):159 - Finished checking for listObjects permission
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-09-20 20:21:10 [1;31mERROR[m i.a.i.b.AirbyteExceptionHandler(uncaughtException):26 - Something went wrong in the connector. See the logs for more details.
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: ZRTQTW0S24851SVE; S3 Extended Request ID: t2YsecukKjCOmpRmOuU9y9rgvS6HivVYOCd+f/JKlj7Y9UU76HluCTyJWqW86EdtkzKHKhpi6/rOavG/rVMt/g==; Proxy: null)
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819) ~[aws-java-sdk-core-1.12.6.jar:?]
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1403) ~[aws-java-sdk-core-1.12.6.jar:?]
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1372) ~[aws-java-sdk-core-1.12.6.jar:?]
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145) ~[aws-java-sdk-core-1.12.6.jar:?]
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802) ~[aws-java-sdk-core-1.12.6.jar:?]
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770) ~[aws-java-sdk-core-1.12.6.jar:?]
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744) ~[aws-java-sdk-core-1.12.6.jar:?]
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704) ~[aws-java-sdk-core-1.12.6.jar:?]
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686) ~[aws-java-sdk-core-1.12.6.jar:?]
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550) ~[aws-java-sdk-core-1.12.6.jar:?]
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530) ~[aws-java-sdk-core-1.12.6.jar:?]
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5437) ~[aws-java-sdk-s3-1.12.6.jar:?]
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5384) ~[aws-java-sdk-s3-1.12.6.jar:?]
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - at com.amazonaws.services.s3.AmazonS3Client.deleteObject(AmazonS3Client.java:2294) ~[aws-java-sdk-s3-1.12.6.jar:?]
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - at com.amazonaws.services.s3.AmazonS3Client.deleteObject(AmazonS3Client.java:2278) ~[aws-java-sdk-s3-1.12.6.jar:?]
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - at io.airbyte.integrations.destination.s3.S3Destination.attemptWriteAndDeleteS3Object(S3Destination.java:152) ~[io.airbyte.airbyte-integrations.connectors-destination-s3-0.40.3.jar:?]
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - at io.airbyte.integrations.destination.s3.S3Destination.attemptS3WriteAndDelete(S3Destination.java:139) ~[io.airbyte.airbyte-integrations.connectors-destination-s3-0.40.3.jar:?]
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - at io.airbyte.integrations.destination.s3.S3Destination.attemptS3WriteAndDelete(S3Destination.java:129) ~[io.airbyte.airbyte-integrations.connectors-destination-s3-0.40.3.jar:?]
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - at io.airbyte.integrations.destination.redshift.RedshiftStagingS3Destination.check(RedshiftStagingS3Destination.java:67) ~[io.airbyte.airbyte-integrations.connectors-destination-redshift-0.40.3.jar:?]
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - at io.airbyte.integrations.destination.jdbc.copy.SwitchingDestination.check(SwitchingDestination.java:56) ~[io.airbyte.airbyte-integrations.connectors-destination-jdbc-0.40.3.jar:?]
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - at io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:121) ~[io.airbyte.airbyte-integrations.bases-base-java-0.40.3.jar:?]
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:97) ~[io.airbyte.airbyte-integrations.bases-base-java-0.40.3.jar:?]
2022-09-20 20:21:11 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - at io.airbyte.integrations.destination.redshift.RedshiftDestination.main(RedshiftDestination.java:64) ~[io.airbyte.airbyte-integrations.connectors-destination-redshift-0.40.3.jar:?]
2022-09-20 20:21:12 [32mINFO[m i.a.w.p.KubePodProcess(close):714 - (pod: airbyte / ion-redshift-check-1689dd73-ca66-4760-95fd-67e93943e088-0-wfajt) - Closed all resources for pod
2022-09-20 20:21:12 [1;31mERROR[m i.a.w.g.DefaultCheckConnectionWorker(run):86 - Error checking connection, status: Optional.empty, exit code: 1
2022-09-20 20:21:12 [32mINFO[m i.a.w.t.TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling...Rogerio Ferreira Dos Santos
09/20/2022, 8:39 PM@Saj Dider (Airbyte) is here 🙂
Rogerio Ferreira Dos Santos
09/21/2022, 6:29 PM@Saj Dider (Airbyte) the credentials that i'm using have access to s3, but don't have access to redshift. I need to set permissions to redshift either?
s
Saj Dider (Airbyte)
09/21/2022, 6:38 PMThanks @Rogerio Ferreira Dos Santos, it's almost certainly a permissions issue. Have you doubled checked iam permissions? Maybe this will help: https://docs.aws.amazon.com/redshift/latest/mgmt/authorizing-redshift-service.html
Saj Dider (Airbyte)
09/21/2022, 6:39 PMAlso check if you specified a bucket_path for the s3 staging. It's written as optional but we have an open issue about it here: https://github.com/airbytehq/airbyte/issues/16325. Might be related but probably not.
Saj Dider (Airbyte)
10/05/2022, 3:14 PM@Rogerio Ferreira Dos Santos were you able to solve the problem here? There's another user asking about the same issue. Wondering if we have to escalate this