we are trying to get Airbyte up and running on EKS...
# replication-troubleshooting
j
we are trying to get Airbyte up and running on EKS with sources of stripe and an in house Postgres's database. Destination is snowflake. We are able to get the connections established for both source and destination and verified the credentials correct. Setting up the connection is able to pull the schemas. for the sources. The issues is when running the sync it keeps failing. We have been thorough the logs but unclear on where the issue could be or how we can get more detailed logs. The message we seem to get is
Copy code
Failure Origin: airbyte_platform, Message: Something went wrong within the airbyte platform
1:46PM 12/07
2022-12-07 20:15:36 - Additional Failure Information: scheduledEventId=72, startedEventId=73, activityType='RunWithJobOutput', activityId='d9c45409-2372-30d6-ae8f-29bdb4949aa2', identity='', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED
I have downloaded the full logs here is the last part of the log... note, this was trying to write to S3 instead of snowflake as a test.
Copy code
22-12-07 20:05:32 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - integration args: {check=null, config=source_config.json}
2022-12-07 20:05:32 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Running integration: io.airbyte.integrations.destination.s3.S3Destination
2022-12-07 20:05:32 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Command: CHECK
2022-12-07 20:05:32 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
2022-12-07 20:05:33 [33mWARN[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):117 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-12-07 20:05:33 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - S3 format config: {"format_type":"Avro","compression_codec":{"codec":"no compression"}}
2022-12-07 20:05:33 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Creating S3 client...
2022-12-07 20:05:37 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Started testing if IAM user can call listObjects on the destination bucket
2022-12-07 20:05:39 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Finished checking for listObjects permission
2022-12-07 20:05:39 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Started testing if all required credentials assigned to user for single file uploading
2022-12-07 20:05:40 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Finished checking for normal upload mode
2022-12-07 20:05:40 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Started testing if all required credentials assigned to user for multipart upload
2022-12-07 20:05:40 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Initiated multipart upload to mudflap.data.snowflake.loadstage//airbyte/test_1670443540085 with full ID MMGkcQrUGmDTk7znWge4J9yrJ9nXqRkFZu8QZ4G.ZNB3LipHsG64nk.nms6wXPMaFjf5RUQon4Wg2E5PhRmQtqphGE2j3Z62l4zegSJj0Xazg1jzokYue88h1NrWeHubM.vEC8TW8veSIip6r72r_3QX2oz7FKRostkQ1TCEOns-
2022-12-07 20:05:40 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Called close() on [MultipartOutputStream for parts 1 - 10000]
2022-12-07 20:05:40 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Called close() on [MultipartOutputStream for parts 1 - 10000]
2022-12-07 20:05:40 [33mWARN[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):117 - [MultipartOutputStream for parts 1 - 10000] is already closed
2022-12-07 20:05:40 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - [Manager uploading to mudflap.data.snowflake.loadstage//airbyte/test_1670443540085 with id MMGkcQrUG...Q1TCEOns-]: Uploading leftover stream [Part number 1 containing 3.34 MB]
2022-12-07 20:05:41 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - [Manager uploading to mudflap.data.snowflake.loadstage//airbyte/test_1670443540085 with id MMGkcQrUG...Q1TCEOns-]: Finished uploading [Part number 1 containing 3.34 MB]
2022-12-07 20:05:41 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - [Manager uploading to mudflap.data.snowflake.loadstage//airbyte/test_1670443540085 with id MMGkcQrUG...Q1TCEOns-]: Completed
2022-12-07 20:05:41 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Finished verification for multipart upload mode
2022-12-07 20:05:41 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Completed integration: io.airbyte.integrations.destination.s3.S3Destination
2022-12-07 20:05:41 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Completed destination: io.airbyte.integrations.destination.s3.S3Destination
2022-12-07 20:05:44 [32mINFO[m i.a.w.p.KubePodProcess(close):737 - (pod: data-eng / destination-s3-check-11-0-nixlg) - Closed all resources for pod
2022-12-07 20:05:44 [32mINFO[m i.a.w.t.TemporalAttemptExecution(get):162 - Stopping cancellation check scheduling...
2022-12-07 20:05:44 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 - 
2022-12-07 20:05:44 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 - ----- END CHECK -----
2022-12-07 20:05:44 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 - 
2022-12-07 20:05:44 [33mWARN[m i.t.i.w.ActivityWorker$TaskHandlerImpl(logExceptionDuringResultReporting):365 - Failure during reporting of activity result to the server. ActivityId = d9c45409-2372-30d6-ae8f-29bdb4949aa2, ActivityType = RunWithJobOutput, WorkflowId=connection_manager_145da5f5-a373-4b9d-8957-0383c8946517, WorkflowType=ConnectionManagerWorkflow, RunId=68b13988-1ec6-48cf-8475-3aa10c94288e
io.grpc.StatusRuntimeException: NOT_FOUND: invalid activityID or activity already timed out or invoking workflow is completed
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) ~[grpc-stub-1.50.2.jar:1.50.2]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) ~[grpc-stub-1.50.2.jar:1.50.2]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) ~[grpc-stub-1.50.2.jar:1.50.2]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.respondActivityTaskCompleted(WorkflowServiceGrpc.java:3840) ~[temporal-serviceclient-1.17.0.jar:?]
s
Hey @Jaye Howell, were you running into this error with both postgres and stripe sources? can you share the full log? it may be resource related, can you try increasing Airbyte resources and reset the sync to see if that will help? https://docs.airbyte.io/operator-guides/scaling-airbyte#memory