Does anyone try to connect Redshift as source? I g...
# replication-troubleshooting
u
Does anyone try to connect Redshift as source? I got below error: Redshift connector version - 0.3.51 Helm chart version- 0.43.2 server & worker latest one
Copy code
2022-12-14 09:48:51 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - integration args: {check=null, config=source_config.json}
Log4j2Appender says: Running integration: io.airbyte.integrations.destination.redshift.RedshiftDestination
2022-12-14 09:48:51 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Running integration: io.airbyte.integrations.destination.redshift.RedshiftDestination
Log4j2Appender says: Command: CHECK
2022-12-14 09:48:51 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Command: CHECK
Log4j2Appender says: Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
2022-12-14 09:48:51 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
Log4j2Appender says: Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-12-14 09:48:51 [33mWARN[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):117 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
Log4j2Appender says: Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-12-14 09:48:51 [33mWARN[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):117 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
Log4j2Appender says: The "standard" upload mode is not performant, and is not recommended for production. Please use the Amazon S3 upload mode if you are syncing a large amount of data.
2022-12-14 09:48:51 [33mWARN[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):117 - The "standard" upload mode is not performant, and is not recommended for production. Please use the Amazon S3 upload mode if you are syncing a large amount of data.
Log4j2Appender says: Using destination type: STANDARD
2022-12-14 09:48:51 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Using destination type: STANDARD
Log4j2Appender says: HikariPool-1 - Starting...
2022-12-14 09:48:51 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - HikariPool-1 - Starting...
Log4j2Appender says: HikariPool-1 - Start completed.
2022-12-14 09:48:51 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - HikariPool-1 - Start completed.
Collecting content into /tmp/toBePublished15349657165706088974.tmp before uploading.
Publishing to S3 (bucket=hk01prod-data-airbyte; key=job-logging/workspace/536cef4f-f26d-44af-b866-f57c69737c09/0/logs.log/20221214094946_project-airbyte-main-worker-f5b986846-4zgqm_a256005acc5d4e89908d10c88acaa665):
Log4j2Appender says: HikariPool-1 - Shutdown initiated...
2022-12-14 09:49:51 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - HikariPool-1 - Shutdown initiated...
Log4j2Appender says: HikariPool-1 - Shutdown completed.
2022-12-14 09:49:53 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - HikariPool-1 - Shutdown completed.
Log4j2Appender says: Completed integration: io.airbyte.integrations.destination.redshift.RedshiftDestination
2022-12-14 09:49:53 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Completed integration: io.airbyte.integrations.destination.redshift.RedshiftDestination
Log4j2Appender says: completed destination: class io.airbyte.integrations.destination.redshift.RedshiftDestination
2022-12-14 09:49:53 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - completed destination: class io.airbyte.integrations.destination.redshift.RedshiftDestination
2022-12-14 09:49:54 [32mINFO[m i.a.w.p.ExitCodeWatcher(persistExitCode):113 - Received exit code 0 for pod ion-redshift-check-536cef4f-f26d-44af-b866-f57c69737c09-0-jljhp
Log4j2Appender says: (pod: project-airbyte / ion-redshift-check-536cef4f-f26d-44af-b866-f57c69737c09-0-jljhp) - Closed all resources for pod
2022-12-14 09:49:54 [32mINFO[m i.a.w.p.KubePodProcess(close):745 - (pod: project-airbyte / ion-redshift-check-536cef4f-f26d-44af-b866-f57c69737c09-0-jljhp) - Closed all resources for pod
Log4j2Appender says: Stopping cancellation check scheduling...
2022-12-14 09:49:54 [32mINFO[m i.a.w.t.TemporalAttemptExecution(get):163 - Stopping cancellation check scheduling...
Log4j2Appender says:
2022-12-14 09:49:54 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 -
Log4j2Appender says: ----- END CHECK -----
2022-12-14 09:49:54 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 - ----- END CHECK -----
Log4j2Appender says:
2022-12-14 09:49:54 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 -
m
@郭子麒 (數據管理部)Jacky Kwok Normally this is connected to networking issues between Airbyte instance and Redshift cluster. Assuming you use AWS: You need to whitelist the IP of your NAT gateway (if private subnets were added) of the VPC you deployed Airbyte in to your Redshift cluster. You also could whitelist the IP of your nodes but they could change. https://aws.amazon.com/premiumsupport/knowledge-center/cannot-connect-redshift-cluster/
Let me know if this is helpful, otherwise happy to jump on a call to help out
u
Forgot to mention, we are hosting Airbtye on EKS , which is already allowed to access Redshift Cluster. I can telnet the redshift but failed to connect it in airbyte. I have eabled the debug log and log as below:
Copy code
2022-12-16 03:41:38 [32mINFO[m i.a.w.p.KubePodProcess(close):745 - (pod: project-airbyte / ion-postgres-check-0860c427-b4b0-4b73-a0d5-9dd3aefaf563-0-iaskh) - Closed all resources for pod
2022-12-16 03:41:38 [36mDEBUG[m i.f.k.c.d.i.AbstractWatchManager(scheduleReconnect):140 - Ignoring error for already closed/closing connection
Log4j2Appender says: Check connection job subprocess finished with exit code 0
2022-12-16 03:41:38 [36mDEBUG[m i.a.w.g.DefaultCheckConnectionWorker(run):92 - Check connection job subprocess finished with exit code 0
Log4j2Appender says: Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@1c29fcfe[status=failed,message=State code: 42704; Message: ERROR: type "jsonb" does not exist]
2022-12-16 03:41:38 [36mDEBUG[m i.a.w.g.DefaultCheckConnectionWorker(run):93 - Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@1c29fcfe[status=failed,message=State code: 42704; Message: ERROR: type "jsonb" does not exist]
not sure what it means : Message: ERROR: type "jsonb" does not exist
resolved due to network issue