Hi Team, I'm trying to add Postgres as a source in...
# ask-community-for-troubleshooting
m
Hi Team, I'm trying to add Postgres as a source in Airbyte using SSH tunnel. I've correct username/IP and SSH private key
RSA
which has
Copy code
-----BEGIN RSA PRIVATE KEY-----
<content>
-----END RSA PRIVATE KEY-----
It tries to make connection but eventually fails. Here are the logs. It's urget, response will be highly appriciated!!
Copy code
2022-03-09 05:23:53 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$8):523 - Setting stdout...
2022-03-09 05:23:53 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):534 - Setting stderr...
2022-03-09 05:23:54 INFO i.a.w.p.KubePodProcess(<init>):502 - Reading pod IP...
2022-03-09 05:23:54 INFO i.a.w.p.KubePodProcess(<init>):504 - Pod IP: 10.0.2.114
2022-03-09 05:23:54 INFO i.a.w.p.KubePodProcess(<init>):511 - Using null stdin output stream...
2022-03-09 05:23:54 ERROR i.a.c.i.LineGobbler(voidCall):82 - SLF4J: Class path contains multiple SLF4J bindings.
2022-03-09 05:23:54 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-03-09 05:23:54 INFO i.a.i.s.p.PostgresSource(main):361 - starting source: class io.airbyte.integrations.source.postgres.PostgresSource
2022-03-09 05:23:54 ERROR i.a.c.i.LineGobbler(voidCall):82 - SLF4J: Found binding in [jar:file:/airbyte/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2022-03-09 05:23:54 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-03-09 05:23:54 INFO i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {check=null, config=source_config.json}
2022-03-09 05:23:54 ERROR i.a.c.i.LineGobbler(voidCall):82 - SLF4J: Found binding in [jar:file:/airbyte/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2022-03-09 05:23:54 ERROR i.a.c.i.LineGobbler(voidCall):82 - SLF4J: See <http://www.slf4j.org/codes.html#multiple_bindings> for an explanation.
2022-03-09 05:23:54 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-03-09 05:23:54 INFO i.a.i.b.IntegrationRunner(runInternal):105 - Running integration: io.airbyte.integrations.base.ssh.SshWrappedSource
2022-03-09 05:23:54 ERROR i.a.c.i.LineGobbler(voidCall):82 - SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2022-03-09 05:23:54 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-03-09 05:23:54 INFO i.a.i.b.IntegrationRunner(runInternal):106 - Command: CHECK
2022-03-09 05:23:54 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-03-09 05:23:54 INFO i.a.i.b.IntegrationRunner(runInternal):107 - Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
2022-03-09 05:23:54 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-03-09 05:23:54 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-03-09 05:23:54 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-03-09 05:23:54 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-03-09 05:23:54 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-03-09 05:23:54 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-03-09 05:23:54 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-03-09 05:23:54 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword multiline - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-03-09 05:23:54 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-03-09 05:23:54 INFO i.a.i.b.s.SshTunnel(getInstance):170 - Starting connection with method: SSH_KEY_AUTH
👀 1
✅ 1
h
Hey when you try check do you see a new Mysql pod showing up?
m
@Harshith (Airbyte) yes. new pod showed up, it runs for some time and gets into completion state, but source didn't add. If I try to see the logs. Only
main
container logs appear.
h
Hey can you share logs of server and scheduler logs?
m
@Harshith (Airbyte) here are the logs of scheduler
Copy code
2022-03-09 05:14:04 INFO i.a.s.a.SchedulerApp(main):227 - workspaceRoot = /workspace
2022-03-09 05:14:05 INFO i.a.s.a.SchedulerApp(main):230 - temporalHost = airbyte-temporal-svc:7233
2022-03-09 05:14:07 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-09 05:14:11 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-09 05:14:15 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-09 05:14:19 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-09 05:14:22 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-09 05:14:24 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-09 05:14:26 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-09 05:14:28 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-09 05:14:30 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-09 05:14:32 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-09 05:14:34 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-09 05:14:36 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-09 05:14:38 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-09 05:14:40 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-09 05:14:42 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-09 05:14:44 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-09 05:14:46 INFO i.a.s.a.SchedulerApp(main):235 - Creating Job DB connection pool...
2022-03-09 05:14:46 WARN i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):66 - Waiting for database to become available...
2022-03-09 05:14:46 INFO i.a.d.i.j.JobsDatabaseInstance(lambda$static$2):25 - Testing if jobs database is ready...
2022-03-09 05:14:47 INFO i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):91 - Database available!
2022-03-09 05:14:47 INFO i.a.d.Databases(createPostgresDatabaseWithRetry):49 - Database available!
2022-03-09 05:14:47 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable CONFIG_DATABASE_USER: 'postgres_user'
2022-03-09 05:14:47 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable CONFIG_DATABASE_PASSWORD: '*****'
2022-03-09 05:14:47 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable CONFIG_DATABASE_URL: 'jdbc:<postgresql://cdpass-dev-postgres-instance.cxlcw2mudakk.us-east-2.rds.amazonaws.com:5432/airbyte_dev>'
2022-03-09 05:14:47 WARN i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):66 - Waiting for database to become available...
2022-03-09 05:14:47 INFO i.a.d.i.c.ConfigsDatabaseInstance(lambda$static$3):28 - Testing if airbyte_configs has been created and seeded...
2022-03-09 05:14:47 INFO i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):91 - Database available!
2022-03-09 05:14:47 INFO i.a.d.Databases(createPostgresDatabaseWithRetry):49 - Database available!
2022-03-09 05:14:47 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable SECRET_PERSISTENCE: 'NONE'
2022-03-09 05:14:47 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable SECRET_PERSISTENCE: 'NONE'
2022-03-09 05:14:47 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable SECRET_PERSISTENCE: 'NONE'
2022-03-09 05:14:47 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable MINIMUM_WORKSPACE_RETENTION_DAYS: '1'
2022-03-09 05:14:47 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable MAXIMUM_WORKSPACE_RETENTION_DAYS: '60'
2022-03-09 05:14:47 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable MAXIMUM_WORKSPACE_SIZE_MB: '5000'
2022-03-09 05:14:47 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable DEPLOYMENT_MODE: 'OSS'
2022-03-09 05:14:47 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable ACTIVITY_MAX_ATTEMPT: '10'
2022-03-09 05:14:47 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable ACTIVITY_MAX_TIMEOUT_SECOND: '30'
2022-03-09 05:14:47 INFO i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):174 - Waiting for temporal server...
2022-03-09 05:14:47 WARN i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):185 - Waiting for default namespace to be initialized in temporal...
2022-03-09 05:14:49 INFO i.t.s.WorkflowServiceStubsImpl(<init>):184 - Created GRPC client for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=airbyte-temporal-svc:7233}}
2022-03-09 05:14:54 INFO i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):201 - Found temporal default namespace!
2022-03-09 05:14:54 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable PUBLISH_METRICS: 'false'
2022-03-09 05:14:54 INFO i.a.s.a.SchedulerApp(main):278 - Launching scheduler...
2022-03-09 05:14:54 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable SYNC_JOB_MAX_ATTEMPTS: '3'
2022-03-09 05:14:54 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_MAIN_CONTAINER_CPU_REQUEST: 'null'
2022-03-09 05:14:54 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_MAIN_CONTAINER_CPU_LIMIT: 'null'
2022-03-09 05:14:54 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_MAIN_CONTAINER_MEMORY_REQUEST: 'null'
2022-03-09 05:14:54 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_MAIN_CONTAINER_MEMORY_LIMIT: 'null'
2022-03-09 05:14:54 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_KUBE_TOLERATIONS: ''
2022-03-09 05:14:54 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_KUBE_NODE_SELECTORS: ''
2022-03-09 05:14:54 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_SECRET: ''
2022-03-09 05:14:54 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_POLICY: 'IfNotPresent'
2022-03-09 05:14:54 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_KUBE_SOCAT_IMAGE: 'alpine/socat:1.7.4.1-r1'
2022-03-09 05:14:54 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_KUBE_BUSYBOX_IMAGE: 'busybox:1.28'
2022-03-09 05:14:54 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_KUBE_CURL_IMAGE: 'curlimages/curl:7.77.0'
2022-03-09 05:14:54 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable DEFAULT_WORKER_STATUS_CHECK_INTERVAL: 'PT30S'
2022-03-09 05:14:54 INFO i.a.c.f.EnvVariableFeatureFlags(usesNewScheduler):14 - New Scheduler: false
2022-03-09 05:14:54 INFO i.a.s.a.SchedulerApp(start):146 - Start running the old scheduler
h
Looks good to me. Server?
m
@Marcos Marx (Airbyte) They were bit lengthy. Here they are
@Marcos Marx (Airbyte) I've verified the SSH tunnel connection from external tool. It's working fine... But there is only issue with Airbyte... What do you think, I'm doing wrong?
Hey @Marcos Marx (Airbyte) I've resolved it. It was basically connectivity issue. Bastion host was not accepting traffic from network in which Airbyte pods are deployed. So I've added security group rule in bastion's security group allowing traffic from EKS pods at port 22. It worked. Thanks 🙂