Hi Team, Airbyte external schema/external view con...
# replication-troubleshooting
a
Hi Team, Airbyte external schema/external view contain schema are not discoverable in Airbyte for Redshift source connector:0.3.15 and I am getting this error :
Copy code
2022-11-29 08:39:36 INFO i.a.w.t.TemporalAttemptExecution(get):108 - Docker volume job log path: /tmp/workspace/b88f61d3-0dac-450d-be8e-890756668ee2/0/logs.log
2022-11-29 08:39:36 INFO i.a.w.t.TemporalAttemptExecution(get):113 - Executing worker wrapper. Airbyte version: 0.40.15
2022-11-29 08:39:36 INFO i.a.c.i.LineGobbler(voidCall):114 - Checking if airbyte/source-redshift:0.3.15 exists...
2022-11-29 08:39:36 INFO i.a.c.i.LineGobbler(voidCall):114 - airbyte/source-redshift:0.3.15 was found locally.
2022-11-29 08:39:36 INFO i.a.w.p.DockerProcessFactory(create):119 - Creating docker container = source-redshift-discover-b88f61d3-0dac-450d-be8e-890756668ee2-0-meimj with resources io.airbyte.config.ResourceRequirements@66d3a24c[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=]
2022-11-29 08:39:36 INFO i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/b88f61d3-0dac-450d-be8e-890756668ee2/0 --log-driver none --name source-redshift-discover-b88f61d3-0dac-450d-be8e-890756668ee2-0-meimj --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=true -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e WORKER_JOB_ATTEMPT=0 -e WORKER_CONNECTOR_IMAGE=airbyte/source-redshift:0.3.15 -e AIRBYTE_VERSION=0.40.15 -e WORKER_JOB_ID=b88f61d3-0dac-450d-be8e-890756668ee2 airbyte/source-redshift:0.3.15 discover --config source_config.json
2022-11-29 08:39:37 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):109 - starting source: class io.airbyte.integrations.source.redshift.RedshiftSource
2022-11-29 08:39:38 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):109 - integration args: {discover=null, config=source_config.json}
2022-11-29 08:39:38 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):109 - Running integration: io.airbyte.integrations.source.redshift.RedshiftSource
2022-11-29 08:39:38 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):109 - Command: DISCOVER
2022-11-29 08:39:38 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):109 - Integration config: IntegrationConfig{command=DISCOVER, configPath='source_config.json', catalogPath='null', statePath='null'}
2022-11-29 08:39:38 WARN i.a.w.i.DefaultAirbyteStreamFactory(internalLog):106 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-11-29 08:39:38 WARN i.a.w.i.DefaultAirbyteStreamFactory(internalLog):106 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-11-29 08:39:38 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):109 - HikariPool-1 - Starting...
2022-11-29 08:39:38 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):109 - HikariPool-1 - Start completed.
2022-11-29 08:39:39 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):109 - HikariPool-1 - Driver does not support get/set network timeout for connections. ([Amazon][JDBC](10220) Driver does not support this optional feature.)
2022-11-29 08:39:39 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):109 - Internal schemas to exclude: [catalog_history, information_schema, pg_catalog, pg_internal]
2022-11-29 08:39:40 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):109 - HikariPool-1 - Shutdown initiated...
2022-11-29 08:39:40 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):109 - HikariPool-1 - Shutdown completed.
2022-11-29 08:39:40 ERROR i.a.w.i.DefaultAirbyteStreamFactory(internalLog):105 - Something went wrong in the connector. See the logs for more details.
Stack Trace: java.lang.NullPointerException: null value in entry: isNullable=null
	at com.google.common.collect.CollectPreconditions.checkEntryNotNull(CollectPreconditions.java:33)
	at com.google.common.collect.ImmutableMapEntry.<init>(ImmutableMapEntry.java:54)
	at com.google.common.collect.ImmutableMap.entryOf(ImmutableMap.java:339)
	at com.google.common.collect.ImmutableMap$Builder.put(ImmutableMap.java:449)
	at io.airbyte.integrations.source.jdbc.AbstractJdbcSource.getColumnMetadata(AbstractJdbcSource.java:267)
	at io.airbyte.db.jdbc.JdbcDatabase$1.tryAdvance(JdbcDatabase.java:81)
	at java.base/java.util.Spliterator.forEachRemaining(Spliterator.java:332)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at io.airbyte.db.jdbc.DefaultJdbcDatabase.bufferedResultSetQuery(DefaultJdbcDatabase.java:56)
	at io.airbyte.integrations.source.jdbc.AbstractJdbcSource.discoverInternal(AbstractJdbcSource.java:190)
	at io.airbyte.integrations.source.redshift.RedshiftSource.discoverInternal(RedshiftSource.java:90)
	at io.airbyte.integrations.source.redshift.RedshiftSource.discoverInternal(RedshiftSource.java:30)
	at io.airbyte.integrations.source.relationaldb.AbstractDbSource.discoverWithoutSystemTables(AbstractDbSource.java:241)
	at io.airbyte.integrations.source.relationaldb.AbstractDbSource.getTables(AbstractDbSource.java:498)
	at io.airbyte.integrations.source.relationaldb.AbstractDbSource.discover(AbstractDbSource.java:110)
	at io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:127)
	at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:97)
	at io.airbyte.integrations.source.redshift.RedshiftSource.main(RedshiftSource.java:136)
2022-11-29 08:39:40 INFO i.a.w.t.TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling...
Resolved the issue. Redshift source connector version is downgraded to 0.3.10.
m
@ANISH R do yo mind creating a Github issue for this?
a
u
Please look my comment in the issue.