https://linen.dev logo
Join Slack
Powered by
Hello everyone, I'm trying to create an `Increment...
# replication-troubleshooting
k
Hello everyone, I'm trying to create an 
Incremental | Deduped + history
table by migrating from redshift to postgres but, I encountered a couple of errors along the way. I'm using a 
datetime
column as the 
cursor field
: 1- I was first getting an error from airbyte saying that the table 
*_stg
needed a 
REPLICA IDENTITY
. I then added the field manually with 
ALTER TABLE analytics.revenue REPLICA IDENTITY FULL;
2- Now I'm getting the same error (log file attached) but with the 
*_scd
table. But the issue seems to be that it is having problems created the 
_scd
table in the first place.
logs-15681.txt
🥲 1
✍️ 1
u
@[DEPRECATED] Marcos Marx turned this message into Zendesk ticket 2712 to ensure timely resolution!
k
Hello, I managed to partially solve the issue. I had to manually create the 
_scd
table using the query in the logs, then I had to manually set the tables in the RDS to set the table replication identity to full using 
alter table tablename replica identity full;
Any reason why airbyte is having issues created the 
_scd
during the sync?
the version I'm using is 
0.39.41
could there be a fix to this issue in a new release?
h
Hey 2 things 1. Could you try full refresh and see if that is working? 2. Incase above is not could you create a issue in github?
k
Hey, thanks for getting back at me. Full refresh does work
h
Could you try with a fresh connection (incremental) and see if that is helping?
k
Yeah I already did that, but was still failing.
h
Got it. Could you create a github issue on this with the logs so that team can look and get back to you
Copy code
2022-10-10 20:27:45 [44msource[0m > java.util.NoSuchElementException: No value present
2022-10-10 20:27:45 [44msource[0m > 	at java.util.Optional.orElseThrow(Optional.java:377) ~[?:?]
2022-10-10 20:27:45 [44msource[0m > 	at io.airbyte.integrations.source.relationaldb.AbstractDbSource.getIncrementalStream(AbstractDbSource.java:284) ~[io.airbyte.airbyte-integrations.connectors-source-relational-db-0.39.37-alpha.jar:?]
2022-10-10 20:27:45 [44msource[0m > 	at io.airbyte.integrations.source.relationaldb.AbstractDbSource.createReadIterator(AbstractDbSource.java:236) ~[io.airbyte.airbyte-integrations.connectors-source-relational-db-0.39.37-alpha.jar:?]
2022-10-10 20:27:45 [44msource[0m > 	at io.airbyte.integrations.source.relationaldb.AbstractDbSource.getSelectedIterators(AbstractDbSource.java:201) ~[io.airbyte.airbyte-integrations.connectors-source-relational-db-0.39.37-alpha.jar:?]
2022-10-10 20:27:45 [44msource[0m > 	at io.airbyte.integrations.source.relationaldb.AbstractDbSource.getIncrementalIterators(AbstractDbSource.java:172) ~[io.airbyte.airbyte-integrations.connectors-source-relational-db-0.39.37-alpha.jar:?]
2022-10-10 20:27:45 [44msource[0m > 	at io.airbyte.integrations.source.relationaldb.AbstractDbSource.read(AbstractDbSource.java:129) ~[io.airbyte.airbyte-integrations.connectors-source-relational-db-0.39.37-alpha.jar:?]
2022-10-10 20:27:45 [44msource[0m > 	at io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:155) ~[io.airbyte.airbyte-integrations.bases-base-java-0.39.37-alpha.jar:?]
2022-10-10 20:27:45 [44msource[0m > 	at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:107) ~[io.airbyte.airbyte-integrations.bases-base-java-0.39.37-alpha.jar:?]
2022-10-10 20:27:45 [44msource[0m > 	at io.airbyte.integrations.source.redshift.RedshiftSource.main(RedshiftSource.java:126) ~[io.airbyte.airbyte-integrations.connectors-source-redshift-0.39.37-alpha.jar:?]
2022-10-10 20:27:45 [46mreplication-orchestrator[0m > Unknown keyword existingJavaType - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-10-10 20:27:45 [46mreplication-orchestrator[0m > Source has no more messages, closing connection.
My guess is something wrong with the data but yeah suggest you to create a github issue
k
got it, will do
Open in Slack
PreviousNext
Powered by