https://linen.dev logo
j

Jay Patel

05/03/2021, 7:15 AM
Hello everyone! I want to build custom connectors but don't know how to start with it. Any reference for the same?
u

user

05/03/2021, 9:01 AM
Hi Jay! Are you looking to build Sources or Destinations?
u

user

05/03/2021, 9:03 AM
We offer template generation for sources. Here is some general documentation to start with
u

user

05/05/2021, 3:39 AM
Feel free to reach out if you have any questions!
u

user

05/10/2021, 4:47 AM
Hello @Davin Chia (Airbyte), Actually i need both source as well as destination for mariaDB.
u

user

05/10/2021, 9:24 AM
I see. We don't have great documentation for Java connectors, but there are good existing examples on how to implement one
u

user

05/10/2021, 9:26 AM
Here is the Postgres Destination and a Redshift Source as examples. You'll be able to take advantage of the
AbstractJdbcDestination
and
AbstractJdbcSource
to easily implement the MariaDB equivalent
u

user

05/10/2021, 9:26 AM
I'm also happy to set up a call and walk you through what needs to be done đŸ™‚
u

user

06/05/2021, 5:56 AM
Hi @Davin Chia (Airbyte), sorry for delay
u

user

06/05/2021, 6:10 AM
do i need to use CDK to implement this connector?
u

user

06/06/2021, 1:02 AM
Nope. Since these are Java
u

user

06/06/2021, 1:04 AM
But before you try to build a source, I suspect our MySQL source and destinations might be MariaDb compatible. Do you want to give those a shot first?
u

user

06/07/2021, 4:52 AM
i already did that.
u

user

06/07/2021, 4:53 AM
is there any chance if we can store exact formatted table in destination?
u

user

06/07/2021, 4:59 AM
can we store table in destination with same structure as source table?
u

user

06/07/2021, 5:06 AM
The normalization should produce a table that’s very similar
u

user

06/07/2021, 5:07 AM
Did you turn it on?
u

user

06/07/2021, 5:10 AM
no
u

user

06/07/2021, 5:11 AM
any link?
u

user

06/07/2021, 5:14 AM
it should be an option when setting up the connection
u

user

06/07/2021, 5:18 AM
ah! it's because we don't offer normalisation on MySql yet
u

user

06/07/2021, 5:18 AM
this is being worked on this sprint
u

user

06/07/2021, 5:55 AM
okay thanks.
u

user

06/07/2021, 5:56 AM
one last question, why airByte stores whole row as json format in destination table?
u

user

06/07/2021, 6:00 AM
good question
u

user

06/07/2021, 6:00 AM
this helps us decouple deliverying the data from normalising it
u

user

06/07/2021, 6:00 AM
some users don't like or normalisation rules and prefer to use DBT and transform the data in different ways
u

user

06/07/2021, 7:26 AM
okay, got it.
2 Views