https://linen.dev logo
Join Slack
Powered by
Hi Folks, Can data be migrated every time I make m...
# troubleshooting
s
Hi Folks, Can data be migrated every time I make modification to the source database using airbyte, rather then just waiting for it to sync every 5mins? Example -                    Source - Postgres Destination - MySQL If I make a change in Postgres - can it be transferred to mysql immediately or do I have to wait for the sync time - which has been set during connections?
a
Hey you can look at CDC. Here is the documentation https://docs.airbyte.com/understanding-airbyte/cdc
b
Can the data be transferred from mysql to postgres using API Call?
You mean can you start the sync with API call ?
I will be using API call in Nodered and I am looking for it, because I want the data to be migrated only when there is a modification done at the source DB
Technically Yes.
Yeah It's possible
I went through this link
is there any specific category that I should look for. I am a noob to API and Airbyte.
Great that's the API documentation we have
AUth has to be NO AUTH right?
as Its my localhost and I havent set any.
a
Nope. I can't run it. I also modified it to the below file.
the docker-compose has a bunch of environment variables - how is it being called from the .env file? I mean how does it even get to know that what variable gets which value. I checked the .env file which is in airbyte-tests-src-automatic.. - resources-0-32 env file. Question is how is it being called?
m
how is it being called from the .env file?
if you use the syntax like 
- AIRBYTE_VERSION=${VERSION}
docker-compose will read those variables from the env
a
Perfect. Thank you so much
@Marcos Marx (Airbyte) @Harshith (Airbyte) I was successful in setting up the airbyte using just one docker-compose (has other services too in it - like nodered, traccar, snipe it) now I am trying to connect to the databases - I am getting an error.
I have attached the my docker-compose which has all my services at the bottom
It is not even getting connected to any of my company servers too.
h
Hey what does the logs show?
a
any specific logs you want to see -
i mean any specific airbyte service logs?
the airbyte-webapp service has an error-
h
Can you also check scheduler log?
a
ok
do you think - there is any issue with the docker-compose file - which has multiple services in it.
h
In the UI when you click on create you will also find a section where you will find logs for that check connection can you share that? Also ensure that docker has enough memory as you are running multiple services it can be a proble,
a
ok
this one?
i do not think space is an issue.
h
Can you check the output for this
Copy code
docker run --rm --init -i -w /data/1b7a03e8-1d7c-4686-9424-e5ba47ecc361/0 --log-driver none --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e WORKER_JOB_ATTEMPT=0 -e WORKER_CONNECTOR_IMAGE=airbyte/source-mysql:0.5.6 -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_VERSION=0.35.51-alpha -e WORKER_JOB_ID=1b7a03e8-1d7c-4686-9424-e5ba47ecc361 airbyte/source-mysql:0.5.6 check --config source_config.json
a
SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jarfile/airbyte/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jarfile/airbyte/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 2022-03-16 124320 INFO i.a.i.s.m.MySqlSource(main):206 - starting source: class io.airbyte.integrations.source.mysql.MySqlSource 2022-03-16 124320 INFO i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {check=null, config=source_config.json} 2022-03-16 124320 INFO i.a.i.b.IntegrationRunner(run):88 - Sentry transaction event: 3dadca48d8814a1ba64dceb24126245e 2022-03-16 124320 INFO i.a.i.b.IntegrationRunner(runInternal):106 - Running integration: io.airbyte.integrations.base.ssh.SshWrappedSource 2022-03-16 124320 INFO i.a.i.b.IntegrationRunner(runInternal):107 - Command: CHECK 2022-03-16 124320 INFO i.a.i.b.IntegrationRunner(runInternal):108 - Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
h
That's it?
a
yupp
h
When you click on create source was there a new container created?
Got it. May be it's probably beyond me @Marcos Marx (Airbyte) can you help here?
a
hi.
Any update?
all the files are locked - as shown in the picture
can that be the reason/
I unlocked all the folder in this image.
h
Hey @Marcos Marx (Airbyte) is still in behind timeline he will be online in sometime
In the mean time is it possible to try some API based sources like github or something?
a
I am not sure about that. 😅
I have a question - Does Airbyte support TimescaleDB?
h
Hey no we don't support right now. But I think there is an issue open for it you can follow that
a
ok. and apart from that how about mariadb
h
I don't think so. We have our catalog here https://docs.airbyte.com/integrations/sources
a
the regular connections are even not happening.
Hi @Harshith (Airbyte), Did you check the docker-compose I sent - just in case, there might be something which can be fixed to make it work.
h
Sorry if I missed the file can you share again over DM may be
Open in Slack
PreviousNext
Powered by