Hi , I have two questions about using Airbyte.
1....
# ask-community-for-troubleshootingg
Grace Millard
07/08/2021, 2:17 PMHi , I have two questions about using Airbyte.
1. What are the best practice for switching from historical sync to incremental sync ?
2. How does Airbyte keep track of state, is it stored somewhere? It seems that Airbyte is reading our db before it writes to it, is this how it is keeping track of state?
j
Jeff Crooks
07/08/2021, 3:24 PM1 - I think you would start as incremental, 1st load would be all history, and then append afterwards
2 - state is stored in airbyte db after the run. You can see in sync logs the values being set
octavia thanks 1
d
Davin Chia (Airbyte)
07/09/2021, 2:44 AM
Grace, did this answer your questions?
g
Grace Millard
07/09/2021, 7:08 PMYes thank you @Jeff Crooks
🔥 1
Grace Millard
07/15/2021, 9:02 PM@Jeff Crooks @Davin Chia (Airbyte)
Hi again,
I'm trying to determine exactly what variable a connector I'm using (chargebee) is storing as state.
I'm in the default airbyte-db postgres instance, do you know the name of the db and table that state gets stored in?
j
Jeff Crooks
07/15/2021, 10:54 PMJeff Crooks
07/15/2021, 10:55 PMSync log will have workspace #, then can review log output
Jeff Crooks
07/15/2021, 10:55 PMBelieve it shows up in the sync log as well via UI
d
Davin Chia (Airbyte)
07/16/2021, 6:02 AM
There should be
JobsAirbyteDavin Chia (Airbyte)
07/16/2021, 6:04 AM
Yeap, this should appear in the logs. Let me create an example
Davin Chia (Airbyte)
07/16/2021, 6:19 AM
e.g.
Copy code
2021-07-16 06:18:24 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-16 06:18:24 INFO i.a.i.b.IntegrationRunner(run):118 - {} - Completed integration: io.airbyte.integrations.destination.local_json.LocalJsonDestination
2021-07-16 06:18:24 INFO () DefaultReplicationWorker(run):144 - Destination thread complete.
2021-07-16 06:18:24 INFO () DefaultReplicationWorker(run):172 - sync summary: io.airbyte.config.ReplicationAttemptSummary@7d38f8ea[status=completed,recordsSynced=10,bytesSynced=51010,startTime=1626416275507,endTime=1626416304389]
2021-07-16 06:18:24 INFO () DefaultReplicationWorker(run):179 - Source output at least one state message
2021-07-16 06:18:24 INFO () DefaultReplicationWorker(run):185 - State capture: Updated state to: Optional[io.airbyte.config.State@1c06e567[state={"cdc":false,"streams":[{"stream_name":"tiny_table","stream_namespace":"public","cursor_field":["date_prod"],"cursor":"****************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************"}]}]]
2021-07-16 06:18:24 INFO () TemporalAttemptExecution(get):133 - Stopping cancellation check scheduling...Davin Chia (Airbyte)
07/16/2021, 6:20 AM
this are the logs from an incremental state I set up - you are able to see the cursor at the end of the job
Davin Chia (Airbyte)
07/16/2021, 6:20 AM
here
******5 Views