https://linen.dev logo
#ask-community-for-troubleshooting
Title
# ask-community-for-troubleshooting
g

Grace Millard

07/08/2021, 2:17 PM
Hi , I have two questions about using Airbyte. 1. What are the best practice for switching from historical sync to incremental sync ? 2. How does Airbyte keep track of state, is it stored somewhere? It seems that Airbyte is reading our db before it writes to it, is this how it is keeping track of state?
j

Jeff Crooks

07/08/2021, 3:24 PM
1 - I think you would start as incremental, 1st load would be all history, and then append afterwards 2 - state is stored in airbyte db after the run. You can see in sync logs the values being set
octavia thanks 1
d

Davin Chia (Airbyte)

07/09/2021, 2:44 AM
Grace, did this answer your questions?
g

Grace Millard

07/09/2021, 7:08 PM
Yes thank you @Jeff Crooks
🔥 1
@Jeff Crooks @Davin Chia (Airbyte) Hi again, I'm trying to determine exactly what variable a connector I'm using (chargebee) is storing as state. I'm in the default airbyte-db postgres instance, do you know the name of the db and table that state gets stored in?
Sync log will have workspace #, then can review log output
Believe it shows up in the sync log as well via UI
d

Davin Chia (Airbyte)

07/16/2021, 6:02 AM
There should be
Jobs
table in the
Airbyte
database where status is stored
Yeap, this should appear in the logs. Let me create an example
e.g.
Copy code
2021-07-16 06:18:24 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-16 06:18:24 INFO i.a.i.b.IntegrationRunner(run):118 - {} - Completed integration: io.airbyte.integrations.destination.local_json.LocalJsonDestination
2021-07-16 06:18:24 INFO () DefaultReplicationWorker(run):144 - Destination thread complete.
2021-07-16 06:18:24 INFO () DefaultReplicationWorker(run):172 - sync summary: io.airbyte.config.ReplicationAttemptSummary@7d38f8ea[status=completed,recordsSynced=10,bytesSynced=51010,startTime=1626416275507,endTime=1626416304389]
2021-07-16 06:18:24 INFO () DefaultReplicationWorker(run):179 - Source output at least one state message
2021-07-16 06:18:24 INFO () DefaultReplicationWorker(run):185 - State capture: Updated state to: Optional[io.airbyte.config.State@1c06e567[state={"cdc":false,"streams":[{"stream_name":"tiny_table","stream_namespace":"public","cursor_field":["date_prod"],"cursor":"****************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************"}]}]]
2021-07-16 06:18:24 INFO () TemporalAttemptExecution(get):133 - Stopping cancellation check scheduling...
this are the logs from an incremental state I set up - you are able to see the cursor at the end of the job
here
******
is the actual cursor that is saved, and is a date field that I've selected
5 Views