https://linen.dev logo
Join Slack
Powered by
Hello everyone! I have a technical question: when ...
# announcements
a
Hello everyone! I have a technical question: when a source is using incremental, i see the command to start it has the flag 
--state input_state.json
My question is, where is the 
input_state.json
generated? Is it the scheduler that creates the file before running the command or a different machine? Also, where is this data stored?
✅ 1
c
Yes, the scheduler queries the Airbyte Internal Database where it stores these states
In the admin panel, you can export a dump of your airbyte instance, if you browse the produced archive, you have a folder called 
airbyte_db
where you can see the content of the database Or you can tweak the docker-compose file to allow a connection to be established to the postgres that airbyte is using internally
a
Yes i have a connection to the internal postgres db already!
But where is the state stored?
c
In the jobs table
It’s in json format though
a
oh so it queries the job table for the last time it executed that specific job and collects the state from there?
c
Yes, i think so
m
@Alberto Costa can you expand on what changes you made in order to setup a connection to the airbyte internal db? My use case is to export this jobs table to a data warehouse to easily query the last syncs
u
Hello @Manan Kshatriya you should be able to access the internal db reading this docs: https://docs.airbyte.io/operator-guides/configuring-airbyte-db
m
thanks. Dont know how I missed this doc
2 Views
Open in Slack
PreviousNext
Powered by