Hi all, I developed a new source connector using t...
# help-connector-development
m
Hi all, I developed a new source connector using the Connector Builder, is it possible to see in the logs of a job what was the cursor timestamp used during a sync?
k
The knowledge sources don’t document any built‑in way in Connector Builder to show the exact cursor timestamp used for a sync in the job logs. What is documented is: - Airbyte sources emit state messages containing the most recently emitted cursor, and Airbyte persists this state in its internal Postgres database; you can inspect it directly via the
state
table. [[Incremental tutorial](https://airbyte.com/tutorials/incremental-data-synchronization#step-2)] - There is an open GitHub issue requesting better logging of HTTP calls and cursor values for debugging custom connectors, which implies this level of detail is not currently available by default in logs. [[Log HTTP calls issue](https://github.com/airbytehq/airbyte/issues/52077)] So, based on the available docs: - You can see the stored cursor value by querying the internal
state
table, but - There is no documented option in Connector Builder or the UI logs to directly display “cursor timestamp used for this sync” in the job logs. If you need that visibility, the current path (per the issue) is to rely on state inspection or follow/participate in the feature request for enhanced logging. [[Incremental tutorial](https://airbyte.com/tutorials/incremental-data-synchronization#step-2); [Log HTTP calls issue](https://github.com/airbytehq/airbyte/issues/52077)]
m
@kapa.ai, how to query internal postgres table state?
k
A message from kapa.ai
You can query the internal
state
table by connecting directly to Airbyte’s internal Postgres database and running a simple SQL query. 1. Open a shell into the Airbyte DB container and connect to Postgres:
Copy code
bash docker exec -ti airbyte-db psql -U docker -d airbyte
2. Once in
psql
, query the
state
table:
Copy code
sql SELECT * FROM state;
This will show you the stored state JSON (including cursor values) per connection. [[Incremental sync tutorial](https://airbyte.com/tutorials/incremental-data-synchronization#step-2)]
m
@kapa.ai, if I deploying in my machine using the abctl how can I query the state table?
k
A message from kapa.ai
When you deploy with
abctl
, Airbyte runs inside a local Kubernetes (kind) cluster, so you can’t use
docker exec airbyte-db
directly. The knowledge sources don’t document an explicit, step‑by‑step way to reach the internal Postgres in an
abctl
deployment, so the exact commands may differ by version and setup. I’ll outline what is documented and where the gap is. What we know: -
abctl
installs Airbyte into a kind cluster on your local machine. You can interact with resources in that cluster using standard tools like
kubectl
and
helm
, pointing them at the kubeconfig created by
abctl
(for example:
kubectl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig --namespace airbyte-abctl get pods
). [[abctl troubleshoot FAQ](https://docs.airbyte.com/platform/deploying-airbyte/troubleshoot-deploy#faq)] - In a Docker deployment (not
abctl
), you can connect to the internal DB with:
docker exec -ti airbyte-db psql -U docker -d airbyte
and then
SELECT * FROM state;
. [[Incremental sync tutorial](https://airbyte.com/tutorials/incremental-data-synchronization#step-2)] What is not documented in the provided sources: - The exact pod name, service name, or
psql
command to reach the internal Postgres when using
abctl
. - A ready‑made
kubectl exec
or port‑forward example specifically for querying the
state
table in an
abctl
deployment. Given this, the documented approach is: 1. Use
kubectl
with the
abctl
kubeconfig to inspect the cluster and find the Postgres pod/service:
Copy code
bash    kubectl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig --namespace airbyte-abctl get pods
[[abctl troubleshoot FAQ](https://docs.airbyte.com/platform/deploying-airbyte/troubleshoot-deploy#faq)] 2. Once you identify the Postgres pod and credentials, you would
kubectl exec
into that pod and run
psql
similarly to the Docker example, then:
Copy code
sql    SELECT * FROM state;
[[Incremental sync tutorial](https://airbyte.com/tutorials/incremental-data-synchronization#step-2)] However, because the knowledge sources don’t specify the exact pod name, DB user, or DB name for the
abctl
deployment, I can’t safely give you concrete one‑line commands without guessing. If you’d like, you can share the output of:
Copy code
bash kubectl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig --namespace airbyte-abctl get pods
and I can help you interpret it and suggest the next
kubectl exec
step, staying within what’s visible from your environment.
m
@kapa.ai, there is a pod called airbyte-db-0
k
A message from kapa.ai
Since you’re using
abctl
, you should connect to Postgres via
kubectl exec
into the
airbyte-db-0
pod, using the kubeconfig and namespace that
abctl
created. The docs show how to target that cluster and namespace; the pod name
airbyte-db-0
matches what’s documented. [[Developing locally 1.7](https://docs.airbyte.com/platform/1.7/contributing-to-airbyte/developing-locally#using-abctl-for-airbyte-development); [abctl troubleshoot](https://docs.airbyte.com/platform/deploying-airbyte/troubleshoot-deploy#faq)] From the knowledge sources, the pattern is: 1. Point
kubectl
at the
abctl
cluster:
Copy code
bash kind export kubeconfig -n airbyte-abctl kubectl get pods -n airbyte-abctl
You’ve already confirmed there is a pod
airbyte-db-0
. [[Developing locally 1.7](https://docs.airbyte.com/platform/1.7/contributing-to-airbyte/developing-locally#using-abctl-for-airbyte-development)] 2. Exec into the DB pod and open
psql
(user/db names are not explicitly documented for
abctl
, so this part may require you to adjust based on your setup):
Copy code
bash kubectl exec -it -n airbyte-abctl airbyte-db-0 -- /bin/bash # inside the pod, run something like: psql -U airbyte db-airbyte
This
psql
pattern (
psql -U airbyte db-airbyte
) is what’s shown in the Helm/cluster docs and GitHub issue examples, but the knowledge sources do not explicitly confirm it for
abctl
, so treat it as a likely value, not guaranteed. [[Helm V2 upgrade](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#how-to-upgrade); [GitHub dataplane issue](https://github.com/airbytehq/airbyte/issues/58587)] 3. Once in
psql
, query the
state
table as in the incremental sync tutorial:
Copy code
sql SELECT * FROM state;
This is the documented way to inspect stored state (including cursor info). [[Incremental sync tutorial](https://airbyte.com/tutorials/incremental-data-synchronization#step-2)] Because the exact DB name/user for
abctl
are not explicitly documented in the provided sources, if
psql -U airbyte db-airbyte
fails, you’ll need to inspect the pod’s environment or Postgres config to find the correct credentials.