https://linen.dev logo
Join Slack
Powered by
Hi <@U04SE163WC8> I have created an Airbyte source...
# ask-ai
a
Hi @kapa.ai I have created an Airbyte source using Postgres, I configured the "update method" to be CDC/WAL I triggered the sync job and it completed successfully, now the database has no changes 
insert/update/delete
at the moment (from end of working day till next working day), the sync job runs in this timeframe keep running, and the job logs states
Copy code
2024-07-16 12:08:32 source > INFO pool-2-thread-1 i.d.c.c.BaseSourceTask(logStatistics):323 64 records sent during previous 00:10:47.363, last recorded offset of {server=*} partition is {lsn_proc=6177707990192, messageType=UPDATE, lsn_commit=6177707990192, lsn=6177707990192, txId=665484, ts_usec=1721130110047951}
2024-07-16 12:17:59 source > INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):49 CDC events queue size: 0. remaining 10000
I am sure 100% that no transactions are applied to the database, what does "64 records sent during previous 001047.363" mean? and how to handle this case, no changes on the database, I need the sync job to do some test, and then finish the sync job successfully
k
The "64 records sent during previous 001047.363" message in the logs indicates that the sync job has processed 64 records during the last 10 minutes and 47.363 seconds. However, this doesn't necessarily mean that these records represent changes in your source database. In Airbyte's incremental data synchronization, if a sync is executed without making any modifications on the source database, no records will be emitted, and no changes will be made on the destination database [source](https://airbyte.com/tutorials/incremental-data-synchronization#step-7). However, in the case of full refresh data synchronization, even if no data was modified in the Postgres source, the entire table is copied and overwritten in the destination [source](https://airbyte.com/tutorials/full-data-synchronization#step-7). If you are sure that no changes have been made to the database and you are still seeing records being processed, it might be due to some internal operations of the CDC mechanism or the connector itself. If you are experiencing issues or unexpected behavior, it would be helpful to investigate the logs further or raise an issue in the Airbyte GitHub repository or discuss it in the Airbyte forum for more specific guidance.
a
@kapa.ai ignore the point related to records lets focus on how to handle the case when no changes were applied to the database, and the sync job started for streams with sync type ="Incremental | Append + Deduped" I need to know how to make the job finishes if no changes are found
k
A message from kapa.ai
In Airbyte's incremental data synchronization, if a sync is executed without making any modifications on the source database, no records will be emitted, and no changes will be made on the destination database. This is true for both "Incremental | Append" and "Incremental | Append + Deduped" sync modes [source](https://airbyte.com/tutorials/incremental-data-synchronization#step-7). However, the sync job itself does not automatically stop if no changes are detected. The job will continue to run based on the schedule you've set for it. If you want to stop the job manually, you can do so from the Airbyte UI by clicking on the "Cancel Job" button. If you want the sync job to automatically stop if no changes are detected, this is currently not a built-in feature in Airbyte. You might need to implement a custom solution outside of Airbyte to achieve this, such as a script that checks for changes in the source database and only triggers the Airbyte sync job if changes are detected. If you believe this would be a useful feature, you can suggest it on the [Airbyte GitHub repository](https://github.com/airbytehq/airbyte/issues) as a feature request.
a
@kapa.ai based on what you stated above
However, the sync job itself does not automatically stop if no changes are detected. The job will continue to run based on the schedule you've set for it.
for such a running job, when will it stop? when the next schedule time comes? when new modification to the database tables are executed? another trigger?
k
A message from kapa.ai
In Airbyte, a sync job will continue to run until it has finished processing all the data from the source, regardless of whether any changes were detected or not. The job does not automatically stop if no changes are detected in the source data. The duration of a sync job can depend on several factors, including the size of the source data, the performance of the source and destination systems, and the network speed. Once a sync job has started, it will not stop when the next schedule time comes. Instead, the next scheduled job will be queued and will start once the current job has finished. If new modifications are made to the source database while a sync job is running, these changes will not be picked up by the current job. They will be included in the next sync job. If you want to stop a running sync job manually, you can do so from the Airbyte UI. If a sync job is running longer than expected, it might be worth checking the logs for any issues or errors.
Open in Slack
PreviousNext
Powered by