<@U04S8PLJWMT> I want to do a loop call to a sourc...
# ask-ai
m
@kapa.ai I want to do a loop call to a source API because one call only returns 200 records and there close to 100k rows, I need to be able to get all the data and go into an incremental mode after
k
To fetch all records from an API that returns only 200 records per call and has around 100,000 rows, you should use the pagination feature in Airbyte's Connector Builder. Specifically, you can use the Offset Increment pagination method if your API supports
offset
and
limit
parameters. This will allow Airbyte to automatically loop through all pages until all data is retrieved. How to configure: 1. Pagination method: Offset Increment 2. Limit: Set to 200 (or the maximum allowed by your API) 3. Inject limit into outgoing HTTP request: - Inject into: request_parameter - Field name: limit 4. Inject offset into outgoing HTTP request: - Inject into: request_parameter - Field name: offset Airbyte will keep incrementing the offset by the limit (200) and continue fetching until it receives fewer records than the limit, ensuring all ~100k records are retrieved. Here’s an example of how the requests would look:
Copy code
GET https://api.example.com/data?limit=200&offset=0 GET https://api.example.com/data?limit=200&offset=200 GET https://api.example.com/data?limit=200&offset=400 ...
Once the initial full load is complete, you can switch to Incremental Sync mode by specifying a suitable cursor field (like a timestamp or incrementing ID). Airbyte will then only fetch new or updated records in subsequent syncs, using the cursor to track progress. The connector maintains state between syncs to resume from where the last sync left off, ensuring efficient incremental updates [Pagination methods](https://docs.airbyte.com/platform/1.8/connector-development/connector-builder-ui/pagination#offset-increment), [Supported sync modes](https://docs.airbyte.com/integrations/sources/bigquery#supported-sync-modes). If your API uses a different pagination mechanism (like page numbers or cursors), Airbyte also supports those methods. If none of the standard methods fit, you can implement a custom pagination strategy. Summary: - Use Offset Increment pagination to fetch all records in batches of 200. - After the initial load, configure Incremental Sync with a suitable cursor field for ongoing updates. Let me know if you need details for a different pagination method or help with incremental sync configuration!
m
@kapa.ai I added the paginator option when the stream through the WebUI , it started reading rows until 11401 rows and then it stayed idle for 45 mins. I had to cancel the sync. what could me the issue ?
k
A message from kapa.ai
When a sync using pagination gets stuck after reading a certain number of rows (like 11,401) and then stays idle, common causes include: - Pagination stop condition not being met: If the stop condition is not configured correctly, Airbyte may keep waiting for more data or for a signal to stop, resulting in the sync appearing idle. Double-check that your stop condition is set up to evaluate to true when there are no more pages to fetch. - Cursor or offset extraction issues: If the cursor or offset value is not being extracted correctly from the API response, Airbyte may not be able to construct the next request, causing the sync to stall. - API response issues: If the API starts returning the same data repeatedly, or if it stops returning new data, Airbyte may not detect the end of the data set. - API limits or errors: Some APIs have undocumented limits or may start rate-limiting or blocking requests after a certain threshold, which can cause the sync to hang. Troubleshooting steps: - Check the stop condition logic in your pagination configuration. - Ensure the cursor or offset is being correctly extracted from the API response. - Review the API responses at the point where the sync stalls to see if there is an error or repeated data. - Check for any rate limit headers or error messages in the logs. - Review Airbyte logs for any memory or resource issues, as large syncs can sometimes cause resource exhaustion, leading to stalled jobs [Pagination Troubleshooting](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#troubleshooting). If you continue to experience issues, reviewing the sync logs and monitoring resource usage (CPU, memory) during the sync can provide more insight into the root cause. If the problem persists, consider sharing the logs with the Airbyte community for further assistance. For more detailed troubleshooting tips, see the [Pagination Troubleshooting documentation](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#troubleshooting).
m
@kapa.ai where is the Check the stop condition logic in your pagination configuration, while using the WebUI Builder ?
k
A message from kapa.ai
In the Airbyte WebUI Connector Builder, the stop condition for pagination is configured within the Pagination section when you select a pagination method (such as Cursor Pagination or Custom Pagination Strategy). The stop condition is typically a Jinja template that tells Airbyte when to stop paginating. You will find the stop condition field as part of the pagination configuration options. For example, when using Cursor Pagination, you can specify a stop condition like:
Copy code
{{ response.has_more is false }}
or
Copy code
{{ 'next' not in headers['link'] }}
This field is available when you choose a pagination method that supports custom stop conditions, such as Cursor Pagination or Custom Pagination Strategy. The stop condition uses variables from the API response or headers to determine when to stop fetching more pages. If you do not set this correctly, Airbyte may continue paginating indefinitely or stop too early. For more details and examples, see the [Pagination documentation](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#advanced-cursor-pagination-features) and the [real-world examples section](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#real-world-examples).
m
@kapa.ai I am using Page Increment as Pagination strategy, so how to set the Stop condition ?
k
A message from kapa.ai
When using the Page Increment pagination strategy in Airbyte, you do not need to manually set a stop condition. The Connector Builder automatically stops paginating when a page returns fewer records than the configured page size. This behavior is built-in and does not require additional configuration from you. For example, if your page size is set to 200, Airbyte will continue incrementing the page number and fetching new pages until it receives a page with fewer than 200 records, at which point it will stop fetching more pages. This is described in the documentation: > Either way, your connector will automatically increment the page number by 1 for each subsequent request, and continue until it receives fewer records than the page size you configured. [Page Increment - Pagination methods](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#page-increment) So, with Page Increment, you do not set a stop condition manually—the Connector Builder handles it for you.
m
@kapa.ai is it possible to have a sync process where data is written to target page by page or per defined number of rows ? I am thinking that the 11401 rows where never written to target, and the sync process may be does a one time insert to target. is that the case?
k
A message from kapa.ai
Based on the available knowledge sources, Airbyte’s sync process does not write data to the target page by page or per a defined number of rows as each page is fetched. Instead, Airbyte typically reads data from the source (using pagination if needed), and then writes the data to the destination in batches, but the documentation does not specify that these batches correspond directly to the pages fetched from the source. The documentation does not explicitly state whether Airbyte buffers all data before writing or writes incrementally as each page is fetched. However, it is common for ETL tools to buffer data and then write in larger batches for efficiency, which could explain why you observed that the 11,401 rows were not written to the target before the sync was cancelled. If you suspect that data is not being written until the end of the sync, this could be due to how Airbyte handles batching and writing to the destination, but the knowledge sources do not provide detailed information about the exact timing or granularity of writes to the target during a sync. If you need more granular control over when data is written to the destination (e.g., after each page), this is not described as a configurable option in the current documentation. If this behavior is critical for your use case, you may want to consult Airbyte’s community or support channels for more implementation details or feature requests. Summary: - The knowledge sources do not explicitly state that Airbyte writes to the target page by page or per a defined number of rows. - It is likely that data is written in larger batches, not after each page, but the exact behavior is not detailed in the documentation. - If the sync is interrupted, data fetched but not yet written may be lost, depending on when Airbyte performs the write operation. For more on sync modes and data flow, see [Sync Modes documentation](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes).
m
@kapa.ai attached a log for the given sync. may be you can identify something in it.
k
Sorry @MTA, it looks like your question contains files, which I cannot process. Please reformulate your question without including any files.
m
@kapa.ai below a paste of the log of the last sync. may be you can help identify what is the issue.
k
Sorry @MTA, it looks like your question contains files, which I cannot process. Please reformulate your question without including any files.
m
@kapa.ai identify from this log the issue of the sync
k
Sorry @MTA, it looks like your question contains files, which I cannot process. Please reformulate your question without including any files.
m
this bloc of message keeps coming in the log after it had read the 11401 rows, until i had stopped the sync manually. can you explain this message that repeats ? 2025-11-21 152721 info INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):172 Flush Worker (28344) -- Worker finished flushing. Current queue size: 0 2025-11-21 152757 info INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), %% used: 0.03367428551701215 | Queue
INC_Contacts
, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0 2025-11-21 152757 info INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):129 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0 2025-11-21 152857 info INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), %% used: 0.03367428551701215 | Queue
INC_Contacts
, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0 2025-11-21 152857 info INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):129 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0 2025-11-21 152957 info INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), %% used: 0.03367428551701215 | Queue
INC_Contacts
, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0 2025-11-21 152957 info INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):129 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0 2025-11-21 153057 info INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), %% used: 0.03367428551701215 | Queue
INC_Contacts
, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0 2025-11-21 153057 info INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):129 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0 2025-11-21 153157 info INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), %% used: 0.03367428551701215 | Queue
INC_Contacts
, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0 2025-11-21 153157 info INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):129 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0 2025-11-21 153221 info INFO pool-5-thread-1 i.a.c.i.d.a.DetectStreamToFlush(getNextStreamToFlush):119 flushing: trigger info: null - INC_Contacts, time trigger: true , size trigger: false current threshold b: 25 MB, queue size b: 0 bytes, penalty b: 0 bytes, after penalty b: 0 bytes 2025-11-21 153221 info INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):137 Flush Worker (ba23b) -- Worker picked up work. 2025-11-21 153221 info INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):143 Flush Worker (ba23b) -- Attempting to read from queue namespace: null, stream: INC_Contacts. 2025-11-21 153221 info INFO pool-4-thread-2 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes.. 2025-11-21 153221 info INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):161 Flush Worker (ba23b) -- Batch contains: 0 records, 0 bytes bytes. 2025-11-21 153221 info INFO pool-4-thread-2 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):153 Flushing states 2025-11-21 153221 info INFO pool-4-thread-2 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):207 Flushing states complete
@kapa.ai this bloc of message keeps coming in the log after it had read the 11401 rows, until i had stopped the sync manually. can you explain this message that repeats ? 2025-11-21 152721 info INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):172 Flush Worker (28344) -- Worker finished flushing. Current queue size: 0 2025-11-21 152757 info INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), %% used: 0.03367428551701215 | Queue
INC_Contacts
, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0 2025-11-21 152757 info INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):129 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0 2025-11-21 152857 info INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), %% used: 0.03367428551701215 | Queue
INC_Contacts
, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0 2025-11-21 152857 info INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):129 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0 2025-11-21 152957 info INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), %% used: 0.03367428551701215 | Queue
INC_Contacts
, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0 2025-11-21 152957 info INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):129 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0 2025-11-21 153057 info INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), %% used: 0.03367428551701215 | Queue
INC_Contacts
, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0 2025-11-21 153057 info INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):129 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0 2025-11-21 153157 info INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), %% used: 0.03367428551701215 | Queue
INC_Contacts
, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0 2025-11-21 153157 info INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):129 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0 2025-11-21 153221 info INFO pool-5-thread-1 i.a.c.i.d.a.DetectStreamToFlush(getNextStreamToFlush):119 flushing: trigger info: null - INC_Contacts, time trigger: true , size trigger: false current threshold b: 25 MB, queue size b: 0 bytes, penalty b: 0 bytes, after penalty b: 0 bytes 2025-11-21 153221 info INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):137 Flush Worker (ba23b) -- Worker picked up work. 2025-11-21 153221 info INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):143 Flush Worker (ba23b) -- Attempting to read from queue namespace: null, stream: INC_Contacts. 2025-11-21 153221 info INFO pool-4-thread-2 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes.. 2025-11-21 153221 info INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):161 Flush Worker (ba23b) -- Batch contains: 0 records, 0 bytes bytes. 2025-11-21 153221 info INFO pool-4-thread-2 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):153 Flushing states 2025-11-21 153221 info INFO pool-4-thread-2 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):207 Flushing states complete
k
A message from kapa.ai
The repeated log messages you are seeing, such as:
Copy code
Flush Worker (...) -- Worker finished flushing. Current queue size: 0 [ASYNC QUEUE INFO] ... num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes ... [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
indicate that Airbyte's destination flush workers and buffer manager are idle—they are repeatedly checking for new data to process, but the queue is empty and there are no active threads doing work. This means that after reading the 11,401 rows, no new records are being added to the queue, and the system is essentially waiting for more data or for a termination signal. This behavior is consistent with what is described in the [Airbyte GitHub issue](https://github.com/airbytehq/airbyte/issues/58082): when a sync hangs, the logs show that the workers are running but not processing any new records, and the queue remains empty. The process does not exit or fail, but also does not progress, leading to an idle state until you manually cancel the sync. The logs do not indicate any errors or resource issues; they simply show that the system is in a waiting loop, likely because the pagination did not reach a stop condition or the source stopped providing new data, but the sync process did not recognize that it should finish. This is a known issue pattern where the sync hangs, saying "nothing to do," but does not terminate on its own [GitHub issue](https://github.com/airbytehq/airbyte/issues/58082). Summary: - The repeated log messages mean Airbyte is idle, waiting for more data or a stop signal. - This typically happens when the sync does not detect the end of data and does not terminate automatically. - You are correct that the process does not progress and must be manually stopped in this state. If you need to resolve this, review your pagination configuration and ensure that the sync can detect when all data has been read, so it can terminate properly.