MTA
11/21/2025, 1:52 PMkapa.ai
11/21/2025, 1:52 PMoffset and limit parameters. This will allow Airbyte to automatically loop through all pages until all data is retrieved. How to configure: 1. Pagination method: Offset Increment 2. Limit: Set to 200 (or the maximum allowed by your API) 3. Inject limit into outgoing HTTP request: - Inject into: request_parameter - Field name: limit 4. Inject offset into outgoing HTTP request: - Inject into: request_parameter - Field name: offset Airbyte will keep incrementing the offset by the limit (200) and continue fetching until it receives fewer records than the limit, ensuring all ~100k records are retrieved. Here’s an example of how the requests would look: GET https://api.example.com/data?limit=200&offset=0 GET https://api.example.com/data?limit=200&offset=200 GET https://api.example.com/data?limit=200&offset=400 ... Once the initial full load is complete, you can switch to Incremental Sync mode by specifying a suitable cursor field (like a timestamp or incrementing ID). Airbyte will then only fetch new or updated records in subsequent syncs, using the cursor to track progress. The connector maintains state between syncs to resume from where the last sync left off, ensuring efficient incremental updates [Pagination methods](https://docs.airbyte.com/platform/1.8/connector-development/connector-builder-ui/pagination#offset-increment), [Supported sync modes](https://docs.airbyte.com/integrations/sources/bigquery#supported-sync-modes). If your API uses a different pagination mechanism (like page numbers or cursors), Airbyte also supports those methods. If none of the standard methods fit, you can implement a custom pagination strategy. Summary: - Use Offset Increment pagination to fetch all records in batches of 200. - After the initial load, configure Incremental Sync with a suitable cursor field for ongoing updates. Let me know if you need details for a different pagination method or help with incremental sync configuration!MTA
11/21/2025, 3:05 PMkapa.ai
11/21/2025, 3:05 PMkapa.ai
11/21/2025, 3:06 PMMTA
11/21/2025, 3:13 PMkapa.ai
11/21/2025, 3:13 PMkapa.ai
11/21/2025, 3:13 PM{{ response.has_more is false }} or {{ 'next' not in headers['link'] }} This field is available when you choose a pagination method that supports custom stop conditions, such as Cursor Pagination or Custom Pagination Strategy. The stop condition uses variables from the API response or headers to determine when to stop fetching more pages. If you do not set this correctly, Airbyte may continue paginating indefinitely or stop too early. For more details and examples, see the [Pagination documentation](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#advanced-cursor-pagination-features) and the [real-world examples section](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#real-world-examples).MTA
11/21/2025, 3:16 PMkapa.ai
11/21/2025, 3:16 PMkapa.ai
11/21/2025, 3:16 PMMTA
11/21/2025, 3:18 PMkapa.ai
11/21/2025, 3:19 PMkapa.ai
11/21/2025, 3:19 PMMTA
11/21/2025, 3:24 PMkapa.ai
11/21/2025, 3:24 PMMTA
11/21/2025, 3:25 PMkapa.ai
11/21/2025, 3:25 PMMTA
11/21/2025, 3:26 PMkapa.ai
11/21/2025, 3:26 PMMTA
11/21/2025, 3:29 PMINC_Contacts, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-11-21 152757 info INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):129 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-11-21 152857 info INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), %% used: 0.03367428551701215 | Queue INC_Contacts, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-11-21 152857 info INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):129 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-11-21 152957 info INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), %% used: 0.03367428551701215 | Queue INC_Contacts, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-11-21 152957 info INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):129 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-11-21 153057 info INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), %% used: 0.03367428551701215 | Queue INC_Contacts, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-11-21 153057 info INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):129 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-11-21 153157 info INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), %% used: 0.03367428551701215 | Queue INC_Contacts, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-11-21 153157 info INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):129 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-11-21 153221 info INFO pool-5-thread-1 i.a.c.i.d.a.DetectStreamToFlush(getNextStreamToFlush):119 flushing: trigger info: null - INC_Contacts, time trigger: true , size trigger: false current threshold b: 25 MB, queue size b: 0 bytes, penalty b: 0 bytes, after penalty b: 0 bytes
2025-11-21 153221 info INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):137 Flush Worker (ba23b) -- Worker picked up work.
2025-11-21 153221 info INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):143 Flush Worker (ba23b) -- Attempting to read from queue namespace: null, stream: INC_Contacts.
2025-11-21 153221 info INFO pool-4-thread-2 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..
2025-11-21 153221 info INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):161 Flush Worker (ba23b) -- Batch contains: 0 records, 0 bytes bytes.
2025-11-21 153221 info INFO pool-4-thread-2 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):153 Flushing states
2025-11-21 153221 info INFO pool-4-thread-2 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):207 Flushing states completeMTA
11/21/2025, 3:30 PMINC_Contacts, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-11-21 152757 info INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):129 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-11-21 152857 info INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), %% used: 0.03367428551701215 | Queue INC_Contacts, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-11-21 152857 info INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):129 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-11-21 152957 info INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), %% used: 0.03367428551701215 | Queue INC_Contacts, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-11-21 152957 info INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):129 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-11-21 153057 info INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), %% used: 0.03367428551701215 | Queue INC_Contacts, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-11-21 153057 info INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):129 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-11-21 153157 info INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), %% used: 0.03367428551701215 | Queue INC_Contacts, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2025-11-21 153157 info INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):129 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2025-11-21 153221 info INFO pool-5-thread-1 i.a.c.i.d.a.DetectStreamToFlush(getNextStreamToFlush):119 flushing: trigger info: null - INC_Contacts, time trigger: true , size trigger: false current threshold b: 25 MB, queue size b: 0 bytes, penalty b: 0 bytes, after penalty b: 0 bytes
2025-11-21 153221 info INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):137 Flush Worker (ba23b) -- Worker picked up work.
2025-11-21 153221 info INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):143 Flush Worker (ba23b) -- Attempting to read from queue namespace: null, stream: INC_Contacts.
2025-11-21 153221 info INFO pool-4-thread-2 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..
2025-11-21 153221 info INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):161 Flush Worker (ba23b) -- Batch contains: 0 records, 0 bytes bytes.
2025-11-21 153221 info INFO pool-4-thread-2 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):153 Flushing states
2025-11-21 153221 info INFO pool-4-thread-2 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):207 Flushing states completekapa.ai
11/21/2025, 3:30 PMkapa.ai
11/21/2025, 3:31 PMFlush Worker (...) -- Worker finished flushing. Current queue size: 0 [ASYNC QUEUE INFO] ... num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes ... [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0 indicate that Airbyte's destination flush workers and buffer manager are idle—they are repeatedly checking for new data to process, but the queue is empty and there are no active threads doing work. This means that after reading the 11,401 rows, no new records are being added to the queue, and the system is essentially waiting for more data or for a termination signal. This behavior is consistent with what is described in the [Airbyte GitHub issue](https://github.com/airbytehq/airbyte/issues/58082): when a sync hangs, the logs show that the workers are running but not processing any new records, and the queue remains empty. The process does not exit or fail, but also does not progress, leading to an idle state until you manually cancel the sync. The logs do not indicate any errors or resource issues; they simply show that the system is in a waiting loop, likely because the pagination did not reach a stop condition or the source stopped providing new data, but the sync process did not recognize that it should finish. This is a known issue pattern where the sync hangs, saying "nothing to do," but does not terminate on its own [GitHub issue](https://github.com/airbytehq/airbyte/issues/58082). Summary: - The repeated log messages mean Airbyte is idle, waiting for more data or a stop signal. - This typically happens when the sync does not detect the end of data and does not terminate automatically. - You are correct that the process does not progress and must be manually stopped in this state. If you need to resolve this, review your pagination configuration and ensure that the sync can detect when all data has been read, so it can terminate properly.