<@U04S8PLJWMT> I have set up on Airbyte Cloud, thr...
# ask-ai
m
@kapa.ai I have set up on Airbyte Cloud, through an custom connector, an API in POST call that returns data from source. The PAI call returns a mximum of 200 rows. What is happening is that I keep receiving multiple pages of data but exactly the same data. Basically, the 200 rows are repeated on each page, which poses a problem because I am not getting the rest of the data. this what I have configured in the paginator section see screenshot. How can I solve this problem ?
k
Sorry @MTA, it looks like your question contains files, which I cannot process. Please reformulate your question without including any files.
m
@kapa.ai I have set up on Airbyte Cloud, through an custom connector, an API in POST call that returns data from source. The PAI call returns a mximum of 200 rows. What is happening is that I keep receiving multiple pages of data but exactly the same data. Basically, the 200 rows are repeated on each page, which poses a problem. I have set up the paginator mode, with page size of 200 rows, tested with less value but it doesn't see to have any effect on the number of rows, it's always a group of 200 rows
k
It sounds like your connector is successfully paginating, but the API is returning the same set of 200 records for each page, rather than different records for each page. This is a common issue and can be caused by a few misconfigurations or API behaviors: ### Possible Causes & Troubleshooting Steps 1. Pagination Parameter Not Injected Correctly - Ensure that the pagination parameters (such as
page
,
page_number
, or
offset
) are being correctly injected into your POST request. If the API expects the page number or offset in the request body (not as a query parameter), make sure your paginator is configured to inject these values into the correct location (e.g.,
body_json
or
body_data
). - Double-check the parameter names and their locations as required by the API documentation. If the API expects
page
in the body, but your connector injects it as a query parameter, the API will ignore it and always return the first page of results, causing the repeated data issue you’re seeing[Pagination Troubleshooting](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#troubleshooting). 2. Verify First Request Behavior - Some APIs require the pagination parameter to be present even on the first request, while others do not. Make sure your configuration matches the API’s expectations. You may need to enable or disable "Inject Page Number on First Request" depending on the API[Pagination Troubleshooting](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#troubleshooting). 3. Test with Different Page Sizes - You mentioned that changing the page size has no effect. This could indicate that the API is ignoring the page size parameter, or that it is not being injected correctly. Try testing with very small page sizes (e.g., 2–5) and verify if the response changes[Testing Pagination](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#testing-pagination). 4. Check API Documentation - Confirm the correct pagination method and parameter names. Some APIs use
page
, others use
offset
, and some may use a cursor. Using the wrong method or parameter name will result in the same page being returned repeatedly[Page Increment](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#page-increment). 5. Validate API Response - Manually test the API outside of Airbyte (e.g., with Postman or curl) using different pagination parameters to ensure the API itself returns different data for each page. ### Next Steps - Review your paginator configuration in Airbyte and ensure the pagination parameters are injected into the correct part of the POST request (body, not query, if required). - Check the parameter names and their values in the outgoing requests. - Test the API manually to confirm its pagination behavior. - Consult the API documentation for the correct pagination approach. If after these checks the issue persists, it may be necessary to share your connector configuration and a sample API request/response for further troubleshooting. For more details, see the troubleshooting section in the Airbyte docs: [Pagination Troubleshooting](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#troubleshooting).
m
@kapa.ai here is the last part of the request "...api/site/customers/search?sort_type=id&sort_order=asc&page_size=2", response is 400 rows, 200 per page and same 200 rows in bith pages.
k
A message from kapa.ai
Based on your description and the knowledge sources, the issue is likely due to how the pagination parameters are being injected into your API requests. If you are always receiving the same 200 rows per page, it suggests that the API is not receiving (or not recognizing) the correct pagination parameter for the page number or offset, and is therefore returning the first page repeatedly. Here are the key points to check and troubleshoot: 1. Parameter Injection Location Ensure that the pagination parameters (
page
,
page_number
,
offset
, etc.) are being injected into the correct part of the request. Since your API uses a POST call, the parameters may need to be in the request body (as JSON or form data), not just in the query string. The Airbyte Connector Builder allows you to specify where to inject these parameters: as request parameters, headers, or in the body. If the API expects the page number in the body but you are only sending it in the query string, the API will ignore it and return the same data each time[Pagination Troubleshooting](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#troubleshooting). 2. Correct Parameter Names Double-check the API documentation to confirm the exact parameter names and their expected location. For example, if the API expects
page
or
page_number
in the body, but you are using
page_size
or only passing it in the URL, it may not work as intended[Page Increment](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#page-increment). 3. Manual API Testing Test the API manually (e.g., with Postman or curl) by sending requests with different page numbers or offsets. Confirm that the API returns different data for each page when the correct parameters are provided. 4. Connector Builder Configuration In the Airbyte Connector Builder, make sure you have: - Set the correct pagination mode (likely "Page Increment" or "Offset Increment"). - Configured the page size and starting page correctly. - Injected the page number/offset into the correct location (body, if required). - Used the correct field names as per the API documentation. 5. Stop Condition If the API always returns 200 rows, ensure your stop condition is set so that pagination stops when fewer than the page size is returned, or when there are no more records[Pagination Troubleshooting](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#troubleshooting). Summary: The most likely cause is that the page number or offset is not being injected into the correct part of the request, or the parameter name is incorrect. Review your Connector Builder configuration and the API documentation to ensure alignment. If the issue persists after these checks, the knowledge sources recommend reviewing the API's requirements and testing with sample data to debug further[Pagination Troubleshooting](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#troubleshooting). If you need more specific help, please provide the exact configuration of your paginator in Airbyte and a sample of your request body.
m
@kapa.ai I have found the correct configuration now, the pagination method and the parameters to inject to limit and change pages. I have read the log and saw that in full refresh mode all the data is first written in a temporary table. The process is idle after writing the 70k rows in the temporary table. My expectation would be to copy the data to the target table but it has been 30mins and still no rows in target and it seems that the sync process still expects data from source, what can i do ?
k
A message from kapa.ai
Based on the knowledge sources, your observation matches typical Airbyte behavior: during a full refresh sync, data is first written to a temporary (tmp) table in the destination. After all data is loaded into the tmp table, Airbyte should finalize the sync by moving (or copying) the data from the tmp table to the final target table. If the process is idle after writing all rows to the tmp table and no data appears in the target table after a significant wait (e.g., 30 minutes), this can indicate an issue during the finalization step. This situation has been reported by other users as well, where the sync gets stuck at the "executing finalization of tables" phase, especially with large data volumes. In such cases, the sync may appear to hang, and cancelling the job does not always resolve the issue immediately. The root cause can be resource constraints, a bug in the connector, or an issue with the destination database itself[Restart an existing failed sync from tmp tables](https://discuss.airbyte.io/t/3049). Recommended steps: 1. Check Resource Usage: Ensure your destination database and Airbyte instance have enough resources (CPU, memory, disk space) to handle the finalization step, as this can be resource-intensive, especially with large tables. 2. Review Logs: Download and review the sync logs for any errors or warnings during the finalization phase. This can provide clues if there is a database lock, timeout, or other error. 3. Wait or Retry: Sometimes, with large datasets, the finalization can take longer than expected. If resource usage is normal and there are no errors, you may want to wait longer. If it remains stuck, you may need to cancel and retry the sync. 4. Manual Intervention: If the data is present in the tmp table and you need the data urgently, you could manually move the data from the tmp table to the target table in your database. However, this is not the standard Airbyte workflow and should be done with caution. 5. Try a Smaller Sync: To isolate the issue, try syncing a smaller dataset to see if the finalization step completes successfully. This can help determine if the problem is related to data volume. 6. Check for Known Issues: There are known issues where temp tables are not cleaned up after a failed sync, and the process can get stuck at the finalization step. These are being tracked and improved by the Airbyte team[What happens if a sync fails?](https://discuss.airbyte.io/t/77). If the problem persists after these steps, consider sharing your logs with the Airbyte community or filing a GitHub issue for further assistance. In summary: It is not normal for the process to remain idle for a long time after writing to the tmp table. Check resources, review logs, and consider manual intervention or retrying the sync. This issue has been observed by other users, especially with large data volumes, and may require further troubleshooting or support from the Airbyte team[Restart an existing failed sync from tmp tables](https://discuss.airbyte.io/t/3049), [What happens if a sync fails?](https://discuss.airbyte.io/t/77).
m
@kapa.ai is there any known options we can add in order to know when there is no more data sent by the api ?
k
A message from kapa.ai
Yes, Airbyte provides a way to define a stop condition in your pagination configuration to determine when there is no more data to fetch from the API. This is especially important for cursor-based or custom pagination strategies. The stop condition is a template (using Jinja syntax) that evaluates to
true
when pagination should stop. You can use variables from the API response, headers, or other context to define this condition. Common patterns include: - When a boolean flag in the response indicates no more results:
Copy code
{{ response.more_results is false }}
- When a cursor or token is missing or null:
Copy code
{{ response.pagination.next_cursor is none }}
- When the data array is empty:
Copy code
{{ not response.data }}
- When a specific link is missing in the headers (e.g., for GitHub's Link header):
Copy code
{{ 'next' not in headers['link'] }}
- Combining multiple checks:
Copy code
{{ response.data|length == 0 or response.pagination.next_cursor is none }}
You can configure this in the Connector Builder UI or in the YAML for config-based connectors. The stop condition ensures that Airbyte knows when to stop paginating, even if the API does not explicitly indicate the end of data in a standard way. For more details and examples, see the documentation on [Advanced Cursor Pagination Features](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#advanced-cursor-pagination-features) and [YAML Reference for CursorPagination](https://docs.airbyte.com/platform/connector-development/config-based/understanding-the-yaml-file/reference#/definitions/CursorPagination).
m
@kapa.ai will this work with offset pagination mode ?
k
A message from kapa.ai
For offset pagination mode in Airbyte, the stop condition is handled automatically: Airbyte will continue paginating until it receives fewer records than the configured page size (limit). This is the default and expected behavior for offset-based pagination, as described in the documentation: > your connector will automatically increment the
offset
for subsequent requests based on the number of records it receives, and will continue until it receives fewer records than the limit you configured [Offset Increment - Connector Builder UI](https://docs.airbyte.com/platform/1.8/connector-development/connector-builder-ui/pagination#offset-increment) So, you do not need to manually specify a stop condition for offset pagination in most cases. Airbyte will stop fetching new pages when the API returns a page with fewer records than the limit. If your API behaves differently (for example, always returns the same number of records even when there is no more data), you may need to use a different pagination strategy or work with a custom stop condition, but this is not standard for offset pagination as supported by Airbyte's built-in paginator.
m
@kapa.ai Source didn't send any records in the last 1 hour 31 minutes 7 seconds, exceeding the configured 1 hour 30 minutes threshold and the replication restartd for a second attempt, I am trying to solve this
k
A message from kapa.ai
The message you are seeing indicates that Airbyte's heartbeat mechanism detected that the source did not send any records or state messages within the configured threshold (in your case, 1 hour 30 minutes). As a result, Airbyte considers the source unresponsive and restarts the replication attempt. This is a built-in safeguard to prevent jobs from hanging indefinitely if the source stops responding. ### How Heartbeats Work - Airbyte monitors for
RECORD
or
STATE
messages from the source as "heartbeats." - If no heartbeat is received within the configured time (
heartbeat-max-seconds-between-messages
), the sync attempt is failed and restarted. - The default threshold is 3 hours, but it can be configured in the
flags.yaml
file. - The relevant configuration options are: - `heartbeat-max-seconds-between-messages`: Maximum allowed time between messages (default: 3 hours). - `heartbeat.failSync`: If true (default), the sync fails on a missed heartbeat; if false, it does not fail the sync[Technical Details - Heartbeats](https://docs.airbyte.com/platform/1.8/understanding-airbyte/heartbeats#technical-details). ### Troubleshooting Steps 1. Check Source Performance - If your source is slow (e.g., due to large data, slow queries, or API rate limits), it may not emit records frequently enough. Optimize your source or queries if possible. 2. Increase Heartbeat Threshold - If your syncs are expected to take longer between messages, consider increasing the
heartbeat-max-seconds-between-messages
value in your
flags.yaml
file to allow more time between heartbeats. 3. Check for API or Network Issues - If the source is an API, ensure it is not rate-limited or experiencing downtime. 4. Review Connector and Sync Configuration - Make sure your connector is configured to emit records or state messages regularly, especially for large or complex syncs. 5. Logs and Support - Review the Airbyte logs for more details on where the process is stalling. - If the issue persists, and you are using Airbyte Cloud, you can reach out to Airbyte Support for further assistance[Heartbeats - Known Causes](https://docs.airbyte.com/platform/1.8/understanding-airbyte/heartbeats). Summary: The error is due to the heartbeat timeout. You can address it by optimizing your source, increasing the heartbeat threshold, or reviewing your connector configuration. If the problem continues, consult the logs or contact support for deeper investigation.
m
@kapa.ai I am sure there is no more data in the source but the sync doesnt stop. so that this mean that it's the source that is not indicating there is no more data or is it airbyte that is not getting the message correctly ?
k
A message from kapa.ai
If you are certain there is no more data in the source but the sync does not stop, the most likely cause—according to the Airbyte documentation—is that the stop condition for pagination is not being met. This can happen if: - The source API is not providing a clear indicator (such as an empty data array, a missing cursor, or a boolean flag) that there is no more data. - The connector’s pagination configuration is not correctly interpreting the API’s signal that there is no more data. - The stop condition template in your connector is not correctly referencing the field or value that indicates the end of data. Airbyte relies on the stop condition you configure (or the default logic for offset pagination) to know when to stop paginating. If the stop condition is not triggered, Airbyte will keep requesting more pages, even if the source is returning empty or repeated data. This is a common troubleshooting point for infinite or hanging syncs[Pagination Troubleshooting](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#troubleshooting). In summary: It is usually not Airbyte failing to receive a message, but rather the connector’s pagination configuration not correctly detecting the end-of-data signal from the source. Double-check your stop condition and ensure it matches the API’s actual response when there is no more data. If the API does not provide a clear signal, you may need to adjust your pagination logic or consult the API documentation for a reliable indicator[Pagination Troubleshooting](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#troubleshooting).
m
@kapa.ai I have an API that needs the word fetch and a list of fields need to be extracted in the request, example ...api/search?fetch=criteria&sort=asc will return the filed criteria. I was looking to put this in place in the Record Selector, Extractor and Field Path but I don't know where i can specifiy where to add the specific "fetch" expected by the API
k
A message from kapa.ai
To include the
fetch
parameter (and its value, such as
criteria
) in your API request, you should not use the Record Selector, Extractor, or Field Path for this purpose. Instead, you should configure this as a request parameter in your connector configuration. In the Airbyte Connector Builder, you can add query parameters to your requests in the generic parameter section at the bottom of the stream configuration form. Here, you can specify key-value pairs for query parameters, such as:
Copy code
yaml request_parameters:   fetch: criteria   sort: asc
This will ensure that every request sent to your API includes
?fetch=criteria&sort=asc
in the URL. - Record Selector is used to specify the path in the API response where the records are located (e.g.,
data.items
). - Extractor and Field Path are used for extracting data from the response, not for configuring request parameters. For more details on how to configure query parameters in the Connector Builder, see the documentation on [custom parameter injection](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/pagination#custom-parameter-injection). Summary: Add the
fetch
parameter in the request parameters section of your stream configuration, not in the Record Selector, Extractor, or Field Path. This will ensure the API receives the expected query parameter in each request.