This message was deleted.

Based on the information provided, it seems like you might be facing issues with pagination and duplicate data. Here are a few things you could consider: 1. Pagination: Airbyte's Connector Builder offers a Pagination section which implements the most common pagination methods used by APIs. When enabled, the connector will use the pagination configuration you have provided to request consecutive pages of data from the API until there are no more pages to fetch. If your API supports pagination, you should enable this feature and configure it according to your API's documentation. If your API uses cursor pagination, you might need to provide a cursor value to navigate through the API's records. This can help prevent issues like duplicate or skipped records ([source](https://docs.airbyte.com/connector-development/connector-builder-ui/pagination#cursor-pagination)). 2. Duplicate Data: There have been instances where Airbyte users have reported issues with duplicate data when using incremental sync or append mode ([source](https://github.com/airbytehq/airbyte/issues/39566), [source](https://github.com/airbytehq/airbyte/issues/34570)). It's possible that a similar issue might be causing the duplicate rows in your case. You might want to check your sync mode and configuration to ensure they are set up correctly. 3. Rate Limiting: Most APIs enforce rate limits. If your API has a rate limit and you're hitting it, it could potentially cause issues with data extraction. Your connector should gracefully handle those (i.e: without failing the connector process). The most common way to handle rate limits is to implement backoff ([source](https://docs.airbyte.com/connector-development/best-practices#rate-limiting)). If you're still facing issues, it might be helpful to open an issue on the [Airbyte GitHub](https://github.com/airbytehq/airbyte/issues) page or start a discussion on the [Airbyte Discussions](https://github.com/airbytehq/airbyte/discussions) page. The Airbyte team and community can provide more specific help and guidance.

Yes, you can delete the table and its corresponding

airbyte_internal

table if you are not satisfied with the results. However, please note that the mechanism by which a destination connector accomplishes the full refresh will vary from destination to destination. For certified database and data warehouse destinations, Airbyte will be recreating the final table each sync. This allows Airbyte to leave the previous sync's data viewable by writing to a "final-table-tmp" location as the sync is running, and at the end dropping the old "final" table, and renaming the new one into place. However, please be aware that this may not be possible for all destinations, and Airbyte may need to erase the existing data at the start of each full-refresh sync. Also, if you are using normalization, it might happen that, on full refresh overwrite, DBT drops the views created on top of the tables managed by DBT. This is a DBT known issue. Sources: - [Airbyte Documentation](https://docs.airbyte.com/using-airbyte/core-concepts/sync-modes/full-refresh-overwrite-deduped#destination-specific-mechanism-for-full-refresh) - [Airbyte Forum Post](https://discuss.airbyte.io/t/747)

A message from kapa.ai

Airbyte does not expose SQL queries to the source in the logs to avoid exposing sensitive information. However, you can try to set

LOG_LEVEL=DEBUG

to check if you get more information. If this doesn't provide the information you need, you can open an issue on the [Airbyte GitHub repository](https://github.com/airbytehq/airbyte/issues/new?assignees=&labels=type%2Fenhancement%2C+needs-triage&template=feature-request.md&title=) to request this feature. For the issue with the ServiceNow API returning empty fields, you might want to check the API documentation or contact ServiceNow support for more information. It could be due to the API endpoint, parameters, or the data in the ServiceNow instance. Source: [Airbyte Forum Post](https://discuss.airbyte.io/t/726)