https://linen.dev logo
Join Slack
Powered by
<@U04SE163WC8> I see the following log and the job...
# ask-ai
j
@kapa.ai I see the following log and the job stuck for the postgresql destination?
Copy code
No stats have been persisted for job 8 stream StreamNameAndNamespace
k
Based on the information available in the knowledge sources, it seems that there are a few known issues related to jobs getting stuck when using the PostgreSQL destination in Airbyte. Here are some steps you can take to troubleshoot and potentially resolve the issue: 1. Check for Temporary Table Issues: - One of the maintainers suggested a workaround for a similar issue by deleting the temporal database inside the Airbyte PostgreSQL server. Here are the steps: 1. Ensure all correctly running jobs are completed. 2. Stop all new jobs from being run. 3. Delete the temporal table. 4. Redeploy Airbyte containers (a restart of the temporal service might be sufficient). This workaround forces the temporal service to recreate the temporal database, which resolved the issue for the user. Note that this is a hacky solution but might be effective. Source: [GitHub Discussion #28859](https://github.com/airbytehq/airbyte/discussions/28859) 2. Review Sync Configuration: - Ensure that the sync configuration is correct and that the temporary tables are being created as expected. The logs should show the creation of temporary tables with names like 
_airbyte_tmp_<stream_name>
. Example log entries:
Copy code
plaintext    Write config: WriteConfig{streamName=ad_creative_analytics, namespace=null, outputSchemaName=public, tmpTableName=_airbyte_tmp_rji_ad_creative_analytics, outputTableName=_airbyte_raw_ad_creative_analytics, syncMode=append}
Source: [GitHub Issue #38534](https://github.com/airbytehq/airbyte/issues/38534) 3. Check for Casing Issues: - There might be issues related to the casing of table names. Ensure that the table names in the source and destination match in terms of casing. A mismatch can cause the sync job to fail. Example error:
Copy code
plaintext    java.lang.RuntimeException: org.postgresql.util.PSQLException: ERROR: relation "public.public_raw__stream_Job" does not exist
Source: [GitHub Discussion #34423](https://github.com/airbytehq/airbyte/discussions/34423) If these steps do not resolve the issue, you might need to provide more specific logs or details about your configuration for further assistance. Additionally, you can check the [Airbyte documentation](https://docs.airbyte.com/) for more detailed troubleshooting steps. If you need further help, you can also open a new issue or discussion on the [Airbyte GitHub repository](https://github.com/airbytehq/airbyte) to get support from the community and maintainers.
j
@kapa.ai looks like using incremental and append_dedup will have problems
k
A message from kapa.ai
Using the Incremental Append + Deduped sync mode in Airbyte can indeed have some limitations and potential issues. Here are some key points to consider: ### Known Limitations 1. Cursor Column Updates: - If modifications to the underlying records are made without properly updating the cursor field, the updated records won't be picked up by the Incremental sync as expected. This is because the source connectors extract delta rows using a SQL query that relies on the cursor field.
Copy code
sql    select * from table where cursor_field > 'last_sync_max_cursor_field_value'
Source: [Airbyte Documentation - Known Limitations](https://docs.airbyte.com/using-airbyte/core-concepts/sync-modes/incremental-append-deduped#known-limitations) 2. Inclusive Cursors: - Airbyte provides an at-least-once delivery guarantee, meaning it is acceptable for sources to re-send some data when run incrementally. This is particularly relevant when a source's cursor is not very granular. For example, if a cursor field has the granularity of a day, running the source twice in the same day may result in duplicate records. - By convention, sources should prefer resending data if the cursor field is ambiguous. Source: [Airbyte Documentation - Inclusive Cursors](https://docs.airbyte.com/using-airbyte/core-concepts/sync-modes/incremental-append-deduped#inclusive-cursors) ### Practical Issues 1. Destination Compatibility: - The Incremental Append + Deduped mode is only available for a few destinations. If you are using a destination that does not support this mode, you will not be able to use it. - Example: A user reported that this mode was not available for MySQL as a destination. Source: [GitHub Issue #33242](https://github.com/airbytehq/airbyte/issues/33242) 2. Performance Concerns: - Scanning the full table to figure out duplicates can be time-consuming and may lead to performance issues, especially if the replication slot accumulates a massive amount of data. - Example: A user expressed concerns about the replication slot accumulating data to a dangerous level due to the time taken for each batch. Source: [Airbyte Forum Post](https://discuss.airbyte.io/t/1702) ### Recommendations - Ensure Proper Cursor Updates: Make sure that any modifications to records also update the cursor field to ensure that the incremental sync picks up the changes. - Check Destination Compatibility: Verify that your destination supports the Incremental Append + Deduped mode. - Monitor Performance: Keep an eye on the performance and disk space usage, especially if you are dealing with large datasets. If you encounter specific issues or need further assistance, consider opening a new issue or discussion on the [Airbyte GitHub repository](https://github.com/airbytehq/airbyte) to get support from the community and maintainers.
Open in Slack
PreviousNext
Powered by