@kapa.ai How can I safely (without breaking airbyte) remove old jobs from the airbyte database? Our database instance is getting slower when working with the jobs table

The knowledge sources don't contain detailed, step-by-step instructions specifically for safely pruning old job records from the Airbyte jobs table without breaking Airbyte. However, there is one relevant piece of information: Temporal History Retention Airbyte has a built-in mechanism to automatically manage job/history retention. You can configure how long job history is kept by setting the environment variable:

TEMPORAL_HISTORY_RETENTION_IN_DAYS=<number_of_days>

This was introduced in Airbyte v0.45.5 and allows you to automatically purge old job history after a specified number of days. [storage issue discussion] Important Cautions If you are considering direct database manipulation to delete job records, the knowledge sources strongly advise: • Always back up your database before making any changes. • Be aware that deleting records directly can cause data inconsistencies if dependencies between tables are not handled correctly. • Make sure to delete related records across all dependent tables, not just the main jobs table. [forum post] Recommendation The safest approach is to use the

TEMPORAL_HISTORY_RETENTION_IN_DAYS

environment variable rather than manually deleting rows from the database. For more specific guidance on the exact tables and safe deletion procedures, I'd recommend reaching out to the Airbyte community forum or consulting the official Airbyte documentation, as the knowledge sources available to me don't provide a detailed safe SQL procedure for jobs table cleanup. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.