Airbyte uses S3 (or other object storage like GCS or Azure) to store two main types of data:
state and
logs. -
State storage is used for checkpointing and tracking the progress of syncs, which is essential for reliable and resumable data integration. -
Log storage is used to persist logs generated by Airbyte jobs, making them accessible even after pods are restarted or if you want to view logs from previous runs. This is especially important in distributed or cloud-native deployments where ephemeral containers may lose local logs. Regarding your question about logs: The logs stored in S3 are not fundamentally different in content from what is output to STDOUT/STDERR by Airbyte components. Airbyte emits logs to STDOUT/STDERR, and these are also collected and persisted to object storage (like S3) for durability and centralized access. This means you can use log collectors to capture STDOUT/STDERR for your own observability tools, but Airbyte itself requires object storage as the "source of truth" for logs to ensure they are available for the UI and for troubleshooting, even after restarts or failures. As confirmed in the Airbyte forum, "The logs are simultaneously output on
stdout
, so you can install other logging collectors on the nodes and pipe them to other logging solutions." The S3 storage is used to ensure logs are durable and accessible across the platform, not just transiently available in container output streams. This is important for features like viewing logs in the Airbyte UI and for post-mortem analysis after jobs have completed or failedĀ [Forum: K8S Production reliable deployement](
https://discuss.airbyte.io/t/346). In summary: - S3 storage is used for durable, centralized storage of state and logs. - The logs in S3 are the same as those output to STDOUT/STDERR, but persisted for long-term access and UI integration. - You can still collect logs from STDOUT/STDERR for your own logging solutions, but Airbyte requires object storage for its own log management and reliability needs.