Jared Parco
11/24/2025, 5:30 PMkapa.ai
11/24/2025, 5:31 PM[source] image: airbyte/source-sftp:... resources: ResourceRequirements(...) This will help you verify if the connector is actually receiving the resources you expect it to use. If not, review your configuration steps and precedence (instance-wide, connector definition, connector-specific, connection-specific) to ensure the right settings are in place [Configuring Connector Resources](https://docs.airbyte.com/platform/1.6/operator-guides/configuring-connector-resources#debugging). ### 2. Connector Version and Platform Version - Performance improvements for S3 source connectors have been released in newer Airbyte versions. Specifically, with the introduction of CDK 2.0 and Pydantic V2, serialization overhead has been reduced, resulting in faster syncs. However, these improvements are available in Airbyte version 1.8 and later. Since you are on version 1.6, you may not benefit from these enhancements. Upgrading to at least version 1.8 is recommended to take advantage of these speed improvements [Faster sync speed](https://docs.airbyte.com/integrations/speed-improvements). ### 3. Kubernetes Node and CPU Type - CPU performance is directly correlated with sync speed. Benchmarks show that using higher-performance CPUs (e.g., C4 or N2D on GCP) can significantly improve throughput. If possible, consider running your Airbyte workloads on nodes with better CPUs [Faster sync speed](https://docs.airbyte.com/integrations/speed-improvements). ### 4. Connector-Specific Bottlenecks - Connector logic may be a bottleneck. If the connector is not multi-threaded or is limited by the way it reads from S3, increasing resources may not help. Review the connector’s documentation or GitHub issues for known performance limitations or configuration options. ### 5. Logging and Observability - Excessive logging can impact performance. For large syncs, a high volume of logs can slow down the process. Consider reducing log verbosity if possible [Scaling Airbyte](https://docs.airbyte.com/platform/1.6/operator-guides/scaling-airbyte#what-to-scale). ### 6. General Best Practices - Start with default resource allocations and only increase for specific connectors or connections as needed. Use observability tools and logs to identify if CPU or memory is truly a bottleneck before increasing allocations [Configuring Connector Resources](https://docs.airbyte.com/platform/1.6/operator-guides/configuring-connector-resources). --- Summary: - Upgrade to Airbyte 1.8+ for S3 source speed improvements. - Confirm resource settings are applied and recognized. - Use high-performance CPUs if possible. - Check for connector-specific bottlenecks and reduce log verbosity if needed. If after these steps the performance is still not satisfactory, it may be a limitation of the current S3 source connector implementation in your Airbyte version. In that case, upgrading Airbyte and the connector is likely your best path forward.Jared Parco
11/24/2025, 6:04 PMkapa.ai
11/24/2025, 6:04 PMkapa.ai
11/24/2025, 6:04 PMJOB_MAIN_CONTAINER_* environment variables. These variables control the CPU and memory requests and limits for job containers, including the orchestrator. You can set these in your Kubernetes deployment or Helm values: - JOB_MAIN_CONTAINER_CPU_REQUEST - JOB_MAIN_CONTAINER_CPU_LIMIT - JOB_MAIN_CONTAINER_MEMORY_REQUEST - JOB_MAIN_CONTAINER_MEMORY_LIMIT Adjusting these values will allow the orchestrator to utilize more resources, which may help alleviate bottlenecks if the orchestrator is under-provisioned. However, note that these settings will also apply to the read and write containers, as the same environment variables are used for all job containers. There is currently no documented way to set orchestrator-specific resource limits independently from the read/write pods in Airbyte 1.6. For more granular control, Airbyte supports specifying resource requirements at the connector or connection level via the resource_requirements field, but this is primarily for connector pods, not specifically for the orchestrator. For reference, see the official answer in the Airbyte community forum: > "Yes, the orchestrator pods get their resource requirements from the JOB_MAIN_CONTAINER_* environment variables. These environment variables are used to configure the resource requirements for the job containers, which include the orchestrator pods." [Orchestrator container gets same resource limit/request values as read/write containers](https://discuss.airbyte.io/t/3775) If you need to scale up orchestrator resources, increase the values of these environment variables accordingly.