<@U04S8PLJWMT> we are running into issues with our...
# ask-ai
j
@kapa.ai we are running into issues with our S3 source connector performing incredibly slow. We are using the self-managed version of Airbyte, version 1.6. Looking at the resources consumed on Kubernetes, it looks like the source connector isn’t using the majority of the resources allotted to it. What areas should we look at to improve the performance of this source connector
k
If your S3 source connector is performing slowly and not utilizing the resources allocated in Kubernetes, here are some areas to investigate and actions you can take, based on the available documentation and community discussions: ### 1. Resource Allocation and Configuration - Resource allocation alone may not guarantee better performance if the connector or job is not designed to utilize more CPU or memory. Ensure that your resource settings are correctly applied and recognized by Airbyte. You can check the job logs to confirm the detected configuration for each sync job. Look for log entries like:
Copy code
[source] image: airbyte/source-sftp:... resources: ResourceRequirements(...)
This will help you verify if the connector is actually receiving the resources you expect it to use. If not, review your configuration steps and precedence (instance-wide, connector definition, connector-specific, connection-specific) to ensure the right settings are in place [Configuring Connector Resources](https://docs.airbyte.com/platform/1.6/operator-guides/configuring-connector-resources#debugging). ### 2. Connector Version and Platform Version - Performance improvements for S3 source connectors have been released in newer Airbyte versions. Specifically, with the introduction of CDK 2.0 and Pydantic V2, serialization overhead has been reduced, resulting in faster syncs. However, these improvements are available in Airbyte version 1.8 and later. Since you are on version 1.6, you may not benefit from these enhancements. Upgrading to at least version 1.8 is recommended to take advantage of these speed improvements [Faster sync speed](https://docs.airbyte.com/integrations/speed-improvements). ### 3. Kubernetes Node and CPU Type - CPU performance is directly correlated with sync speed. Benchmarks show that using higher-performance CPUs (e.g., C4 or N2D on GCP) can significantly improve throughput. If possible, consider running your Airbyte workloads on nodes with better CPUs [Faster sync speed](https://docs.airbyte.com/integrations/speed-improvements). ### 4. Connector-Specific Bottlenecks - Connector logic may be a bottleneck. If the connector is not multi-threaded or is limited by the way it reads from S3, increasing resources may not help. Review the connector’s documentation or GitHub issues for known performance limitations or configuration options. ### 5. Logging and Observability - Excessive logging can impact performance. For large syncs, a high volume of logs can slow down the process. Consider reducing log verbosity if possible [Scaling Airbyte](https://docs.airbyte.com/platform/1.6/operator-guides/scaling-airbyte#what-to-scale). ### 6. General Best Practices - Start with default resource allocations and only increase for specific connectors or connections as needed. Use observability tools and logs to identify if CPU or memory is truly a bottleneck before increasing allocations [Configuring Connector Resources](https://docs.airbyte.com/platform/1.6/operator-guides/configuring-connector-resources). --- Summary: - Upgrade to Airbyte 1.8+ for S3 source speed improvements. - Confirm resource settings are applied and recognized. - Use high-performance CPUs if possible. - Check for connector-specific bottlenecks and reduce log verbosity if needed. If after these steps the performance is still not satisfactory, it may be a limitation of the current S3 source connector implementation in your Airbyte version. In that case, upgrading Airbyte and the connector is likely your best path forward.
j
@kapa.ai it seems like the bottleneck exists in the replication-orchestrator. Is there any way to scale up this?
k
A message from kapa.ai
Yes, you can scale the resources allocated to the orchestrator container in Airbyte. The orchestrator pod receives its resource requirements from the
JOB_MAIN_CONTAINER_*
environment variables. These variables control the CPU and memory requests and limits for job containers, including the orchestrator. You can set these in your Kubernetes deployment or Helm values: -
JOB_MAIN_CONTAINER_CPU_REQUEST
-
JOB_MAIN_CONTAINER_CPU_LIMIT
-
JOB_MAIN_CONTAINER_MEMORY_REQUEST
-
JOB_MAIN_CONTAINER_MEMORY_LIMIT
Adjusting these values will allow the orchestrator to utilize more resources, which may help alleviate bottlenecks if the orchestrator is under-provisioned. However, note that these settings will also apply to the read and write containers, as the same environment variables are used for all job containers. There is currently no documented way to set orchestrator-specific resource limits independently from the read/write pods in Airbyte 1.6. For more granular control, Airbyte supports specifying resource requirements at the connector or connection level via the
resource_requirements
field, but this is primarily for connector pods, not specifically for the orchestrator. For reference, see the official answer in the Airbyte community forum: > "Yes, the orchestrator pods get their resource requirements from the
JOB_MAIN_CONTAINER_*
environment variables. These environment variables are used to configure the resource requirements for the job containers, which include the orchestrator pods." [Orchestrator container gets same resource limit/request values as read/write containers](https://discuss.airbyte.io/t/3775) If you need to scale up orchestrator resources, increase the values of these environment variables accordingly.