https://linen.dev logo
Join Slack
Powered by
Hello! I'm deploying airbyte in a Kubernetes clust...
# replication-ask-ai
l
Hello! I'm deploying airbyte in a Kubernetes cluster and I'm confunsed on how to properly set the resource requirements values. My problem is the following: When a destination/source pod is created it runs the socat images. How can I change the requests for the remote-stdin/remoted-stdout containers. At the moment they request 500 cpu k8s units and I'm unable to run multiple syncs because of this request. How can I change it? I see at the logs that kubernetes reads thes requirements from some place, but i can't find it.
Copy code
2023-09-11 14:15:41 INFO i.a.w.p.KubeProcessFactory(create):118 - Attempting to start pod = source-mailchimp-read-527-0-nsnjq for airbyte/source-mailchimp:0.4.1 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@557ba5bb[cpuRequest=0.1,cpuLimit=0.2,memoryRequest=,memoryLimit=,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@77a40b4b[cpuRequest=0.05,cpuLimit=0.2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@1fc4a452[cpuRequest=0.01,cpuLimit=0.5,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=null, stdOut=io.airbyte.config.ResourceRequirements@3839bf01[cpuRequest=0.5,cpuLimit=1,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts io.airbyte.config.AllowedHosts@2733e68f[hosts=[*.<http://api.mailchimp.com|api.mailchimp.com>, *.<http://datadoghq.com|datadoghq.com>, *.<http://datadoghq.eu|datadoghq.eu>, *.<http://sentry.io|sentry.io>],additionalProperties={}]
Where can I change this io.airbyte.config.ResourceRequirements for the stdOut/stdIn container? Our team really wants to use Airbyte as the main tool, but this has been a bottleneck for us Thanks a lot!
k
New message text here
l
@kapa.ai what is 
io.airbyte.config.ResourceRequirements
in the logs
k
io.airbyte.config.ResourceRequirements
in the logs refers to the resource requirements for running Airbyte jobs, specifically for source and destination containers. These requirements are logged as part of the job logs when the containers are created. The resource requirements include CPU and memory requests and limits. For example, in a Docker log, it might look like this:
Copy code
Creating docker container = destination-e2e-test-write-39-0-vnqtl with resources io.airbyte.config.ResourceRequirements@1d86d7c9[cpuRequest=<null>,cpuLimit=<null>,memoryRequest=200Mi,memoryLimit=200Mi]
And in a Kubernetes log, it might look like this:
Copy code
2022-08-12 01:22:20 INFO i.a.w.p.KubeProcessFactory(create):100 - Attempting to start pod = source-intercom-check-480195-0-abvnr for airbyte/source-intercom:0.1.24 with resources io.airbyte.config.ResourceRequirements@11cc9fb9[cpuRequest=2,cpuLimit=2,memoryRequest=200Mi,memoryLimit=200Mi]
These logs help you to debug issues like a job running out-of-memory. You can navigate to the Job in the UI and look for the log to confirm the right configuration is being detected. For more information, you can refer to the [Airbyte documentation on configuring connector resources](https://docs.airbyte.com/operator-guides/configuring-connector-resources).
l
How can I change the resource requirements of the 
remote-stdin
container @kapa.ai
k
A message from kapa.ai
Based on the information provided in the extracts, there isn't a direct way to change the resource requirements of the 
remote-stdin
container specifically. However, you can configure resource requirements for Airbyte jobs at three different levels: instance-wide, connector-specific, and connection-specific. The narrower the scope of the requirement, the higher its precedence. For instance-wide requirements, you can set environment variables like 
JOB_MAIN_CONTAINER_CPU_REQUEST
, 
JOB_MAIN_CONTAINER_CPU_LIMIT
, 
JOB_MAIN_CONTAINER_MEMORY_REQUEST
, and 
JOB_MAIN_CONTAINER_MEMORY_LIMIT
. For connector-specific and connection-specific requirements, you can run SQL commands to update the 
resource_requirements
in the 
actor_definition
and 
connection
tables respectively. If you're looking to change the Docker images used by Airbyte, such as the 
remote-stdin
container, you can override environment variables in your worker deployment as mentioned in a [previous Slack conversation](https://airbytehq.slack.com/archives/C021JANJ6TY/p1667390739186989). For more detailed instructions, please refer to the [Airbyte documentation on configuring connector resources](https://docs.airbyte.com/operator-guides/configuring-connector-resources).
l
which containers of the destination/source pod the 
JOB_MAIN_CONTAINER_CPU_REQUEST
affects? @kapa.ai
k
A message from kapa.ai
The 
JOB_MAIN_CONTAINER_CPU_REQUEST
environment variable affects the main job container of the source and destination pods in Airbyte. This variable is used to define the minimum CPU usage for the job container. The units follow either Docker or Kubernetes, depending on the deployment. By default, it is not set, meaning there is no minimum CPU usage defined. This configuration is important for managing resource allocation and ensuring that your Airbyte jobs have the necessary resources to run efficiently. For more information, you can refer to the [Airbyte documentation on configuring Airbyte](https://docs.airbyte.com/operator-guides/configuring-airbyte) and [configuring connector resources](https://docs.airbyte.com/operator-guides/configuring-connector-resources).
2 Views
Open in Slack
PreviousNext
Powered by