Hi team, Getting this request quota must be specified for sync pods error. I am specifying worker.resources and jobs.resources in ~/charts/airbyte/values.yaml. Are there any additional mappings I should look out for/ any thoughts?

pods "...sync-...." is forbidden: failed quota: request-quota: must specify requests.cpu,requests.memory.', type='io.fabric8.kubernetes.client.KubernetesClientException', nonRetryable=...

It’s possible to share the complete logs error @Renzo B?

@Marcos Marx (Airbyte) at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:151) ~[temporal-sdk-1.6.0.jar:?] at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:73) ~[temporal-sdk-1.6.0.jar:?] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.base/java.lang.Thread.run(Thread.java:833) ~[?:?] Caused by: io.temporal.failure.ApplicationFailure: message='Error while getting spec from image my-registry/airbyte/destination-s3:1.15', type='io.airbyte.workers.WorkerException', nonRetryable=false at io.airbyte.workers.DefaultGetSpecWorker.run(DefaultGetSpecWorker.java:78) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?] at io.airbyte.workers.DefaultGetSpecWorker.run(DefaultGetSpecWorker.java:23) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?] at java.base/java.lang.Thread.run(Thread.java:833) ~[?:?] Caused by: io.temporal.failure.ApplicationFailure: message='Failure executing: POST at: https://172.20.0.1/api/v1/namespaces/mynamespace/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "sync-6609a393" is forbidden: failed quota: request-quota: must specify requests.cpu,requests.memory.', type='io.airbyte.workers.WorkerException', nonRetryable=false at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:148) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?] at io.airbyte.workers.process.AirbyteIntegrationLauncher.spec(AirbyteIntegrationLauncher.java:44) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?] at io.airbyte.workers.DefaultGetSpecWorker.run(DefaultGetSpecWorker.java:48) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?] at io.airbyte.workers.DefaultGetSpecWorker.run(DefaultGetSpecWorker.java:23) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?] at java.base/java.lang.Thread.run(Thread.java:833) ~[?:?] Caused by: io.temporal.failure.ApplicationFailure: message='Failure executing: POST at: https://172.20.0.1/api/v1/namespaces/mynamespace/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "sync-6609a393" is forbidden: failed quota: request-quota: must specify requests.cpu,requests.memory.', type='io.fabric8.kubernetes.client.KubernetesClientException', nonRetryable=false at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:583) ~[kubernetes-client-5.3.1.jar:?] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:520) ~[kubernetes-client-5.3.1.jar:?] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:487) ~[kubernetes-client-5.3.1.jar:?] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:448) ~[kubernetes-client-5.3.1.jar:?] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:263) ~[kubernetes-client-5.3.1.jar:?] at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:870) ~[kubernetes-client-5.3.1.jar:?] at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:365) ~[kubernetes-client-5.3.1.jar:?] at io.fabric8.kubernetes.client.utils.CreateOrReplaceHelper.createOrReplace(CreateOrReplaceHelper.java:53) ~[kubernetes-client-5.3.1.jar:?] at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:411) ~[kubernetes-client-5.3.1.jar:?] at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:86) ~[kubernetes-client-5.3.1.jar:?] at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:394) ~[kubernetes-client-5.3.1.jar:?] at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:86) ~[kubernetes-client-5.3.1.jar:?] at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:481) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?] at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:144) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?] at io.airbyte.workers.process.AirbyteIntegrationLauncher.spec(AirbyteIntegrationLauncher.java:44) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?] at io.airbyte.workers.DefaultGetSpecWorker.run(DefaultGetSpecWorker.java:48) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?] at io.airbyte.workers.DefaultGetSpecWorker.run(DefaultGetSpecWorker.java:23) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.30-alpha.jar:?] at java.base/java.lang.Thread.run(Thread.java:833) ~[?:?]

(Renzo’s coworker here) I think I got it! The exception is thrown in airbyte-workers/src/main/java/io/airbyte/workers/process/KubePodProcess.java where the pod is built. That pod contains several containers: the main container, which uses the resources specified by

job.resources

, and several sidecar containers, whose resource requirements are hardcoded in the same file. See local variable

sidecarResources

defined on line 417, which uses

DEFAULT_SIDECAR_RESOURCES

, which is defined on line 104, which only specifies memory limits and not CPU limits. The sidecars need to have their CPU requests and limits defined. Note that the init container doesn’t seem to have its resources specified either (see line 164 of the same file) Our cluster’s requirement to set these parameters is a standard pattern for managing a multi-tenant Kubernetes cluster, so this shouldn’t be that unusual of a use case.

Hey can you raise an issue around this so that team can comment and track this

Will do

Hi, do you know when this issue will be looked at by the team?

Not yet @Ryan Cheatham 😞