https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> I have an kubernetes helm chart dep...
# replication-ask-ai
g
@kapa.ai I have an kubernetes helm chart deployed 2.0.19 Airbyte instance. I am seeing a lot of errors like this one in the airbyte-server logs.
Copy code
2026-02-26 21:33:47,529 [io-executor-thread-7]  ERROR   i.a.s.a.ApiHelper(execute):47 - Unexpected Exception
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Operation not permitted (SDK Attempt Count: 4)
Any thoughts?
k
The knowledge sources don’t contain a direct explanation for 
SdkClientException: Unable to execute HTTP request: Operation not permitted
in 2.0.19, so I can only infer based on related issues. There are a few known AWS-related problems around Helm 2.0.18–2.0.19 and 2.0.1 server: 1. Workload API / sidecar auth issues In 2.0.1 OSS, connector-sidecar containers sometimes don’t receive the 
WORKLOAD_API_AUTH_HEADER
and 
WORKLOAD_API_BEARER_TOKEN
env vars, which then causes failures when they try to talk to the workload API server. [connector-sidecar auth] A related 500-error issue suggests checking that: ◦ 
API_AUTHORIZATION_ENABLED=true
is set in the ConfigMap, and ◦ those two env vars are correctly wired to the containers. [HTTP 500 issue] While that issue focuses on 401/500s rather than “Operation not permitted”, misconfigured workload API auth can cause repeated failed HTTP calls from the server, which may surface as generic 
SdkClientException
in logs. 2. Helm 2.0.18/2.0.19 stability problems There is an open report that 
airbyte-server
fails when installing 2.0.18/2.0.19, without much detail. [server failed 2.0.18/19] This suggests there may be unresolved issues in these chart versions, so your error could be another symptom. 3. AWS permissions / IRSA / S3 Other issues show that when AWS IAM/IRSA is misconfigured, Airbyte logs S3-related exceptions (403s, falling back to node role, etc.). [IRSA fallback; workload pods SA] Those logs are different (403/authorization errors), but if your 
SdkClientException
is coming from S3 or Secrets Manager calls, it’s worth confirming: ◦ The pod is using the intended service account / IAM role. ◦ The role has the S3 and Secrets Manager policies recommended in the docs. [AWS infra] Given what’s available, here’s what you can concretely check: 1. Confirm where the exception comes from ◦ Look a few lines above/below the 
SdkClientException
to see which AWS client is failing (S3, Secrets Manager, or workload API HTTP client). ◦ If it’s S3/Secrets Manager, compare your IAM role policies with the examples in the AWS deployment docs. [AWS infra] 2. Validate workload API auth configuration (Kubernetes) ◦ In your Helm values / ConfigMap, verify: ▪︎ 
API_AUTHORIZATION_ENABLED=true
. [HTTP 500 issue] ◦ Inspect a running connector-sidecar pod and check if it has: ▪︎ 
WORKLOAD_API_AUTH_HEADER
▪︎ 
WORKLOAD_API_BEARER_TOKEN
If they’re missing, you’re likely hitting the same bug as in 2.0.1 OSS. [connector-sidecar auth] 3. Consider chart/version issues Since there is at least one open report of 
airbyte-server
failing on 2.0.18/2.0.19, you may be encountering another manifestation of that. [server failed 2.0.18/19] If possible, testing with a newer chart/app version (once available) or following any fixes mentioned in that GitHub issue would be advisable. Because the sources don’t show this exact 
Operation not permitted
stack trace, I can’t definitively say what’s causing it. If you can share (sanitized) surrounding log lines that show which client is failing and any HTTP status codes, you can then compare them directly with the linked GitHub issues to see if you’re hitting the same known problems. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
g
@kapa.ai Here is the full log entry:
Copy code
2026-02-26 21:33:47,529 [io-executor-thread-7]  ERROR   i.a.s.a.ApiHelper(execute):47 - Unexpected Exception
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Operation not permitted (SDK Attempt Count: 4)
        at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:130)
        at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:95)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:73)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
        at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
        at software.amazon.awssdk.services.s3.DefaultS3Client.listObjectsV2(DefaultS3Client.java:7327)
        at software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable$ListObjectsV2ResponseFetcher.nextPage(ListObjectsV2Iterable.java:154)
        at software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable$ListObjectsV2ResponseFetcher.nextPage(ListObjectsV2Iterable.java:145)
        at software.amazon.awssdk.core.pagination.sync.PaginatedResponsesIterator.next(PaginatedResponsesIterator.java:58)
        at io.airbyte.commons.storage.AbstractS3StorageClient.list(StorageClient.kt:602)
        at io.airbyte.commons.logging.LogClient.getLogs(LogClient.kt:63)
        at io.airbyte.commons.logging.LogClientManager.getLogs(LogClientManager.kt:49)
        at io.airbyte.commons.server.converters.JobConverter.getAttemptLogs(JobConverter.kt:112)
        at io.airbyte.commons.server.converters.JobConverter.getAttemptInfoRead(JobConverter.kt:93)
        at io.airbyte.server.apis.controllers.AttemptApiController.getAttemptForJob$lambda$0(AttemptApiController.kt:49)
        at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.kt:32)
        at io.airbyte.server.apis.controllers.AttemptApiController.getAttemptForJob(AttemptApiController.kt:43)
        at io.airbyte.server.apis.controllers.$AttemptApiController$Definition$Exec.dispatch(Unknown Source)
        at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invokeUnsafe(AbstractExecutableMethodsDefinition.java:461)
        at io.micronaut.context.DefaultBeanContext$BeanContextUnsafeExecutionHandle.invokeUnsafe(DefaultBeanContext.java:4438)
        at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:272)
        at io.micronaut.web.router.DefaultUriRouteMatch.execute(DefaultUriRouteMatch.java:38)
        at io.micronaut.http.server.RouteExecutor.executeRouteAndConvertBody(RouteExecutor.java:465)
        at io.micronaut.http.server.RouteExecutor.lambda$callRoute$5(RouteExecutor.java:442)
        at io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:92)
        at io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:232)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
        Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: Operation not permitted
        Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 2 failure: Unable to execute HTTP request: Operation not permitted
        Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 3 failure: Unable to execute HTTP request: Operation not permitted
Caused by: java.net.SocketException: Operation not permitted
        at java.base/sun.nio.ch.Net.connect0(Native Method)
        at java.base/sun.nio.ch.Net.connect(Net.java:589)
        at java.base/sun.nio.ch.Net.connect(Net.java:578)
        at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:583)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
        at java.base/java.net.Socket.connect(Socket.java:751)
        at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
        at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$DelegatingHttpClientConnectionManager.connect(ClientConnectionManagerFactory.java:86)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)
        at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:259)
        at software.amazon.awssdk.http.apache.ApacheHttpClient.access$600(ApacheHttpClient.java:104)
        at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:236)
        at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:233)
        at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:102)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:88)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:64)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:46)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:74)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:43)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:79)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:41)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:55)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:39)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.executeRequest(RetryableStage.java:93)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:56)
        ... 45 common frames omitted
k
The stack trace shows the AWS SDK failing on an S3 
ListObjectsV2
call when Airbyte tries to fetch job logs:
Copy code
software.amazon.awssdk.services.s3.DefaultS3Client.listObjectsV2(...)
...
at io.airbyte.commons.storage.AbstractS3StorageClient.list(StorageClient.kt:602)
at io.airbyte.commons.logging.LogClient.getLogs(LogClient.kt:63)
The immediate cause is a low‑level 
java.net.SocketException: Operation not permitted
when opening the TCP connection, not an S3 authorization error. From the available docs and issues, Airbyte expects: • A reachable S3 endpoint from the 
airbyte-server
pod. • Correct IAM permissions on the role/credentials used by the pod, including 
s3:ListBucket
and related actions on your bucket. [AWS infra] And we see related S3 list/logs failures in other deployments when S3 access is misconfigured (though those are 403s, not 
Operation not permitted
). [IRSA fallback] Given that, here’s what you can concretely verify: 1. S3 IAM policy for the role used by 
airbyte-server
 Ensure the IAM role (or access key) that 
airbyte-server
uses has at least the policy recommended in the docs, with your bucket name substituted:
Copy code
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListAllMyBuckets",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": "arn:aws:s3:::YOUR-S3-BUCKET-NAME"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:GetObject",
        "s3:GetObjectAcl",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::YOUR-S3-BUCKET-NAME/*"
    }
  ]
}
[AWS infra] 2. That the pod is using the intended IAM identity Other users have seen S3 log access fail when pods fall back to the node IAM role instead of the IRSA role or when the role isn’t correctly associated with the service account. [IRSA fallback] Double‑check your Helm values and service account/IRSA wiring if you’re using IRSA. 3. Network reachability from the 
airbyte-server
pod to S3 The 
Operation not permitted
at the socket level suggests the connection is being blocked (for example, by network policy, firewall, or egress restrictions) rather than S3 rejecting the request. The knowledge sources don’t provide specific network‑policy guidance, so I can’t say exactly what in your cluster is blocking it, only that Airbyte expects the cluster to be able to reach S3’s HTTPS endpoint. [AWS infra] The docs and issues don’t mention this exact 
SocketException: Operation not permitted
pattern, so I can’t definitively attribute it to a known Airbyte bug. Based on what is documented, the next steps are: • Confirm the S3 IAM policy matches the documented example for your bucket. • Confirm the pod is assuming the correct IAM role (if using IRSA). • Check cluster/network configuration to ensure the 
airbyte-server
pod has outbound HTTPS access to S3. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
3 Views
Open in Slack
PreviousNext
Powered by