We are following datahub aws guide to setup. We are running following service in the datahub 1. datahub-acryl-datahub-actions 2. datahub-datahub-frontend 3. datahub-datahub-gms 4. datahub-schema-registry We have following aws components 1. Kafka 2. RDS(Mysql) 3. Opensearch To make connection with elasticsearch (opensearch) datahub services

datahub-datahub-gms

not able to connect with elasticsearch. Getting following error in the log

2022/07/16 11:25:03 Received 403 from <https://vpc--datahubes-vsqu3edgv5tsiyshvnq.ap-south-1.es.amazonaws.com:443>. Sleeping 1s
2022/07/16 11:25:04 Received 403 from <https://vpc-datahubes-vsqu3edgv5tsiyshvnq.ap-south-1.es.amazonaws.com:443>. Sleeping 1s

While we have attached the serviceAccount with the pods and the AWS IAM role is also whitelisted in the resource policy with proper permission. I want to confirm that does datahub support AWS signature version 4 to make connection with AWS elasticsearch.

I got same error, but i found it my ES access policy is configured wrong, i fixed it is working fine now

Hi @fancy-thailand-73281! Can you provide some extra context about how you fixed it?

Hi @little-megabyte-1074 I add access policy in security configuration to the Elastic search in AWS.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:{aws_account_number}:domain/{elastic_search_name}/*"
    }
  ]
}

In this policy we are allowing everyone to connect elasticsearch cluster.

You can restrict as per your needs, instead of * you can restrict to your application role ARN

Hello, I am experiencing this same problem. When I use the policy stated above that grants access to all principals I am able to connect to opensearch. However, when I update the resource policy to restrict to a specific role (see below) I hit the same 403 issue.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::{account_id:role/{my_role}"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:{region}:{account_id}:domain/{my_domain}/*"
    }
  ]
}

@numerous-tent-25319 were you able to get this working using a resource policy with a role ARN specified as the principal?

Hello! @chilly-daybreak-28356 were you able to get it working? I’m currently facing the same problem as you. It works just fine when there is no principal specified, but as soon as I restrict the policy to the role the pods are using it stops working.

I have managed to get it working only by allowing all access to opensearch

Hey yall, i’m experiencing the same issue on my deployment (EKS, datahub-helm, AWS opensearch ES 7.9/7.10). I couldn’t believe that our access policies weren’t setup correctly (we followed nearly every guide out there for this), so we put up a sidecar container next to

datahub-gms

service and created an opensearch client from within it. This container has the same exact env vars, certs, uses the same serviceAccount, etc as

datahub-gms

. Using a python opensearch client and boto3, we’re able to reach opensearch with only role specific access policies applied. This leads me to believe the clients used by

datahub-gms

and

datahub-elasticsearch-setup-job

are configured to only access elasticsearch/opensearch with username:password. happy to help the datahub team investigate this further if needed (or be corrected)!