This message was deleted.
# opal
s
This message was deleted.
a
Hi @Prakhar Saxena :) Could you please add your opal server and opal client config and also the http request you are sending? Would make it easier to reproduce and see what is misconfigured.
p
Copy code
opal_server:
  image: permitio/opal-server:latest
  environment:
    - UVICORN_NUM_WORKERS=1
    - OPAL_POLICY_REPO_URL=(Github Repo for userManagement for policy)
    - OPAL_POLICY_REPO_POLLING_INTERVAL=30
    - OPAL_AUTH_PUBLIC_KEY=${OPAL_AUTH_PUBLIC_KEY}
    - OPAL_AUTH_PRIVATE_KEY=${OPAL_AUTH_PRIVATE_KEY}
    - OPAL_AUTH_MASTER_TOKEN=${OPAL_AUTH_MASTER_TOKEN}
    - OPAL_AUTH_JWT_AUDIENCE=<https://api.opal.ac/v1/>
    - OPAL_AUTH_JWT_ISSUER=<https://opal.ac/>
    - OPAL_BROADCAST_URI=<postgres://postgres:test@broadcast_channel:5432/usermanagement>
    - OPAL_DATA_CONFIG_SOURCES={"config":{"entries":[{"url":"<postgresql://db:5432/usermanagement?user=postgres&password=test>","config":{"headers":{"Authorization":"Bearer ${OPAL_CLIENT_TOKEN}"},"fetcher":"PostgresFetchProvider","query":"SELECT * from public.user;","connection_params":{"password":"test"}},"topics":["policy_data"],"dst_path":"user"}]}}
  ports:
    - "7002:7002"
  depends_on:
    - broadcast_channel
  networks:
        - network-ums


opal_client:
  container_name: opal-client
  build:
    context: ./
    dockerfile: Dockerfile_OPAL
  environment:
    - OPAL_SERVER_URL=<http://opal_server:7002>
    - OPAL_LOG_FORMAT_INCLUDE_PID=true
    - OPAL_CLIENT_TOKEN=${OPAL_CLIENT_TOKEN}
    - OPAL_INLINE_OPA_LOG_FORMAT=http
    - OPAL_FETCH_PROVIDER_MODULES=opal_common.fetcher.providers,opal_fetcher_postgres.provider
    - OPAL_AUTH_JWT_AUDIENCE=<https://api.opal.ac/v1/>
    - OPAL_AUTH_JWT_ISSUER=<https://opal.ac/>
  ports:
    - "7766:7000"
    - "8181:8181"
  depends_on:
   - opal_server
   - db
The request i am sending on opal server using its rest api is: { "id": "randomid", "entries": [ { "url": "postgresql://db:5432/usermanagement?user=postgres&amp;password=test", "config": { "headers": { "Authorization": "Bearer ${OPAL_CLIENT_TOKEN}" } }, "topics": [ "policy_data" ], "dst_path": "/user", "save_method": "PATCH", "data": [ { "op": "add", "path": "/result", "value": { "first_name": "abch", "id": 112, "last_name": "xyz", "password": "1234", "role": "Admin", "updated_at": "today", "username": "abcd" } } ] } ], "reason": "user abcd is added to the system", "callback": { "callbacks": [] } }
a
@Prakhar Saxena the url instructions you are sending will by default by processed by the built-in HTTP fetcher. What you need is to tell OPAL to use the postgres fetcher. There is an example docker compose file in the repo and also an example for the request you should be sending after OPAL is up:
Copy code
{
  "config": {
    "entries": [
      {
        "url": "<postgresql://postgres@example_db:5432/postgres>",
        "config": {
          "fetcher": "PostgresFetchProvider",
          "query": "SELECT * from city;",
          "connection_params": {
            "password": "postgres"
          }
        },
        "topics": [
          "policy_data"
        ],
        "dst_path": "cities"
      }
    ]
  }
}
(Please note that you need to both change the opal configuration according to the docker compose example i linked to and also change the http request. Only doing one will not work.)
p
@Asaf Cohen Following your advice I have change my request json: Request Method: POST Request URI: http://localhost:7002/data/config But still I'm unable to get the desired output, I want to update the content of the user with id 1. Please help me where I'm doing wrong.
Copy code
{
  "id": "randomid",
  "entries": [
    {
      "url": "<postgresql://db:5432/usermanagement?user=postgres&password=test>",
      "config": {
        "fetcher": "PostgresFetchProvider",
        "query": "SELECT * from public.user;",
        "connection_params": {
          "password": "test"
        }
      },
      "topics": [
        "policy_data"
      ],
      "dst_path": "/user",
      "save_method": "PATCH",
      "data": [
        {
          "op": "remove",
          "path": "/user/1"
        }
      ]
    },
    {
      "config": {
        "headers": {
          "Authorization": "Bearer {$OPAL_CLIENT_TOKEN}"
        }
      }
    }
  ],
  "reason": "user 1 is removed from the system",
  "callback": {
    "callbacks": []
  }
}
a
Hi @Prakhar Saxena did you try to also change the OPAL server and client configuration as demonstrated by the example docker compose ? what are you seeing in the logs when sending this request?
p
Hi, @Asaf Cohen (Permit.io) I am getting this in my logs when sending the above request
a
Hi @Prakhar Saxena, we can help you over a short zoom call to configure the postgres integration. Please click here to book time with us: https://calendly.com/asaf-cohen/30min?back=1&amp;month=2023-09&amp;date=2023-09-18
1
i
Hi @Prakhar Saxena
Did you able to get your OPAL and Postgres connection fixed?
p
Hi, Yes !
i
@Prakhar Saxena Thanks for your concern Can I ask for some doubts that I have form you?
p
Yes, please !
i
I went with the example that is given in the repo https://github.com/permitio/opal-fetcher-postgres/blob/master/docker-compose.yml But still, I can't figure out if I update the DB from another service how will it trigger OPAL client so that OPAL will fetch the new data from the db and update the OPA data could you please help me to figure out this. Really thanks
p
for that there is one API you need to hit to update your OPA data. It is mentioned in the docs https://docs.opal.ac/tutorials/trigger_data_updates. In the url key you have to mention from where you want to fetch the data and in the destination key you have to add the path to you opa data.
Check Option-2 in the doc
i
Thanks I read and try that but in that way it update the OPA state and then how can I then update the DB? Then in the next time when I stop and start the process I loads the old DB data
Then what I got is every time I update the DB from another service, I need to hit this endpoint and tell OPAL to load the new data? isn't it?
p
You can create a webhook for this api and add that in your update or create api(s). Same for updating another DB as well.
So you don't have to hit this again and again.
i
Thank a lot Prakhar Do you know is there a way to trigger OPAL server from postgresDB when ever some update operation happen in the DB. In the docs they are telling about postgres triggers.
p
You can use Listen/Notify in postgres.
i
Ok Prakhar. I will look into that also. Thanks a lot for your time.
👍 1