This message was deleted.
# opals
Slackbot
08/11/2023, 7:54 AMThis message was deleted.
f
Filip
08/11/2023, 8:18 AMHey @Denys Andriyanov ☺️ I’ll get this out to the engineering team to get you sorted and see how we can help.
d
Denys Andriyanov
08/11/2023, 8:31 AMtnk u
Denys Andriyanov
08/11/2023, 8:34 AMnote: if i remove ' ' around the OPAL_DATA_CONFIG_SOURCES value i am not getting warning anymore but the entry in data is not getting created.
Denys Andriyanov
08/11/2023, 8:34 AMwhile, again, when doing a manual call with the same payload does the job
o
Or Weis
08/11/2023, 8:36 AMHi @Denys Andriyanov
Checkout the example here, I believe you just need to use the right formatting
https://github.com/permitio/opal/blob/dcdbf50dd72c94d903cf468359f0116e0d3a9ce4/docker/docker-compose-example.yml#L35
d
Denys Andriyanov
08/11/2023, 8:42 AM@Or Weis you mean remove ' ', i did that and no longer getting warning, though i do not see the entry created, but
Denys Andriyanov
08/11/2023, 8:43 AMwhat i found is interesting it seems that it first fetches the date from my sqlite base and it is successfully, but then it overwrites it with data it got from POLICY_REPO_URL, see logs below
Denys Andriyanov
08/11/2023, 8:43 AMopal-fetcher-sqlite-opal_client-1 | 2023-08-11T083837.187773+0000 | opal_client.engine.logger | INFO | Received request. PUT /v1/data/cities
opal-fetcher-sqlite-opal_client-1 | 2023-08-11T083837.188614+0000 | opal_client.engine.logger | INFO | Sent response. PUT /v1/data/cities -> 204
opal-fetcher-sqlite-opal_client-1 | 2023-08-11T083837.188806+0000 | opal_client.engine.logger | INFO | Received request. GET /v1/policies
opal-fetcher-sqlite-opal_client-1 | 2023-08-11T083837.188978+0000 | opal_client.engine.logger | INFO | Sent response. GET /v1/policies -> 200
opal-fetcher-sqlite-opal_client-1 | 2023-08-11T083837.190852+0000 | opal_client.engine.logger | INFO | Received request. PUT /v1/data
opal-fetcher-sqlite-opal_client-1 | 2023-08-11T083837.191371+0000 | opal_client.engine.logger | INFO | Sent response. PUT /v1/data -> 204
Denys Andriyanov
08/11/2023, 8:44 AMyou see it firsts PUT /v1/data/cities (those were fetched from an external source in my case sqlite) but then it PUT /v1/data
Denys Andriyanov
08/11/2023, 8:46 AMmaybe it should be the other way around? so it first PUT the data from POLICY_REPO_URL and after it on top of it put's data from the OPAL_DATA_CONFIG_SOURCES
o
Or Weis
08/11/2023, 8:47 AMYes it's a classic race conditions, you shouldn't start updates before the intial data load is complete.
You can use the health checks the verify the state
https://docs.opal.ac/tutorials/healthcheck_policy_and_update_callbacks/
d
Denys Andriyanov
08/11/2023, 8:58 AMbut i am not starting updates, i expect
OPAL_DATA_CONFIG_SOURCES='{"config":{"entries":[{"topics":["policy_data"],"dst_path":"cities","url":"file:/db/opal.db","config":{"fetcher":"SQLiteFetchProvider","query":"SELECT * FROM city;"}}]}}'
to put my initial data
Denys Andriyanov
08/11/2023, 8:59 AMon top of data from POLICY_REPO_URL, which in my demo case is https://github.com/permitio/opal-example-policy-repo/blob/master/data.json
o
Or Weis
08/11/2023, 8:59 AMSo maybe you have a data.json file in your repo ?
d
Denys Andriyanov
08/11/2023, 9:00 AMno, it's - OPAL_POLICY_REPO_URL=https://github.com/permitio/opal-example-policy-repo
o
Or Weis
08/11/2023, 9:01 AMYeah that has a data.json in it.
You should either not use data,json in parallel to OPAL_DATA_CONFIG_SOURCES; or have them pointed to different sub keys
✅ 1