This message was deleted Permit #opal

Join Slack

This message was deleted.

# opal

Slackbot

01/23/2023, 6:44 PM

This message was deleted.

Or Weis

01/23/2023, 6:51 PM

HI @Avo Sepp 🙂 Your questions are welcome. I’m not that familiar with CosomsDB (Googling as I’m typing this 😅) - but if it provides an HTTP Api you can use the builtin Http data-fetcher

Or Weis

01/23/2023, 6:54 PM

Yep seems like it: https://learn.microsoft.com/en-us/rest/api/cosmos-db/get-a-document

Avo Sepp

01/23/2023, 6:55 PM

You found it faster than I did! Thank you

💜 1

Or Weis

01/23/2023, 6:55 PM

Got a blackbelt in googling , working on my chatGPT skillz now 😛

Or Weis

01/23/2023, 6:58 PM

Note you might still want to fork the HTTP fetcher, just to change the processing a bit - as it seems that the API call also returns a lot of metadata you might need to remove

Avo Sepp

01/23/2023, 6:58 PM

eek. Okay good to know.

Or Weis

01/23/2023, 7:02 PM

Don’t worry it’s super easy to do. Say you only want a specific field called “magic” from a document - Basically replace line 109 here - instead of

return data

with

return data['magic']

Avo Sepp

01/23/2023, 7:03 PM

I think a link is missing

Or Weis

01/23/2023, 7:04 PM

Indeed . Fixed

Avo Sepp

01/23/2023, 7:04 PM

In the case of forking this code. Should I fork the repo and be building on our own pipelines? Right now I’ve just installed via Permit’s public Helm chart.

Avo Sepp

01/23/2023, 7:04 PM

I guess it’s python. I can just insert the modified .py?

Or Weis

01/23/2023, 7:05 PM

Yep. Data-fetchers are extensible modules. You basically make a copy of that file and save it in the save folder under a different name, and you use that name in the event you trigger

Or Weis

01/23/2023, 7:05 PM

https://docs.opal.ac/tutorials/write_your_own_fetch_provider

Or Weis

01/23/2023, 7:07 PM

We can also look at maybe adding some filtering logic to the built-in HTTP fetcher - as part of it’s config

Avo Sepp

01/23/2023, 7:10 PM

that would be pretty useful.

Or Weis

01/23/2023, 7:23 PM

You feel like tackling/ contributing it? Alternatively I can get to it this weekend

👀 1

Avo Sepp

01/23/2023, 7:31 PM

If I have some time, could be fun

Or Weis

01/23/2023, 7:32 PM

Swag package guaranteed - and of course undying fame as an OSS contributor 😇

❤️ 1

Avo Sepp

01/23/2023, 7:33 PM

I’ll give it a look. haha. If it helps me understand OPAL/OPA better. Can’t hurt.

Avo Sepp

01/23/2023, 7:40 PM

Does OPAL offer any sort of transform functionality? Or should I make sure the database I’ll pull from is transformed beforehand.

Or Weis

01/23/2023, 7:59 PM

Custom Data-fetchers are the components that are usually used to transform data - same process
method we’ve been discussing. This filter mechanic can be a first start for builtin transforming logic. We can also look at potentially using the Rego policy to do transformations - though that’s quite a bit of work

Avo Sepp

01/23/2023, 8:08 PM

I had a feeling you’d say custom data-fetchers. haha

😅 1

Open in Slack

Previous Next