This message was deleted.

The

HttpFetchProvider

sets the

event.config

attribute in its constructor if it's not already set. You can move your assignment to before calling the superclass constructor (which is actually what

HttpFetchProvider

does too), e.g.

class KiplotHttpFetchProvider(HttpFetchProvider):

    def __init__(self, event: KiplotHttpFetchEvent) -> None:
        if event.config is None:
            event.config = KiplotHttpFetcherConfig()
        super().__init__(event)

Hmm...that still doesn't work, but what does work is removing the `if`:

def __init__(self, event: KiplotHttpFetchEvent) -> None:
        event.config = KiplotHttpFetcherConfig()        
        super().__init__(event)

Why does this work but not with the

if

? Shouldn't

event.config

be

None

since

KiplotHttpFetchProvider

is the most derived type?

You get an already constructed

event

which might have a

config

set with data. By using Pydantic's ORM mode (which doesn't actually touch any database, it just enables some functionality that's usually used with ORMs), you can upgrade it to

KiplotHttpFetcherConfig

without losing existing data:

class KiplotHttpFetchEvent(HttpFetchEvent):
    fetcher: str = "KiplotHttpFetchProvider"
    config: KiplotHttpFetcherConfig = None
    class Config:
        orm_mode = True

    def __init__(self, event: KiplotHttpFetchEvent) -> None:
        event.config = KiplotHttpFetcherConfig.from_orm(event.config or HttpFetcherConfig())
        super().__init__(event)

pydantic.errors.ConfigError: You must have the config attribute orm_mode=True to use from_orm

class KiplotHttpFetchEvent(HttpFetchEvent):
    fetcher: str = "KiplotHttpFetchProvider"
    config: KiplotHttpFetcherConfig = None

    class Config:
        orm_mode = True

class KiplotHttpFetchProvider(HttpFetchProvider):

    def __init__(self, event: KiplotHttpFetchEvent) -> None:
        event.config = KiplotHttpFetcherConfig.from_orm(event.config or HttpFetcherConfig())     
        super().__init__(event)

My mistake, I meant to add the

orm_mode = True

in

KiplotHttpFetcherConfig

, not

KiplotHttpFetchEvent

Ah - makes sense, that works, but the values of the

resource_type

and

resource_id

fields in

KiplotHttpFetcherConfig

are being lost by the time it reaches `_process_`:

| opal_fetcher_kiplot_http.provider       | INFO  | KiplotHttpFetchProvider received incoming event, resource_type: None resource_id: None

This is the payload I'm sending to the API:

{
  "entries": [
    {
      "config": {
        "fetcher": "KiplotHttpFetchProvider",
        "resource_id": "1234",
        "resource_type": "outcome"
      },
      "dst_path": "/outcomes/1234",
      "save_method": "PUT",
      "topics": [
        "test_topic"
      ],
      "url": "<http://host.docker.internal:44444/test_data.json>"
    }
  ]
}

Any ideas? Thanks for all the help so far btw, appreciate it 🙂

Looks like the

parse_event

function gets called by the base class and the implementation in

HttpFetchProvider

overrides your config with the base class. Try overriding it too:

def parse_event(self, event: FetchEvent) -> HttpFetchEvent:
        return KiplotHttpFetchEvent(**event.dict(exclude={"config"}), config=event.config)

Thanks - that combined with removing the

_init_

override got it working - final minimal implementation of a custom HTTP fetcher looks like this:

from aiohttp import ClientResponse
from pydantic import Field
from typing import Optional

from opal_common.fetcher.providers.http_fetch_provider import HttpFetchProvider, HttpFetcherConfig, HttpFetchEvent
from opal_common.fetcher.logger import get_logger
from opal_common.fetcher.events import FetchEvent

logger = get_logger("kiplot_http_fetch_provider")

class KiplotHttpFetcherConfig(HttpFetcherConfig):
    """
    Config for KiplotHttpFetchProvider-s
    """
    fetcher: str = "KiplotHttpFetchProvider"

class KiplotHttpFetchEvent(HttpFetchEvent):
    fetcher: str = "KiplotHttpFetchProvider"
    config: KiplotHttpFetcherConfig = None

class KiplotHttpFetchProvider(HttpFetchProvider):
    def parse_event(self, event: FetchEvent) -> KiplotHttpFetchEvent:
        return KiplotHttpFetchEvent(**event.dict(exclude={"config"}), config=event.config)

    async def _process_(self, res: ClientResponse):
        self._event: KiplotHttpFetchEvent

        # TODO: process data
        return res

Would be handy to have something similar in the docs - I expect the use-case of needing a HTTP fetcher with custom processing is pretty common - in our case the plan is to use

_process_

to restructure API responses into the correct shape for OPA, allowing the use of all our existing APIs to populate OPA with no application changes

Glad to be of assistance! Regarding docs, that's an excellent idea. Do you mind if I use your code as a base for the example? Let me know if you have any more questions.

Sure, go ahead!