Hello 👋, I am trying to extend the Mixpanel souce connector to support incremental syncs for some of the streams that can have custom properties by using user defined cursors. With that being said I'm not sure of what is the correct approach for retrieving the user defined cursor in a HttpStream if there is no default cursor 🤔 The immediate solutions that comes to my mind is to override the HttpStream read_records() method to get access to the configured stream cursor like in the following example

def read_records(self, sync_mode: SyncMode, cursor_field: List[str] = None, stream_slice: Mapping[str, Any] = None,
                     stream_state: Mapping[str, Any] = None) -> Iterable[Mapping[str, Any]]:
        cursor_name = cursor_field[-1]
        stream_state["cursor_key"] = cursor_name
        return super().read_records(sync_mode, cursor_field, stream_slice, stream_state = stream_state)

After this I will be able to access the cursor in the parse_response() and get_updated_state() methods in order to do the necessary filtering on the records. Is there a better approach to achieve this or is this the only way?

loading...

I saw you submitted the PR https://github.com/airbytehq/airbyte/pull/11501, do you need any extra help here @Ivica Taseski?

I think that the current approach should be fine. I was more wondering if there is some other way to do it 🤔