Heya team, I'm building a connector for an api that pulls different energy production data (Netanders) for the Dutch market. The api produces volume (in mwh) per 15 min segments of the data and uses a date filter with a before and after date (2023-06-13 is captured by before=2023-06-13 and after=2023-06-12) and only allows for a full date of data. I want to incrementally add rows for yesterday but struggling with the incremental config. My idea is to inject the before/ after date dynamically into the url with end date set to today (as date) and use a step window of P1D. This however produces the same start/ end date in the test string. Any clues?

A message from kapa.ai

This was what I already had but this didn't work, tried something else instead so hoping this will

I had a similar issue with with an API that wanted from 01.01.2023T000000, : 02.01.2023T000000 and didn't return data if the to was 01.01.2023 235959 , i got around it by setting the window to 86401 Seconds, the only problem being that if loading more than 1 day of data the window moved by 1 sec per day.. The API i used still gave data for the full day (hour format) when doing increments. This isn't a clean solution but might be worth a try to see if it works.

I basically managed to get this one working:

incremental_sync:
      type: DatetimeBasedCursor
      cursor_field: validfrom
      datetime_format: '%Y-%m-%d'
      end_time_option:
        type: RequestOption
        field_name: validfrom[strictly_before]=
        inject_into: request_parameter
      start_time_option:
        type: RequestOption
        field_name: validfrom[strictly_after]=
        inject_into: request_parameter
      cursor_granularity: "PT15M"
      start_datetime:
        type: MinMaxDatetime
        datetime: '2023-06-11'
        datetime_format: '%Y-%m-%d'
      end_datetime:
        type: MinMaxDatetime
        datetime: '{{ now_utc().strftime(''%Y-%m-%d'') }}'
        datetime_format: '%Y-%m-%d'
      step: P2D

But it fails on the 2nd incremental load somehow and is also suffering from rate limiting it seems. The data is in 15 time windows but the url accepts only full dates. Using P2D guarantees that the end date is 1 full day after the start date (now 2 days in the past). I'm a bit oblivious no how aribyte handles the second sync