I have some "feature wishes" for the connector Builder UI, and am not sure if this is the best place to register them. But here goes: 1. for pagination it should be possible to create a user input value for Page Size, rather than setting a static number in the UI. After all I might want 200 records per page but another person using the same connector might want 20. Plus a user input value could be used in a Stop Condition like this:

{{ last_records | count < {{ config['records_per_page'] }} }}

2. Incremental sync should allow for not just time-based cursoring but also integer/id-based cursoring. The API endpoint I've set up incremental sync for works a lot better if I can "scroll down" to the latest known ID rather than latest known timestamp. That's because this API only allows for starting from a specific date (format YYYY-MM-DD), not from a specific timestamp. So at best I can fetch everything since the last known date, which renders intra-day incremental sync repetitive/pointless and involves a lot of de-depuping. So it's best to simply wait until just after midnight and fetch the last 24 hrs worth of data.

A message from kapa.ai

couldn't agree more with #2 .. i posted a thread in this channel a few days back. the use-case i envision is slightly different than what @Dan Cook has described (i.e., i want to use an integer unix timestamp) but requires the same basic functionality.

Incremental id-based cursoring is something we have discussed internally in the past but we’re sure about the priority. I’ll bring your feedback back to the team and see where we land on this specific topic. As for unix timestamps @David Anderson, I’ll reply on your thread and we can have the discussion there