Hello, not letting the user choose the cursor field for "Full refresh | Overwrite + Deduped" is an issue, considering the cursor is used for the deduplication. It makes this option basically unusable for most connections

In most API connections, this is defined by the source . . . whereas you can select it in most database connections. Is there a specific connection you're having this issue with?

I can't select it in Postgres nor in s3 For s3 I suppose the usual source-defined cursor is applied, though it's not shown, but considering it's the file's last modification timestamp it's also an issue not to be able to change it. What if I have a deduplicate in my latest file? As for Postgres, I can't tell what it's using at all

Which update method are you currently set to use on your Postgres source?

User Defined Cursor. Works a charm for Incremental, but not for Full refresh

Hey Swan. We're interested in prioritizing this one. It would help a lot if you could provide a bit more of a breakdown about when this matters and how you'd like to see it addressed ideally.

Hi Ian, thanks for taking an interest. This really only matters if source is dirty (but then again why else dedupe an overwrite). Here's the issue I had when I wrote this: a company I work with exports data from one of their software and pushes it in s3 as csv. Somehow they messed up and squashed two exports together, meaning one of my files was almost entirely made of deduplicates. There's an update datetime in these files, and they're even formatted correctly so they can be used as strings. But I couldn't use the update datetime because I couldn't select the cursor. I assume it would have used the source-defined cursor, which is the file last modified datetime and therefore wouldn't work. I ended up deleting the weird file and requesting a new export. I only checked Postgres out of curiosity, and came to find out the option was completely unusable there, as there's no way to know which cursor will be used. It's hard to get dirty data in Postgres, but still, if the option exists at all it should be usable.

Thanks! Added to the roadmap. Not immediately sure when this will ship though.