hi! i'm working on the Convex source connector, which is an incremental HttpStream. Can someone describe how pagination is supposed to work, with the

state

property,

stream_state

argument, and

next_page_token

? We only have one cursor, which I was updating in

read_records

. But then

check_availability

started calling

read_records

and advancing the cursor but discarding results, which makes us skip records. From the code it looks like checkpoint state is determined by the (deprecated)

get_updated_state

after calling

read_records

, which means

read_records

must advance the cursor. But the availability check calls

read_records

and discards the result, which means

read_records

must not advance the cursor. This looks impossible to implement correctly.

A message from kapa.ai

I'm considering updating the cursor in

next_page_token

, but

next_page_token

is called by

read_records

. it happens to work because it's called after the record is yielded, so if you discard the iterator immediately after the first result, as the availability check does, then it's correct. but this argument seems sketchy