We are experimenting with a long-living source con...
# connector-developmentm
Maxime edfeed
02/06/2022, 11:37 PMWe are experimenting with a long-living source connector and seem to be observing that the RECORD messages are not processed by the destination connector until the source has exited successfully. Is this the case? My understanding coming from the Singer paradigm was that output of the "tap" could be piped into the "target" for processing live as it comes in (like
tap | targetr
Roman
02/07/2022, 8:12 PMHey are you going to use singer connectors otherwise all CDK based connectors have a configuration (10000) records where we move all these 10000 records to destination
Roman
02/07/2022, 8:51 PMOkay, so the record messages are passed to the destination in batches of 10,000 by default? Is this parameter configurable?
e
Ethan Veres
02/07/2022, 10:11 PM(we are using the Python CDK)
h
Hitesh Khandelwal
02/09/2022, 9:12 AMMy bad looks like there isn't any configuration like that. And it's purely sends on memory awareness but for sure doesn't wait till the whole sync is done
3 Views