We are experimenting with a long-living source connector and seem to be observing that the RECORD messages are not processed by the destination connector until the source has exited successfully. Is this the case? My understanding coming from the Singer paradigm was that output of the "tap" could be piped into the "target" for processing live as it comes in (like
tap | target
). Is this how Airbyte works?
02/07/2022, 8:12 PM
Hey are you going to use singer connectors otherwise all CDK based connectors have a configuration (10000) records where we move all these 10000 records to destination
Okay, so the record messages are passed to the destination in batches of 10,000 by default? Is this parameter configurable?
02/07/2022, 10:11 PM
(we are using the Python CDK)
02/09/2022, 9:12 AM
My bad looks like there isn't any configuration like that. And it's purely sends on memory awareness but for sure doesn't wait till the whole sync is done