Hi everyone, just wondering how you would approach this: We have sources that take a long time to sync. To save us from running Redshift basically 24/7, we’re going to dump the streams into Parquet files in S3 first as a data lake. My question is: how would you get those files into Redshift? I looked at the S3 connector, but this would require setting up a source and connection per synced stream into Redshift, which seems wildly inefficient. I could do a Lambda function, but thought I’d see if there are any other options floating around before I start coding something custom.

Hey @Liam Coley, Sorry for the delay, we’ve been experiencing a large influx of user questions and have been working hard to try to reply to everyone. Were you able to sync your files into Redshift? There’s perhaps a possibility of rewriting the S3 connector to fit your use case.

Thanks @Saj Dider (Airbyte), in the interest of getting something running in the short term I’ve just done an S3 source per stream into Redshift (using the

Standard

option). This seems to work alright, but does require a bit of configuration for each stream (and I am currently hitting some problems with embedded JSON objects in columns). If the S3 source connector could allow one to define multiple streams that would be great, even if it was just a JSON definition like

{stream_name:{pattern: "file/path/pattern**", prefix: "optional_path_prefix"}}