Hi all have a question regarding loading data from...
# ask-community-for-troubleshooting
a
Hi all have a question regarding loading data from s3 to snowflake. I have a folder in s3 where I store 100+ tables each with a respective folder. While I trying out the File connector with s3, the config is forcing me to have a specific table meaning that for my case I would need 100+ connections to feed all of these tables in my Warehouse. Is there a way around this?
u
Not at the moment, there are two issues related to this. • https://github.com/airbytehq/airbyte/issues/3965 Native Source connector for AWS S3 • https://github.com/airbytehq/airbyte/issues/1874 Support directories in File connector
g
I'm currently working on this, building a new incremental, multi-file ready source connector to solve some of these limitations with the current file-source. However, the design I have at the moment still retains a one table per connection, primarily because: • we want to allow (optionally) a schema to be provided rather than relying on inference. • we want to allow glob-style pattern matching to determine which files belong to a table. • we want to allow granular (table-level) settings for what to do in cases of missing or new columns, or malformed rows. Providing these could get a bit unwieldy in one connection for multiple tables but would be keen to understand more about your use-case @Arnis Dizdari :)
u
There is no workaround that I know at the moment, one suggestion is use the Airbyte API to create the source for you.
a
@George Claireaux (Airbyte) For example, I have an issue where our data is loaded via a JDBC (Infor CloudSuite) driver and I want to create tables in my DW Snowflake. I have a python script where I pull the 100+ tables and store them a folder with new incrementals coming hourly. What would you recommend to load the data in the DW. Creating a custom connector?
u
What would you recommend to load the data in the DW. Creating a custom connector?
You definitely can use the CDK to build a custom connector for your use case before the work George finish
👍 1