https://venicedb.org logo
Join Slack
Powered by
Has anyone looked into integrating Venice with Sno...
# general
a
Has anyone looked into integrating Venice with Snowflake as a batch data source or batch ingestion layer? Since we already use Snowflake for transformations, I’m wondering what the main challenges or limitations might be.
k
It's been over a year since I've used Snowflake, so I am not up to date with their current feature set. Are you looking to take your data from Snowflake Tables and load them into Venice? Or do they offer some kind of blob storage now?
Actually, there is a Snowflake Connector for Spark. You might be able to use this. @Nisarg Thakkar would know more
Looks like it copies over the Snowflake table to a blob storage, then Spark uses that as an input
a
Are you looking to take your data from Snowflake Tables and load them into Venice
yes, that would be our scenario.
👍 1
Looks like it copies over the Snowflake table to a blob storage, then Spark uses that as an input
so we need to bring spark for loading?
k
For batch ingestion into Venice, yes.
What is the size of the data?
n
Snowflake -> Venice - VenicePushJob (VPJ): One requirement of VPJ today is that we require the source data to be associated with an Avro schema. If the data can be associated with an Avro schema, it is not too hard to integrate it as a source for VPJ. We have most of the abstractions for it. Lately, we've had many cases where this requirement seemed to be holding us back, and something we should rethink.
f
We have some building blocks that might help towards this… In the venice-duckdb module we have some class for mapping between JDBC type <-> Avro type. Ideally this should be extracted out of that module and made a bit more generic. We should ideally be able to represent the Venice data model as just "abstract" data types / SQL. Then the mapping from the abstract types to Avro can happen behind the scene.
💡 1
This could also help with integration of non-Avro serde (e.g. proto).
I guess from the point of view of a batch SQL processor (no matter if it’s Snowflake, Trino, SparkSQL, FlinkSQL, Hive or anything else…) it would be great if we could do something along the lines of:
Copy code
INSERT INTO my_venice_table SELECT * FROM my_snowflake_table;
Or any other arbitrary query instead of the second table. https://www.w3schools.com/sql/sql_insert_into_select.asp
That ^ kind of integration would require deeper work in the source SQL system… I’m sure for some source systems the pluggability would allow it, but probably not all.
The approach of running VPJ in Spark mode over some Snowflake input is probably the most straightforward in the short term. Though it may require an extra hop or jumping through certain hoops…
k
One important thing to keep in mind, is that Snowflake charges per query time. Select on all of the rows may be pretty expensive. I think bulk load into blob storage would be cheaper and faster
🚀 1
a
This is super helpful. Thank you!
is avro a hard requirement. Would it be a lot of work to support pbuf?
n
Today, yes - Avro is a hard requirement, and quite ingrained in our codebase. We use Avro on the wire and on disk. Supporting more formats would be non-trivial amount of effort We could take a shortcut and convert from Proto -> Avro while writing to Venice and Avro -> Proto when reading. The interfaces to convert data formats exists in some of the read path, but it's not available in the write path today
a
This is great and as long as we can customize the ingestion we should be fine.
n
Does Snowflake offer a compute engine like Spark too? Or is it just a data lake where data is stored?
a
yes, we rely on Snowflake for most of our heavy analytics workloads
5 Views
Open in Slack
PreviousNext
Powered by