I'm still a bit fuzzy on #6. How would we handle this case now that

versionNumber

is persisted in the class in my latest commit?

@Zac Policzer I'm going with a lambda approach to create a new

DaVinciRecordTransformer

for each new version as discussed in the call. But I'm running into an issue of how to get the current store version before the backend is initialized here?

There should be a version for each ingestion task

The change would be to invoke the supplier whenever we gotta create an ingestion task

It looks like my latest commit is missing on branch coverage. I'll take a look in the evening.

@Zac Policzer My unit test for recording transformer error count isn't invoking

processKafkaDataMessage

in

StoreIngestionTask

. Here's my test. What is the correct way to test

processKafkaDataMessage

? Thanks.

Hey @Koorous Vargha, I'm gonna be traveling for a bit. But maybe @Xin(Adam) Chen can help here?

It looks like the test is still failing on the CI. I will need to pick this up later next week as I'll be traveling.

I was able to fix the test and code coverage issue. It's good to review now @Zac Policzer

Thanks for suggesting IntelliJ @Xin(Adam) Chen. I was able to find an open-source version of it. It was a lot easier to debug.

@Koorous Vargha, just approved and merged your PR 🎉

Awesome. Thank you!

Nice! Great work!

I think we made have made a slight oversight. Looking at the interface of record transformer:

Meeting Notes: • persistent side effects. Transformer writes to local file system. ◦ Can add flush to the interface and called in the StoreIngestionTask syncOffset. Call before executeStorageEngineRunnable if recordTransformer isn't null. Default implementation will be to do nothing. ◦ Flush would need to be called prior to checkpoint flush. ◦ Need a path that is provided by the framework. Can provide a default one. Allow user to pass in the path in the factory. Have default path? This can be in backendConfig starting from CachingDaVinciClientFactory. IN ConfigKeys.java called DATA_BASE_PATH. Instead of a function for initing the RecordTransformer we can create a factory. Maybe in the constructor we create a RecordTransformer config object, instead of a bunch of parameters. ◦ Do we want the user be able to call flush? ◦ Need some class level lock on start. Basically not allowing put to go through while it's loading.

@Elijah Grimaldi [dvc][doc] Create MVP for DaVinciRecordTransformer by kvargha · Pull Request #1087 · linkedin/venice (github.com)

I've looked into the PR and caught up a bit on the changes, do you think we could reiterate the concerns in this thread of how to make the record transformer and blob transfer compatible here?

Hey guys, I know it's been a while but I was working on transferring the hashCode of the record transformer function during the BLOB metadata transfer here: https://github.com/linkedin/venice/compare/main...elijahgrimaldi:venice:checksum-transfer is this what you guys had in mind?

I didn’t implement the comparisons yet if the transfer will be compatible

https://github.com/linkedin/venice/pull/1280 Implemented the discussion we had above

I did a small write-up behind the background of DVRT to help you get started @Kristy Lee. Please take a look and don't hesitate to ask questions. https://github.com/linkedin/venice/issues/1563

Thanks for the update. I will go ahead and take a look.

Hi, I have been looking at #1563 and referencing DuckDBDaVinciTransformer for DVRT integration. I may look at integrating this into graph (database), and was wondering if there is a file to look at to get started on this.

message has been deleted

@Kristy Lee We’re on the free tier of Slack, so I think the canvas you created will be automatically deleted after a year. For persistence, maybe you could do it in a google doc?

Ok, I can work on moving the document to Google Docs.

message has been deleted

I have completed the first draft of the documentation. [For DVRT neo4j integration.] I still need to work on some of the details as work progresses.

Jumped on this party and added the following: https://github.com/linkedin/venice/pull/1880

Hi, I need to stop working on VeniceDB Neo4j integration for now. I can't get Neo4j JDBC class connection to work for direct integration without HTTP Server. I will update again if I found how to make it work in the future. Thanks- Kristy.