thank you folks for merging this PR https://github.com/linkedin/venice/pull/293 That should be enough to deploy on k8s with my helm chart. Would you deploy the docker images of the current main branch on a public repo ?

@Felix GV I added some more ideas here https://github.com/linkedin/venice/issues/308#issuecomment-1492078392

Hello folks, I have sent out a patch to have a new package “venice-standalone” that runs all Venice components in a single JVM https://github.com/linkedin/venice/pull/332 The idea is taken from the “Pulsar Standalone” package, it makes it super-easy for users to try out the Service locally and/or to develop applications

Hello, I have a question about Venice Producer. I see activity around abstracting it out / creating an easier to use API: https://github.com/linkedin/venice/blob/main/clients/venice-producer/src/main/java/com/linkedin/venice/producer/VeniceProducer.java I assume someone is actively working on the implementation of this interface. Is there an estimated timeline for the implementation?

@Nisarg Thakkar - Can we consider renaming https://github.com/linkedin/venice/blob/main/clients/venice-producer/src/main/java/com/linkedin/venice/producer/VeniceProducer.java to

VeniceOnlineWriter

? to avoid confusion?

Quick fix for an annoying problem https://github.com/linkedin/venice/pull/348

Hello folks, I need help on this https://github.com/linkedin/venice/issues/202 Probably one approach would be to not rely anymore on DynamicAccessController but on AuthorizerService https://github.com/linkedin/venice/blob/63a7921da1c1814f9d856a6375f84ba392c01abb/i[…]n/java/com/linkedin/venice/authorization/AuthorizerService.java I tried to build some adapter between DynamicAccessController and AuthorizerService (an AuthorizerService implementation that forwards to the DynamicAccessController) but there are things about ACLs that cannot be easily adapted.

I have a noob question about querying data from Venice using admin tool. I have Venice store up and running locally and the process that supposedly populated it with some data. I am trying to query the store using admin tool to confirm that data is in fact there

java -jar /opt/venice/bin/venice-admin-tool-all.jar --query --url <http://venice-controller:5555> --cluster venice-cluster0 --store t1_n1_s1 --key name0

and getting

com.linkedin.venice.client.exceptions.VeniceClientException: Key Schema of store: t1_n1_s1 doesn't exist

error. Did I miss some setup step? The schema is there AFAICT:

[bash, -c, java -jar /opt/venice/bin/venice-admin-tool-all.jar --schemas --url <http://venice-controller:5555> --cluster venice-cluster0 --store t1_n1_s1]
        StdOut:
    [WARN] Running admin tool without SSL.
    2023-04-26 21:52:48 INFO [ControllerClient] [main] Discovered leader controller <http://venice-controller:5555> from <http://venice-controller:5555>
    {
      "cluster" : "venice-cluster0",
      "name" : "t1_n1_s1",
      "error" : null,
      "errorType" : null,
      "id" : 1,
      "derivedSchemaId" : -1,
      "schemaStr" : "\"string\"",
      "exceptionType" : null
    }
    2023-04-26 21:52:48 INFO [ControllerClient] [main] Discovered leader controller <http://venice-controller:5555> from <http://venice-controller:5555>
    {
      "cluster" : "venice-cluster0",
      "name" : "t1_n1_s1",
      "error" : null,
      "errorType" : null,
      "superSetSchemaId" : -1,
      "schemas" : [ {
        "id" : 1,
        "derivedSchemaId" : -1,
        "rmdValueSchemaId" : -1,
        "schemaStr" : "{\"type\":\"record\",\"name\":\"Person\",\"namespace\":\"sink.PulsarVeniceSinkTest\",\"fields\":[{\"name\":\"age\",\"type\":\"int\"},{\"name\":\"name\",\"type\":[\"null\",\"string\"],\"default\":null}]}"
      } ],
      "exceptionType" : null
    }

I set schemas using

java -jar /opt/venice/bin/venice-admin-tool-all.jar --new-store --url <http://venice-controller:5555> --cluster venice-cluster0 --store t1_n1_s1 --key-schema-file /tmp/key.asvc --value-schema-file /tmp/value.asvc

key.asvc is

{
  "name": "key",
  "type": "string"
}

Another small improvement to “Venice Standalone” https://github.com/linkedin/venice/pull/382 With this patch you will be able to run your local changes locally with

./gradlew :services:venice-standalone:run

FYI, all contributors and would be contributors welcome!

Following up on https://venicedb.slack.com/archives/C03SLQWRSLF/p1682546829621559 Last problem I had was that data didn't appear in Venice. I am using VeniceSystemProducer to send the data and so far there are two things that I found out: 1. producer autoflushes data eventually but I had to add explicit flush to make the result more predictable a. producer.flush() hangs if it happens to be called at the same time from different threads. It waits indefinitely in kafka producer's code. I worked around that, possibly this is a known (just not to me) kafka producer behavior. b. 2. The store is not immediately ready after startup and initialization More details on p2: testcontainers starts venice components then the test runs "--new-store"/"--update-store <quotas>"/"--empty-push" using admin tool. This should be enough to start submitting the data. What I found out is that unless I wait (like

Thread.sleep(30000)

) I get no errors on write but I also don't get the data. After I added the sleep the data sent to the store shows up in get requests. The store is in ONLINE state, I tried waiting like this and it passes the check imemdiately:

Awaitility.await().atMost(90, TimeUnit.SECONDS).untilAsserted(() -> {
      ExecResult res = execByService(
              "venice-client",
              "bash",
              "-c",
              "java -jar " + jar + " --describe-store --url " + veniceControllerUrl
                      + " --cluster " + clusterName
                      + " --store " + storeName);

      assertTrue(res.getStdout().contains("\"status\" : \"ONLINE\""));
    });

I have a workaround for now but this brings two questions: • how do I check for the store readiness properly? • why producer.put() does not fail if the store is not ready?

Interesting rant / lessons learned on the Pulsar Improvement Proposal (PIP) process (itself framed as a PIP 😄 …). https://github.com/apache/pulsar/issues/20207

Pulsar Sink PR: https://github.com/linkedin/venice/pull/388 It is draft because I cannot figure out how to trigger new CI workflow to actually run the test 😞 I am not sure if that dependent on project setup or not; IIRC pulsar's GH CI ran new workflows but that might be controlled by some setting. rtfm-ing. The test runs (locally) and one can follow steps in the CI wf to build docker images and run it. There is one sleep (

Thread.sleep(15000)

) in the test which you can remove to repro https://github.com/linkedin/venice/issues/387

I am trying to address this PR comment: https://github.com/linkedin/venice/pull/388/files#r1184075702 and make verifyLatestProtocolPresent work the discoveryUrl provided. It kind of works like

final TransportClient transportClient;
    if (discoveryUrl.isPresent()) {
      this.controllerClient =
          ControllerClient.discoverAndConstructControllerClient(storeName, discoveryUrl.get(), sslFactory, 1);

      /**
       * Verify that the latest {@link com.linkedin.venice.serialization.avro.AvroProtocolDefinition#KAFKA_MESSAGE_ENVELOPE}
       * version in the code base is registered in Venice backend; if not, fail fast in start phase before start writing
       * Kafka messages that Venice backend couldn't deserialize.
       */
      if (verifyLatestProtocolPresent) {
        <http://LOGGER.info|LOGGER.info>("Start verifying the latest protocols at runtime are valid in Venice backend.");
        // Discover the D2 service name for the system store
        // does not return anything useful in case of discoveryUrl
        // e.g. cannot get venice router url
        
        //D2ServiceDiscoveryResponse sysStoreDiscoveryResponse = (D2ServiceDiscoveryResponse) controllerRequestWithRetry(
        //        () -> ControllerClient.discoverCluster(discoveryUrl.get(), storeName, sslFactory, 1),
        //        2);

        String kafkaMessageEnvelopSchemaSysStore = AvroProtocolDefinition.KAFKA_MESSAGE_ENVELOPE.getSystemStoreName();
        ClientConfig clientConfigForKafkaMessageEnvelopeSchemaReader =
                ClientConfig.defaultGenericClientConfig(kafkaMessageEnvelopSchemaSysStore);
        clientConfigForKafkaMessageEnvelopeSchemaReader.setVeniceURL("<http://venice-router:7777>");

        SchemaReader kafkaMessageEnvelopeSchemaReader =
                ClientFactory.getSchemaReader(clientConfigForKafkaMessageEnvelopeSchemaReader, null);
        new SchemaPresenceChecker(kafkaMessageEnvelopeSchemaReader, AvroProtocolDefinition.KAFKA_MESSAGE_ENVELOPE)
                .verifySchemaVersionPresentOrExit();
        <http://LOGGER.info|LOGGER.info>("Successfully verified the latest protocols at runtime are valid in Venice backend.");
      }

      transportClient = new HttpTransportClient(discoveryUrl.get());
    } else {

Is there a way to discover router url using controller discovery url or I have to add/pass the router url?

https://github.com/linkedin/venice/pull/388 addressed all previous code review comments. I'd appreciate another pass of the review.

Hello, I am using IntelliJ, but I am not able to run some integration tests. For instance I have this error. Is there any suggestion about how to configure the IDE to run such tests ?

Gradle suite > Gradle test > com.linkedin.davinci.kafka.consumer.KafkaStoreIngestionServiceWithTopicWiseTest > setUp FAILED

java.lang.IllegalStateException: Could not initialize plugin: interface org.mockito.plugins.MockMaker (alternate: null)

at org.mockito.internal.configuration.plugins.PluginLoader$1.invoke(PluginLoader.java:88)

at com.sun.proxy.$Proxy27.isTypeMockable(Unknown Source)

at org.mockito.internal.util.MockUtil.typeMockabilityOf(MockUtil.java:33)

at org.mockito.internal.util.MockCreationValidator.validateType(MockCreationValidator.java:22)

at org.mockito.internal.creation.MockSettingsImpl.validatedSettings(MockSettingsImpl.java:250)

at org.mockito.internal.creation.MockSettingsImpl.build(MockSettingsImpl.java:232)

at org.mockito.internal.MockitoCore.mock(MockitoCore.java:83)

at org.mockito.Mockito.mock(Mockito.java:1964)

at org.mockito.Mockito.mock(Mockito.java:1879)

at com.linkedin.davinci.kafka.consumer.KafkaStoreIngestionServiceTest.setUp(KafkaStoreIngestionServiceTest.java:88)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:566)

at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)

at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:59)

at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:458)

Here we go with VIP-1 😁 https://github.com/linkedin/venice/pull/411/files

How do you run performance tests at LI for VeniceDB ? (cc @Dave Fisher)

I have this error while running some tests locally, on the “main” branch (@Zac Policzer 😉 )

./gradlew :internal:venice-avro-compatibility-test:testAvro1_7

see the thread…

I’m getting lots of 500s while trying to access GH… anybody else seeing that?

Next step for “venice standalone” https://github.com/linkedin/venice/pull/426

Hello, I’m trying to setup TLS on the router service with a real tls certificate. Looking at the code I see that the SSLFactory instance is not customizable via config properties but it sets the localSslFactory (clearly not usable for prod env) https://github.com/linkedin/venice/blob/d689f173e461b51e531e41bcfb685057637995cb/s[…]uter/src/main/java/com/linkedin/venice/router/RouterServer.java Could you give me some hints on how to configure a tls certificate there? Thanks

I am going to prepare a PR for the initial step of the implementation of VIP-1. I have one philosophical question. Dimension one: only APIs ? 1. send only on PR with the API of the AuthenticationService and the lifecycle (Controller, Router, Server) 2. add also the JWT implementation ◦ This requires many client side changes, on the ControllerClient and on the reader Client and to the CLI tools Point: without the JWT implementation it is hard to get the taste of the change , but the JWT impl adds lot of code Dimension two: Add lifecycle for the AuthorizerService 1. Send also a few minimal changes with the lifycycle of the AuthorizerService + a simple AuthorizerService that requires that there is always a Principal mapped to the request 2. Do not send anything about AuthorizerService Point: without a simple AuthorizerService that requires Authentication is hard to write meaningful end-to-end tests In my fork I have been pushing this set of changes to a quite advanced stage and now we are able to enforce JWT authentication on all the endpoints/services (controller, server, router) and so it is supported on clients. But the story is quite long

We are starting to look at performance characterization for Venice. We have some in-house test tools and are adding support for Venice so we can run some tests. So we can know what to expect, do you have any numbers you can share for specific workloads? Also, how much difference do you see between larger-than-RAM datasets and all-in-RAM datasets?

Hello, I would like to try out the new OnlineProducer API. It seems it doesn’t work out of the box as it requires to set some replication strategy on the store (see the error in the thread >>>). How do I create a store capable of accepting writes? Can you please share the CLI commands set it p ? I haven’t found an end-to-end test as reference

Thanks to the Producer API is has been quite easy to add full (basic) support for Writing and Reading To/From VeniceDB on NoSQLBench 🥳 https://github.com/nosqlbench/nosqlbench/pull/1286 This is one of the tools we have ad DataStax to run performance testing. The support for VeniceDB is still very limited, but we are going to enhance it, we want to demonstrate the power of VeniceDB

Publicly searchable archive of all messages on this slack: http://linen.venicedb.org/

What about creating a channel about performance testing ?

I have ported all the changes that we have in our fork regarding SASL to Kafka https://github.com/linkedin/venice/pull/452 After this patch the main contribution to backport is about JWT Authentication

It would be great if we could start cutting public releases like the alpha internal releases we have in our fork: • basic artifacts and “release” on GH (like this) • docker images • all the jars on a public Maven repository ◦ LinkedIn OSS works well initially ◦ Maven central in the future ◦ (we are deploying them to Datastax OSS now) This is very important for users to start adopting Venice outside of LinkedIn