Enrico Olivelli
03/31/2023, 2:59 PMEnrico Olivelli
03/31/2023, 3:05 PMEnrico Olivelli
04/05/2023, 11:47 AMAndrey
04/06/2023, 9:15 PMSushant Mane
04/06/2023, 9:32 PMVeniceOnlineWriter
? to avoid confusion?Enrico Olivelli
04/12/2023, 1:12 PMEnrico Olivelli
04/14/2023, 10:24 AMAndrey
04/26/2023, 10:07 PMjava -jar /opt/venice/bin/venice-admin-tool-all.jar --query --url <http://venice-controller:5555> --cluster venice-cluster0 --store t1_n1_s1 --key name0
and getting
com.linkedin.venice.client.exceptions.VeniceClientException: Key Schema of store: t1_n1_s1 doesn't exist
error.
Did I miss some setup step?
The schema is there AFAICT:
[bash, -c, java -jar /opt/venice/bin/venice-admin-tool-all.jar --schemas --url <http://venice-controller:5555> --cluster venice-cluster0 --store t1_n1_s1]
StdOut:
[WARN] Running admin tool without SSL.
2023-04-26 21:52:48 INFO [ControllerClient] [main] Discovered leader controller <http://venice-controller:5555> from <http://venice-controller:5555>
{
"cluster" : "venice-cluster0",
"name" : "t1_n1_s1",
"error" : null,
"errorType" : null,
"id" : 1,
"derivedSchemaId" : -1,
"schemaStr" : "\"string\"",
"exceptionType" : null
}
2023-04-26 21:52:48 INFO [ControllerClient] [main] Discovered leader controller <http://venice-controller:5555> from <http://venice-controller:5555>
{
"cluster" : "venice-cluster0",
"name" : "t1_n1_s1",
"error" : null,
"errorType" : null,
"superSetSchemaId" : -1,
"schemas" : [ {
"id" : 1,
"derivedSchemaId" : -1,
"rmdValueSchemaId" : -1,
"schemaStr" : "{\"type\":\"record\",\"name\":\"Person\",\"namespace\":\"sink.PulsarVeniceSinkTest\",\"fields\":[{\"name\":\"age\",\"type\":\"int\"},{\"name\":\"name\",\"type\":[\"null\",\"string\"],\"default\":null}]}"
} ],
"exceptionType" : null
}
I set schemas using
java -jar /opt/venice/bin/venice-admin-tool-all.jar --new-store --url <http://venice-controller:5555> --cluster venice-cluster0 --store t1_n1_s1 --key-schema-file /tmp/key.asvc --value-schema-file /tmp/value.asvc
key.asvc is
{
"name": "key",
"type": "string"
}
Enrico Olivelli
04/27/2023, 8:22 AM./gradlew :services:venice-standalone:run
Felix GV
04/28/2023, 9:42 PMAndrey
04/28/2023, 10:09 PMThread.sleep(30000)
) I get no errors on write but I also don't get the data. After I added the sleep the data sent to the store shows up in get requests.
The store is in ONLINE state, I tried waiting like this and it passes the check imemdiately:
Awaitility.await().atMost(90, TimeUnit.SECONDS).untilAsserted(() -> {
ExecResult res = execByService(
"venice-client",
"bash",
"-c",
"java -jar " + jar + " --describe-store --url " + veniceControllerUrl
+ " --cluster " + clusterName
+ " --store " + storeName);
assertTrue(res.getStdout().contains("\"status\" : \"ONLINE\""));
});
I have a workaround for now but this brings two questions:
• how do I check for the store readiness properly?
• why producer.put() does not fail if the store is not ready?Felix GV
05/02/2023, 6:50 PMAndrey
05/02/2023, 9:14 PMThread.sleep(15000)
) in the test which you can remove to repro https://github.com/linkedin/venice/issues/387Andrey
05/03/2023, 11:11 PMfinal TransportClient transportClient;
if (discoveryUrl.isPresent()) {
this.controllerClient =
ControllerClient.discoverAndConstructControllerClient(storeName, discoveryUrl.get(), sslFactory, 1);
/**
* Verify that the latest {@link com.linkedin.venice.serialization.avro.AvroProtocolDefinition#KAFKA_MESSAGE_ENVELOPE}
* version in the code base is registered in Venice backend; if not, fail fast in start phase before start writing
* Kafka messages that Venice backend couldn't deserialize.
*/
if (verifyLatestProtocolPresent) {
<http://LOGGER.info|LOGGER.info>("Start verifying the latest protocols at runtime are valid in Venice backend.");
// Discover the D2 service name for the system store
// does not return anything useful in case of discoveryUrl
// e.g. cannot get venice router url
//D2ServiceDiscoveryResponse sysStoreDiscoveryResponse = (D2ServiceDiscoveryResponse) controllerRequestWithRetry(
// () -> ControllerClient.discoverCluster(discoveryUrl.get(), storeName, sslFactory, 1),
// 2);
String kafkaMessageEnvelopSchemaSysStore = AvroProtocolDefinition.KAFKA_MESSAGE_ENVELOPE.getSystemStoreName();
ClientConfig clientConfigForKafkaMessageEnvelopeSchemaReader =
ClientConfig.defaultGenericClientConfig(kafkaMessageEnvelopSchemaSysStore);
clientConfigForKafkaMessageEnvelopeSchemaReader.setVeniceURL("<http://venice-router:7777>");
SchemaReader kafkaMessageEnvelopeSchemaReader =
ClientFactory.getSchemaReader(clientConfigForKafkaMessageEnvelopeSchemaReader, null);
new SchemaPresenceChecker(kafkaMessageEnvelopeSchemaReader, AvroProtocolDefinition.KAFKA_MESSAGE_ENVELOPE)
.verifySchemaVersionPresentOrExit();
<http://LOGGER.info|LOGGER.info>("Successfully verified the latest protocols at runtime are valid in Venice backend.");
}
transportClient = new HttpTransportClient(discoveryUrl.get());
} else {
Is there a way to discover router url using controller discovery url or I have to add/pass the router url?Andrey
05/08/2023, 5:18 PMEnrico Olivelli
05/09/2023, 3:46 PMGradle suite > Gradle test > com.linkedin.davinci.kafka.consumer.KafkaStoreIngestionServiceWithTopicWiseTest > setUp FAILED
java.lang.IllegalStateException: Could not initialize plugin: interface org.mockito.plugins.MockMaker (alternate: null)
at org.mockito.internal.configuration.plugins.PluginLoader$1.invoke(PluginLoader.java:88)
at com.sun.proxy.$Proxy27.isTypeMockable(Unknown Source)
at org.mockito.internal.util.MockUtil.typeMockabilityOf(MockUtil.java:33)
at org.mockito.internal.util.MockCreationValidator.validateType(MockCreationValidator.java:22)
at org.mockito.internal.creation.MockSettingsImpl.validatedSettings(MockSettingsImpl.java:250)
at org.mockito.internal.creation.MockSettingsImpl.build(MockSettingsImpl.java:232)
at org.mockito.internal.MockitoCore.mock(MockitoCore.java:83)
at org.mockito.Mockito.mock(Mockito.java:1964)
at org.mockito.Mockito.mock(Mockito.java:1879)
at com.linkedin.davinci.kafka.consumer.KafkaStoreIngestionServiceTest.setUp(KafkaStoreIngestionServiceTest.java:88)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:59)
at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:458)
Enrico Olivelli
05/10/2023, 7:28 AMEnrico Olivelli
05/10/2023, 7:36 AMEnrico Olivelli
05/10/2023, 7:56 AM./gradlew :internal:venice-avro-compatibility-test:testAvro1_7
see the thread…
Felix GV
05/11/2023, 1:48 PMEnrico Olivelli
05/17/2023, 12:07 PMNicolò Boschi
05/17/2023, 1:28 PMEnrico Olivelli
05/18/2023, 12:06 PMChris Bartholomew
05/18/2023, 9:21 PMEnrico Olivelli
05/22/2023, 7:58 AMEnrico Olivelli
05/23/2023, 9:56 AMFelix GV
05/24/2023, 4:01 PMEnrico Olivelli
05/25/2023, 2:28 PMEnrico Olivelli
05/26/2023, 10:03 AMEnrico Olivelli
05/29/2023, 4:13 PM