https://venicedb.org logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
a
full log of wuey:
Copy code
2023-04-26 14:52:48 INFO [PulsarVeniceSinkTest] [Test worker] Querying Venice to check messages
    2023-04-26 14:52:49 INFO [DockerComposeContainer] [docker-java-stream-2089144938] [/cnxsl1dvvgee_zookeeper_1] STDOUT: 2023-04-26T21:52:49,236+0000 [NIOWorkerThread-6] INFO  org.apache.zookeeper.server.NIOServerCnxn - Processing ruok command from /127.0.0.1:43564
    2023-04-26 14:52:50 INFO [DockerComposeContainer] [docker-java-stream-851077002] [/cnxsl1dvvgee_venice-controller_1] STDOUT: 2023-04-26 21:52:50 INFO [AdminSparkServer] [qtp319130257-128] [AUDIT] GET <http://venice-controller:5555/key_schema/t1_n1_s1> {}
    2023-04-26 14:52:50 INFO [DockerComposeContainer] [docker-java-stream-851077002] [/cnxsl1dvvgee_venice-controller_1] STDOUT: 2023-04-26 21:52:50 INFO [AdminSparkServer] [qtp319130257-128] [AUDIT] SUCCESS GET <http://venice-controller:5555/key_schema/t1_n1_s1> {}
    2023-04-26 14:52:50 INFO [DockerComposeContainer] [docker-java-stream-851077002] [/cnxsl1dvvgee_venice-controller_1] STDOUT: 2023-04-26 21:52:50 INFO [MatcherFilter] [qtp319130257-128] The requested route [/key_schema/t1_n1_s1] has not been mapped in Spark for Accept: [null]
    2023-04-26 14:52:50 INFO [DockerComposeContainer] [docker-java-stream-851077002] [/cnxsl1dvvgee_venice-controller_1] STDOUT: 2023-04-26 21:52:50 INFO [AdminSparkServer] [qtp319130257-130] [AUDIT] GET <http://venice-controller:5555/key_schema/t1_n1_s1> {}
    2023-04-26 14:52:50 INFO [DockerComposeContainer] [docker-java-stream-851077002] [/cnxsl1dvvgee_venice-controller_1] STDOUT: 2023-04-26 21:52:50 INFO [AdminSparkServer] [qtp319130257-130] [AUDIT] SUCCESS GET <http://venice-controller:5555/key_schema/t1_n1_s1> {}
    2023-04-26 14:52:50 INFO [DockerComposeContainer] [docker-java-stream-851077002] [/cnxsl1dvvgee_venice-controller_1] STDOUT: 2023-04-26 21:52:50 INFO [MatcherFilter] [qtp319130257-130] The requested route [/key_schema/t1_n1_s1] has not been mapped in Spark for Accept: [null]
    2023-04-26 14:52:50 INFO [DockerComposeContainer] [docker-java-stream-851077002] [/cnxsl1dvvgee_venice-controller_1] STDOUT: 2023-04-26 21:52:50 INFO [AdminSparkServer] [qtp319130257-128] [AUDIT] GET <http://venice-controller:5555/key_schema/t1_n1_s1> {}
    2023-04-26 14:52:50 INFO [DockerComposeContainer] [docker-java-stream-851077002] [/cnxsl1dvvgee_venice-controller_1] STDOUT: 2023-04-26 21:52:50 INFO [AdminSparkServer] [qtp319130257-128] [AUDIT] SUCCESS GET <http://venice-controller:5555/key_schema/t1_n1_s1> {}
    2023-04-26 14:52:50 INFO [DockerComposeContainer] [docker-java-stream-851077002] [/cnxsl1dvvgee_venice-controller_1] STDOUT: 2023-04-26 21:52:50 INFO [MatcherFilter] [qtp319130257-128] The requested route [/key_schema/t1_n1_s1] has not been mapped in Spark for Accept: [null]
    2023-04-26 14:52:50 INFO [PulsarVeniceSinkTest] [Test worker]       venice-client Exec
        [bash, -c, java -jar /opt/venice/bin/venice-admin-tool-all.jar --query --url <http://venice-controller:5555> --cluster venice-cluster0 --store t1_n1_s1 --key name0]
        StdOut:
    [WARN] Running admin tool without SSL.
    WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
    2023-04-26 21:52:49 INFO [AbstractAvroStoreClient] [main] Detected: AVRO_1_9 on the classpath.
    com/linkedin/avro/fastserde/generated/deserialization/AVRO_1_9/MultiGetResponseRecordV1_GenericDeserializer_4281647829766090175_4281647829766090175.java
    2023-04-26 21:52:50 INFO [FastSerdeBase] [main] Starting compilation for the generated source file: /tmp/generated16049280895217718584/com/linkedin/avro/fastserde/generated/deserialization/AVRO_1_9/MultiGetResponseRecordV1_GenericDeserializer_4281647829766090175_4281647829766090175.java
    2023-04-26 21:52:50 INFO [FastSerdeBase] [main] Successfully compiled class MultiGetResponseRecordV1_GenericDeserializer_4281647829766090175_4281647829766090175 defined at source file: /tmp/generated16049280895217718584/com/linkedin/avro/fastserde/generated/deserialization/AVRO_1_9/MultiGetResponseRecordV1_GenericDeserializer_4281647829766090175_4281647829766090175.java
    2023-04-26 21:52:50 INFO [FastSerdeCache] [main] Generated classes dir: /tmp/generated16049280895217718584 and generation of generic FastDeserializer is done for writer schema of type: com.linkedin.venice.read.protocol.response.MultiGetResponseRecordV1 with fingerprint: 4281647829766090175 and reader schema of type: com.linkedin.venice.read.protocol.response.MultiGetResponseRecordV1 with fingerprint: 4281647829766090175
    2023-04-26 21:52:50 WARN [RouterBackedSchemaReader] [main] Requested schema doesn't exist for request path: key_schema/t1_n1_s1
    2023-04-26 21:52:50 INFO [AbstractAvroStoreClient] [main] Got error when trying to warm up client during start phase for store: t1_n1_s1, and will kick off an async warm-up:Failed to initializing Venice Client for store: t1_n1_s1
    2023-04-26 21:52:50 WARN [RouterBackedSchemaReader] [main] Requested schema doesn't exist for request path: key_schema/t1_n1_s1
    2023-04-26 21:52:50 WARN [RouterBackedSchemaReader] [ForkJoinPool.commonPool-worker-3] Requested schema doesn't exist for request path: key_schema/t1_n1_s1
    {
      "error" : "Key Schema of store: t1_n1_s1 doesn't exist"
    }
    tStdErr:
    Exception in thread "main" com.linkedin.venice.client.exceptions.VeniceClientException: Key Schema of store: t1_n1_s1 doesn't exist
        at com.linkedin.venice.client.schema.RouterBackedSchemaReader.getKeySchema(RouterBackedSchemaReader.java:87)
        at com.linkedin.venice.client.store.AbstractAvroStoreClient.getKeySchema(AbstractAvroStoreClient.java:754)
        at com.linkedin.venice.client.store.QueryTool.queryStoreForKey(QueryTool.java:81)
        at com.linkedin.venice.AdminTool.queryStoreForKey(AdminTool.java:600)
        at com.linkedin.venice.AdminTool.main(AdminTool.java:333)
describe store output:
Copy code
[bash, -c, java -jar /opt/venice/bin/venice-admin-tool-all.jar --describe-store --url <http://venice-controller:5555> --cluster venice-cluster0 --store t1_n1_s1]
        StdOut:
    [WARN] Running admin tool without SSL.
    2023-04-26 21:52:46 INFO [ControllerClient] [main] Discovered leader controller <http://venice-controller:5555> from <http://venice-controller:5555>
    WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
    {
      "cluster" : "venice-cluster0",
      "name" : "t1_n1_s1",
      "error" : null,
      "errorType" : null,
      "store" : {
        "name" : "t1_n1_s1",
        "owner" : "",
        "currentVersion" : 1,
        "coloToCurrentVersions" : null,
        "reservedVersion" : 0,
        "partitionCount" : 0,
        "lowWatermark" : 0,
        "enableStoreWrites" : true,
        "enableStoreReads" : true,
        "versions" : [ {
          "storeName" : "t1_n1_s1",
          "number" : 1,
          "createdTime" : 1682545929071,
          "pushJobId" : "init",
          "partitionCount" : 1,
          "partitionerConfig" : {
            "partitionerClass" : "com.linkedin.venice.partitioner.DefaultVenicePartitioner",
            "partitionerParams" : { },
            "amplificationFactor" : 1
          },
          "status" : "ONLINE",
          "activeActiveReplicationEnabled" : false,
          "nativeReplicationSourceFabric" : "",
          "replicationFactor" : 1,
          "pushStreamSourceAddress" : "",
          "nativeReplicationEnabled" : false,
          "incrementalPushEnabled" : true,
          "rmdChunkingEnabled" : false,
          "chunkingEnabled" : false,
          "compressionStrategy" : "NO_OP",
          "hybridStoreConfig" : {
            "rewindTimeInSeconds" : 86400,
            "offsetLagThresholdToGoOnline" : 1000,
            "producerTimestampLagThresholdToGoOnlineInSeconds" : -1,
            "dataReplicationPolicy" : "NONE",
            "bufferReplayPolicy" : "REWIND_FROM_EOP"
          },
          "pushType" : "BATCH",
          "versionSwapDeferred" : false,
          "minActiveReplicas" : 0,
          "useVersionLevelIncrementalPushEnabled" : true,
          "useVersionLevelHybridConfig" : true,
          "dataRecoveryVersionConfig" : null,
          "leaderFollowerModelEnabled" : true,
          "timestampMetadataVersionId" : 1,
          "viewConfigs" : { }
        } ],
        "storageQuotaInByte" : -1,
        "hybridStoreOverheadBypass" : false,
        "readQuotaInCU" : 1000000,
        "hybridStoreConfig" : {
          "rewindTimeInSeconds" : 86400,
          "offsetLagThresholdToGoOnline" : 1000,
          "producerTimestampLagThresholdToGoOnlineInSeconds" : -1,
          "dataReplicationPolicy" : "NONE",
          "bufferReplayPolicy" : "REWIND_FROM_EOP"
        },
        "accessControlled" : true,
        "chunkingEnabled" : false,
        "rmdChunkingEnabled" : false,
        "singleGetRouterCacheEnabled" : false,
        "batchGetRouterCacheEnabled" : false,
        "batchGetLimit" : -1,
        "largestUsedVersionNumber" : 1,
        "incrementalPushEnabled" : true,
        "compressionStrategy" : "NO_OP",
        "clientDecompressionEnabled" : true,
        "numVersionsToPreserve" : 0,
        "migrating" : false,
        "writeComputationEnabled" : false,
        "replicationMetadataVersionId" : -1,
        "readComputationEnabled" : false,
        "bootstrapToOnlineTimeoutInHours" : 24,
        "leaderFollowerModelEnabled" : true,
        "nativeReplicationEnabled" : false,
        "pushStreamSourceAddress" : "",
        "backupStrategy" : "DELETE_ON_NEW_PUSH_START",
        "schemaAutoRegisterFromPushJobEnabled" : false,
        "superSetSchemaAutoGenerationForReadComputeEnabled" : false,
        "latestSuperSetValueSchemaId" : -1,
        "hybridStoreDiskQuotaEnabled" : false,
        "etlStoreConfig" : {
          "etledUserProxyAccount" : "",
          "regularVersionETLEnabled" : false,
          "futureVersionETLEnabled" : false
        },
        "partitionerConfig" : {
          "partitionerClass" : "com.linkedin.venice.partitioner.DefaultVenicePartitioner",
          "partitionerParams" : { },
          "amplificationFactor" : 1
        },
        "backupVersionRetentionMs" : 604800000,
        "replicationFactor" : 1,
        "migrationDuplicateStore" : false,
        "nativeReplicationSourceFabric" : "",
        "storeMetadataSystemStoreEnabled" : false,
        "storeMetaSystemStoreEnabled" : false,
        "daVinciPushStatusStoreEnabled" : false,
        "activeActiveReplicationEnabled" : false,
        "kafkaBrokerUrl" : "kafka:9092",
        "viewConfigs" : { }
      },
      "exceptionType" : null
    }
Querying schema:
Copy code
2023-04-26 14:52:48 INFO [DockerComposeContainer] [docker-java-stream-851077002] [/cnxsl1dvvgee_venice-controller_1] STDOUT: 2023-04-26 21:52:48 INFO [AdminSparkServer] [qtp319130257-127] [AUDIT] GET <http://venice-controller:5555/leader_controller> {cluster_name=venice-cluster0, hostna
me=localhost}
    2023-04-26 14:52:48 INFO [DockerComposeContainer] [docker-java-stream-851077002] [/cnxsl1dvvgee_venice-controller_1] STDOUT: 2023-04-26 21:52:48 INFO [AdminSparkServer] [qtp319130257-127] [AUDIT] SUCCESS GET <http://venice-controller:5555/leader_controller> {cluster_name=venice-cluster0
, hostname=localhost}
    2023-04-26 14:52:48 INFO [DockerComposeContainer] [docker-java-stream-851077002] [/cnxsl1dvvgee_venice-controller_1] STDOUT: 2023-04-26 21:52:48 INFO [AdminSparkServer] [qtp319130257-128] [AUDIT] GET <http://venice-controller:5555/get_key_schema> {cluster_name=venice-cluster0, hostname=
localhost, store_name=t1_n1_s1}
    2023-04-26 14:52:48 INFO [DockerComposeContainer] [docker-java-stream-851077002] [/cnxsl1dvvgee_venice-controller_1] STDOUT: 2023-04-26 21:52:48 INFO [AdminSparkServer] [qtp319130257-128] [AUDIT] SUCCESS GET <http://venice-controller:5555/get_key_schema> {cluster_name=venice-cluster0, h
ostname=localhost, store_name=t1_n1_s1}
    2023-04-26 14:52:48 INFO [DockerComposeContainer] [docker-java-stream-851077002] [/cnxsl1dvvgee_venice-controller_1] STDOUT: 2023-04-26 21:52:48 INFO [AdminSparkServer] [qtp319130257-124] [AUDIT] GET <http://venice-controller:5555/leader_controller> {cluster_name=venice-cluster0, hostna
me=localhost}
    2023-04-26 14:52:48 INFO [DockerComposeContainer] [docker-java-stream-851077002] [/cnxsl1dvvgee_venice-controller_1] STDOUT: 2023-04-26 21:52:48 INFO [AdminSparkServer] [qtp319130257-124] [AUDIT] SUCCESS GET <http://venice-controller:5555/leader_controller> {cluster_name=venice-cluster0
, hostname=localhost}
    2023-04-26 14:52:48 INFO [DockerComposeContainer] [docker-java-stream-851077002] [/cnxsl1dvvgee_venice-controller_1] STDOUT: 2023-04-26 21:52:48 INFO [AdminSparkServer] [qtp319130257-127] [AUDIT] GET <http://venice-controller:5555/get_all_value_schema> {cluster_name=venice-cluster0, hos
tname=localhost, store_name=t1_n1_s1}
    2023-04-26 14:52:48 INFO [DockerComposeContainer] [docker-java-stream-851077002] [/cnxsl1dvvgee_venice-controller_1] STDOUT: 2023-04-26 21:52:48 INFO [AdminSparkServer] [qtp319130257-127] [AUDIT] SUCCESS GET <http://venice-controller:5555/get_all_value_schema> {cluster_name=venice-clust
er0, hostname=localhost, store_name=t1_n1_s1}
    2023-04-26 14:52:48 INFO [PulsarVeniceSinkTest] [Test worker]       venice-client Exec
        [bash, -c, java -jar /opt/venice/bin/venice-admin-tool-all.jar --schemas --url <http://venice-controller:5555> --cluster venice-cluster0 --store t1_n1_s1]
        StdOut:
    [WARN] Running admin tool without SSL.
    2023-04-26 21:52:48 INFO [ControllerClient] [main] Discovered leader controller <http://venice-controller:5555> from <http://venice-controller:5555>
    {
      "cluster" : "venice-cluster0",
      "name" : "t1_n1_s1",
      "error" : null,
      "errorType" : null,
      "id" : 1,
      "derivedSchemaId" : -1,
      "schemaStr" : "\"string\"",
      "exceptionType" : null
    }
    2023-04-26 21:52:48 INFO [ControllerClient] [main] Discovered leader controller <http://venice-controller:5555> from <http://venice-controller:5555>
    {
      "cluster" : "venice-cluster0",
      "name" : "t1_n1_s1",
      "error" : null,
      "errorType" : null,
      "superSetSchemaId" : -1,
      "schemas" : [ {
        "id" : 1,
        "derivedSchemaId" : -1,
        "rmdValueSchemaId" : -1,
        "schemaStr" : "{\"type\":\"record\",\"name\":\"Person\",\"namespace\":\"sink.PulsarVeniceSinkTest\",\"fields\":[{\"name\":\"age\",\"type\":\"int\"},{\"name\":\"name\",\"type\":[\"null\",\"string\"],\"default\":null}]}"
      } ],
      "exceptionType" : null
    }
z
Hrm. Lets try an alternative approach (just to verify things). There should be a bash script for querying the store which hits the server instead of the controller. Lets try and use that and compare it's results. From the repository root it should be at
Copy code
clients/venice-thin-client/bin/query_key.sh
(I mostly suggest this as the query script is what we usually give to users because we don't want to give them the admin tool)
x
I guess there might be a path mismatch in the admin tool when querying. From the log above, 
get_key_schema
returns a response but 
key_schema/t1_n1_s1
returns 
{}
z
ah
Copy code
[ForkJoinPool.commonPool-worker-3] Requested schema doesn't exist for request path: key_schema/t1_n1_s1
yeah that is sketchy
The query script above should unblock you, but should patch this
a
the script does not seem to be packed in the docker image 😞 I'll figure this out
z
huh, interesting
@Manoj Nagarajan @Sushant Mane? Why didn't we package the query script?
a
fwiw I built Venice images locally (but I didn't change the Dockerfile)
x
I think that script contains the Linkedin internal artifactory path so it’s not included (maybe)
s
Can you use thin client instead of admin-tool?
x
@Andrey Can you please also file a Github issue so we can track and fix this?
a
Will do
Copy code
root@venice-client:/opt/venice# bin/query_key.sh
Usage: bin/query_key.sh <fabric> <store_name> <key_string> [is_vson_store]
Do not include any space in key_string, and replace all " by \".
What do I use as a fabric parameter?
s
if this is multi-region setup then the region name
z
Hrm. Actually, reading this script, not sure this will work
it's using curli
🤔 1
s
@Andrey - can you try this?
Copy code
java -jar /opt/venice/bin/venice-thin-client-all.jar $storeName $key $router false ""  2>/dev/null
👍 1
try this if you've router address you don't need colo name
a
Copy code
root@venice-client:/opt/venice# bin/query_key.sh <http://venice-controller:5555> t1_n1_s1 name0
grep: bin/../config/sslconfig.properties: No such file or directory
ERROR: bin/../config/sslconfig.properties does not contain keystore.password
I am running this from venice-client, should I use some other node?
z
ah, not controller
There should be a router endpoint
a
It is looking for sslconfig.properties - i don't see these on any venice node
s
pass 
""
empty string
a
Opted out to this:
Copy code
root@venice-client:/opt/venice# java -jar /opt/venice/bin/venice-thin-client-all.jar t1_n1_s1 name0 <http://venice-controller:5555> false ""

ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See <http://www.slf4j.org/codes.html#StaticLoggerBinder> for further details.
com/linkedin/avro/fastserde/generated/deserialization/AVRO_1_9/MultiGetResponseRecordV1_GenericDeserializer_4281647829766090175_4281647829766090175.java
Exception in thread "main" com.linkedin.venice.client.exceptions.VeniceClientException: Key Schema of store: t1_n1_s1 doesn't exist
	at com.linkedin.venice.client.schema.RouterBackedSchemaReader.getKeySchema(RouterBackedSchemaReader.java:87)
	at com.linkedin.venice.client.store.AbstractAvroStoreClient.getKeySchema(AbstractAvroStoreClient.java:754)
	at com.linkedin.venice.client.store.QueryTool.queryStoreForKey(QueryTool.java:81)
	at com.linkedin.venice.client.store.QueryTool.main(QueryTool.java:49)
s
endpoint should be of router
iirc it's 
venice-router:7777
?
a
ah
Copy code
root@venice-client:/opt/venice# java -jar /opt/venice/bin/venice-thin-client-all.jar t1_n1_s1 name0 <http://venice-router:7777> false ""

ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See <http://www.slf4j.org/codes.html#StaticLoggerBinder> for further details.
com/linkedin/avro/fastserde/generated/deserialization/AVRO_1_9/MultiGetResponseRecordV1_GenericDeserializer_4281647829766090175_4281647829766090175.java
Key string parsed successfully. About to make the query.
key-class=java.lang.String
value-class=null
request-path=storage/t1_n1_s1/Cm5hbWUw?f=b64
key=name0
value=null
🙌 2
z
I hope null was the answer you wanted 😅
😂 1
a
now why
Copy code
value-class=null
and value=null
not what I expected 😉
😅 1
Value avro is set to
Copy code
"schemaStr" : "{\"type\":\"record\",\"name\":\"Person\",\"namespace\":\"sink.PulsarVeniceSinkTest\",\"fields\":[{\"name\":\"age\",\"type\":\"int\"},{\"name\":\"name\",\"type\":[\"null\",\"string\"],\"default\":null}]}"
      } ]
see 
venice-admin-tool-all.jar --schemas
output above I'd expect it in value-class
Null value can be the tool issue or the sink issue, tbd
x
Have you ingested any data yet? If the value is null, the value-class is 
null
as well IIRC.
a
I hoped so 😉 The sink was supposed to push some data, similar to samza client
But this is progress already, really appreciate your help
z
value-null is the correct behavior when we get a null key in Venice
✅ 1
(Just verified on a store)
a
is there a way to get list of keys from the store?
or count of keys
z
So, there are stats emitted by the server that one can use to deduce this.
But there isn't an api for listing all keys or scanning everything unless your key schema is specified to as to make available a partial key get (and have everything under that top level hierarchical key)
x
^ A scan on the kakfa topic (version topic) might work?
z
That would work
But, gotta rep the interface
(and since Andrey is gonna be using Pulsar at some point)
a
let me experiment, thank you for the ideas
There are messages in the kafka topic (sorry, I gave up on attempts to setup Avro serializer there when it started asking for schema registry url in some config). I sent 10, there are 10 (max message 11 makes the tool block/wait)
Copy code
root@kafka:/opt/kafka# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic t1_n1_s1_v1 --max-messages 10

�qڡG㪋�;��⯘��a
              "JDK major version1path/opt/venichost"venice-controllerpid7versionN/useroot"venice-controller

�qڡG㪋�;�������a
                "JDK major version1path/opt/venichost"venice-controllerpid7versionN/useroot"venice-controller

�qڡG㪋�;��Ʊ���a
               "JDK major version1path/opt/venichost"venice-controllerpid7versionN/useroot ��nz@>͛3L[?�4�"venice-controller

}9'CGL��b��o���a
                 "JDK major version1path/opt/venichost"venice-controllerpid7versionN/useroot"venice-controller

}9'CGL��b��o���a
                 "JDK major version1path/opt/venichost"venice-controllerpid7versionN/useroot"venice-controller

}9'CGL��b��o���a
                 "JDK major version1path/opt/venichost"venice-controllerpid7versionN/useroot"venice-controller

}9'CGL��b��o���a
                 "JDK major version1path/opt/venichost"venice-controllerpid7versionN/useroot ?I��m�;+�E]�m
                                                                                                          "venice-controller

;���RB�7/�y{�˘��a
                 "JDK major version1path/opt/venichost"venice-controllerpid7versionN/useroot"venice-controller

;���RB�7/�y{�ۘ��a
                "JDK major version1path/opt/venichost"venice-controllerpid7versionN/userootkafka:9092t1_n1_s1_rt����a"venice-controller

;���RB�7/�y{�ۘ��a
                "JDK major version1path/opt/venichost"venice-controllerpid7versionN/useroot ���*y�	�-�e�"venice-controller
Processed a total of 10 messages
s
Quick note: you can use admin tool to dump kafka topic
a
any chance that 
java -jar /opt/venice/bin/venice-thin-client-all.jar
does not process value / does not return it?
Copy code
root@venice-client:/opt/venice# java -jar /opt/venice/bin/venice-admin-tool-all.jar --dump-kafka-topic --kafka-bootstrap-servers kafka:9092  --kafka-topic-name t1_n1_s1_v1 --url <http://venice-controller:5555> --cluster venice-cluster0
[WARN] Running admin tool without SSL.
2023-04-27 00:15:11 INFO [ControllerClient] [main] Discovered leader controller <http://venice-controller:5555> from <http://venice-controller:5555>
2023-04-27 00:15:11 INFO [ControllerClient] [main] Discovered leader controller <http://venice-controller:5555> from <http://venice-controller:5555>
2023-04-27 00:15:11 INFO [KafkaTopicDumper] [main] Found 1 value schemas for store t1_n1_s1
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2023-04-27 00:15:11 INFO [ConsumerConfig] [main] ConsumerConfig values:
	allow.auto.create.topics = false
	<http://auto.commit.interval.ms|auto.commit.interval.ms> = 5000
	auto.offset.reset = latest
	bootstrap.servers = [kafka:9092]
	check.crcs = true
	client.dns.lookup = use_all_dns_ips
	client.id =
	client.rack =
	<http://connections.max.idle.ms|connections.max.idle.ms> = 540000
	<http://default.api.timeout.ms|default.api.timeout.ms> = 60000
	enable.auto.commit = true
	enable.shallow.iterator = false
	exclude.internal.topics = true
	fetch.max.bytes = 52428800
	<http://fetch.max.wait.ms|fetch.max.wait.ms> = 500
	fetch.min.bytes = 1
	group.id = null
	group.instance.id = null
	<http://heartbeat.interval.ms|heartbeat.interval.ms> = 3000
	interceptor.classes = []
	internal.leave.group.on.close = true
	isolation.level = read_uncommitted
	key.deserializer = class com.linkedin.venice.serialization.KafkaKeySerializer
	<http://li.client.cluster.metadata.expire.time.ms|li.client.cluster.metadata.expire.time.ms> = 3600000
	li.client.software.name.and.commit = li-oss-consumer-java
	li.update.metadata.last.refresh.time.upon.node.disconnect = true
	linkedin.enable.client.resonse.leakcheck = false
	linkedin.least.loaded.node.algorithm = VANILLA
	linkedin.pool.class.name = null
	linkedin.skip.metadata.cache.update.upon.unassign = false
	max.partition.fetch.bytes = 1048576
	<http://max.poll.interval.ms|max.poll.interval.ms> = 300000
	max.poll.records = 500
	<http://metadata.max.age.ms|metadata.max.age.ms> = 300000
	metric.reporters = []
	metrics.num.samples = 2
	metrics.recording.level = INFO
	metrics.replace.on.duplicate = false
	<http://metrics.sample.window.ms|metrics.sample.window.ms> = 30000
	partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
	receive.buffer.bytes = 65536
	<http://reconnect.backoff.max.ms|reconnect.backoff.max.ms> = 1000
	<http://reconnect.backoff.ms|reconnect.backoff.ms> = 50
	<http://request.timeout.ms|request.timeout.ms> = 30000
	<http://retry.backoff.ms|retry.backoff.ms> = 100
	sasl.client.callback.handler.class = null
	sasl.jaas.config = null
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	sasl.kerberos.min.time.before.relogin = 60000
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	sasl.kerberos.ticket.renew.window.factor = 0.8
	sasl.login.callback.handler.class = null
	sasl.login.class = null
	sasl.login.refresh.buffer.seconds = 300
	sasl.login.refresh.min.period.seconds = 60
	sasl.login.refresh.window.factor = 0.8
	sasl.login.refresh.window.jitter = 0.05
	sasl.mechanism = GSSAPI
	security.protocol = PLAINTEXT
	security.providers = null
	send.buffer.bytes = 131072
	<http://session.timeout.ms|session.timeout.ms> = 10000
	ssl.cipher.suites = null
	ssl.context.provider.class = org.apache.kafka.common.security.ssl.SimpleSslContextProvider
	ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
	ssl.endpoint.identification.algorithm = https
	ssl.key.password = null
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.location = null
	ssl.keystore.password = null
	ssl.keystore.type = JKS
	ssl.protocol = TLS
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.location = null
	ssl.truststore.password = null
	ssl.truststore.type = JKS
	value.deserializer = class com.linkedin.venice.serialization.avro.KafkaValueSerializer

2023-04-27 00:15:11 INFO [Utils] [main] Loaded schema from resource path: avro/KafkaMessageEnvelope/v1/KafkaMessageEnvelope.avsc
2023-04-27 00:15:11 INFO [Utils] [main] Loaded schema from resource path: avro/KafkaMessageEnvelope/v2/KafkaMessageEnvelope.avsc
2023-04-27 00:15:11 INFO [Utils] [main] Loaded schema from resource path: avro/KafkaMessageEnvelope/v3/KafkaMessageEnvelope.avsc
2023-04-27 00:15:11 INFO [Utils] [main] Loaded schema from resource path: avro/KafkaMessageEnvelope/v4/KafkaMessageEnvelope.avsc
2023-04-27 00:15:11 INFO [Utils] [main] Loaded schema from resource path: avro/KafkaMessageEnvelope/v5/KafkaMessageEnvelope.avsc
2023-04-27 00:15:11 INFO [Utils] [main] Loaded schema from resource path: avro/KafkaMessageEnvelope/v6/KafkaMessageEnvelope.avsc
2023-04-27 00:15:11 INFO [Utils] [main] Loaded schema from resource path: avro/KafkaMessageEnvelope/v7/KafkaMessageEnvelope.avsc
2023-04-27 00:15:11 INFO [Utils] [main] Loaded schema from resource path: avro/KafkaMessageEnvelope/v8/KafkaMessageEnvelope.avsc
2023-04-27 00:15:11 INFO [Utils] [main] Loaded schema from resource path: avro/KafkaMessageEnvelope/v9/KafkaMessageEnvelope.avsc
2023-04-27 00:15:11 INFO [Utils] [main] Loaded schema from resource path: avro/KafkaMessageEnvelope/v10/KafkaMessageEnvelope.avsc
2023-04-27 00:15:11 INFO [Utils] [main] Loaded schema from resource path: avro/KafkaMessageEnvelope/v11/KafkaMessageEnvelope.avsc
2023-04-27 00:15:11 INFO [InternalAvroSpecificSerializer] [main] Serializer doesn't have schemaReader
2023-04-27 00:15:11 INFO [AppInfoParser] [main] Kafka version: 2.4.1.65
2023-04-27 00:15:11 INFO [AppInfoParser] [main] Kafka commitId: 4b7ef9d07758b292
2023-04-27 00:15:11 INFO [AppInfoParser] [main] Kafka startTimeMs: 1682554511880
2023-04-27 00:15:11 INFO [KafkaConsumer] [main] [Consumer clientId=consumer-1, groupId=null] Subscribed to partition(s): t1_n1_s1_v1--1
2023-04-27 00:15:12 INFO [Metadata] [main] [Consumer clientId=consumer-1, groupId=null] Cluster ID: 4iGwXn6hS8uy4RbA6ehsTQ
Something went wrong during topic dump
org.apache.kafka.common.errors.TimeoutException: Failed to get offsets by times in 60000ms
	at org.apache.kafka.clients.consumer.internals.Fetcher.fetchOffsetsByTimes(Fetcher.java:563)
	at org.apache.kafka.clients.consumer.internals.Fetcher.beginningOrEndOffset(Fetcher.java:582)
	at org.apache.kafka.clients.consumer.internals.Fetcher.beginningOffsets(Fetcher.java:567)
	at org.apache.kafka.clients.consumer.KafkaConsumer.beginningOffsets(KafkaConsumer.java:2194)
	at org.apache.kafka.clients.consumer.KafkaConsumer.beginningOffsets(KafkaConsumer.java:2171)
	at com.linkedin.venice.KafkaTopicDumper.<init>(KafkaTopicDumper.java:121)
	at com.linkedin.venice.AdminTool.dumpKafkaTopic(AdminTool.java:1410)
	at com.linkedin.venice.AdminTool.main(AdminTool.java:384)
This is my attempt to dump topic using admin tool
s
Can you also specify the partitions
--kafka-topic-partition 0
QQ: how many partitions are there?
a
should be one
Copy code
2023-04-27 00:21:12 INFO [KafkaConsumer] [main] [Consumer clientId=consumer-1, groupId=null] Subscribed to partition(s): t1_n1_s1_v1-0
2023-04-27 00:21:12 INFO [Metadata] [main] [Consumer clientId=consumer-1, groupId=null] Cluster ID: 4iGwXn6hS8uy4RbA6ehsTQ
2023-04-27 00:21:12 INFO [KafkaTopicDumper] [main] Starting from offset: 0
2023-04-27 00:21:12 INFO [KafkaTopicDumper] [main] End offset for partition 0 is 10
2023-04-27 00:21:12 INFO [KafkaTopicDumper] [main] Found a control message, continue
2023-04-27 00:21:12 INFO [KafkaTopicDumper] [main] Found a control message, continue
2023-04-27 00:21:12 INFO [KafkaTopicDumper] [main] Found a control message, continue
2023-04-27 00:21:12 INFO [KafkaTopicDumper] [main] Found a control message, continue
2023-04-27 00:21:12 INFO [KafkaTopicDumper] [main] Found a control message, continue
2023-04-27 00:21:12 INFO [KafkaTopicDumper] [main] Found a control message, continue
2023-04-27 00:21:12 INFO [KafkaTopicDumper] [main] Found a control message, continue
2023-04-27 00:21:12 INFO [KafkaTopicDumper] [main] Found a control message, continue
2023-04-27 00:21:12 INFO [KafkaTopicDumper] [main] Found a control message, continue
2023-04-27 00:21:12 INFO [KafkaTopicDumper] [main] Found a control message, continue
so it found 10 messages, all of the control (whatever it means)
s
that means they are internal messages (control)
can you pass 
-lm
and paste the output?
it's possible that it may dump data in a file without -lm option
a
Copy code
2023-04-27 00:24:13 INFO [KafkaConsumer] [main] [Consumer clientId=consumer-1, groupId=null] Subscribed to partition(s): t1_n1_s1_v1-0
2023-04-27 00:24:13 INFO [Metadata] [main] [Consumer clientId=consumer-1, groupId=null] Cluster ID: 4iGwXn6hS8uy4RbA6ehsTQ
2023-04-27 00:24:13 INFO [KafkaTopicDumper] [main] Starting from offset: 0
2023-04-27 00:24:13 INFO [KafkaTopicDumper] [main] End offset for partition 0 is 10
2023-04-27 00:24:13 INFO [KafkaTopicDumper] [main] CTRL START_OF_SEGMENT Offset:0 ProducerMd=(guid:1ebe71daa11247e3aa8bd8063be590dd,seg:0,seq:0,mts:1682552589297,lts:-1) LeaderMd=(host:venice-controller,uo:-1,ukcId:-1)
2023-04-27 00:24:13 INFO [KafkaTopicDumper] [main] CTRL START_OF_PUSH Offset:1 ProducerMd=(guid:1ebe71daa11247e3aa8bd8063be590dd,seg:0,seq:1,mts:1682552589391,lts:-1) LeaderMd=(host:venice-controller,uo:-1,ukcId:-1)
2023-04-27 00:24:13 INFO [KafkaTopicDumper] [main] CTRL END_OF_SEGMENT Offset:2 ProducerMd=(guid:1ebe71daa11247e3aa8bd8063be590dd,seg:0,seq:2,mts:1682552589411,lts:-1) LeaderMd=(host:venice-controller,uo:-1,ukcId:-1)
2023-04-27 00:24:13 INFO [KafkaTopicDumper] [main] CTRL START_OF_SEGMENT Offset:3 ProducerMd=(guid:7d1a392743474cbd950f62cf08a2fb6f,seg:0,seq:0,mts:1682552590586,lts:-1) LeaderMd=(host:venice-controller,uo:-1,ukcId:-1)
2023-04-27 00:24:13 INFO [KafkaTopicDumper] [main] CTRL START_OF_PUSH Offset:4 ProducerMd=(guid:7d1a392743474cbd950f62cf08a2fb6f,seg:0,seq:1,mts:1682552590601,lts:-1) LeaderMd=(host:venice-controller,uo:-1,ukcId:-1)
2023-04-27 00:24:13 INFO [KafkaTopicDumper] [main] CTRL END_OF_PUSH Offset:5 ProducerMd=(guid:7d1a392743474cbd950f62cf08a2fb6f,seg:0,seq:2,mts:1682552590605,lts:-1) LeaderMd=(host:venice-controller,uo:-1,ukcId:-1)
2023-04-27 00:24:13 INFO [KafkaTopicDumper] [main] CTRL END_OF_SEGMENT Offset:6 ProducerMd=(guid:7d1a392743474cbd950f62cf08a2fb6f,seg:0,seq:3,mts:1682552590608,lts:-1) LeaderMd=(host:venice-controller,uo:-1,ukcId:-1)
2023-04-27 00:24:13 INFO [KafkaTopicDumper] [main] CTRL START_OF_SEGMENT Offset:7 ProducerMd=(guid:083b838ca85242f39089372ffb8e797b,seg:0,seq:0,mts:1682552591088,lts:-1) LeaderMd=(host:venice-controller,uo:-1,ukcId:-1)
2023-04-27 00:24:13 INFO [KafkaTopicDumper] [main] CTRL TOPIC_SWITCH Offset:8 ProducerMd=(guid:083b838ca85242f39089372ffb8e797b,seg:0,seq:1,mts:1682552592106,lts:-1) LeaderMd=(host:venice-controller,uo:-1,ukcId:-1)
2023-04-27 00:24:13 INFO [KafkaTopicDumper] [main] CTRL END_OF_SEGMENT Offset:9 ProducerMd=(guid:083b838ca85242f39089372ffb8e797b,seg:0,seq:2,mts:1682552592114,lts:-1) LeaderMd=(host:venice-controller,uo:-1,ukcId:-1)
s
looks like there is no data being written
a
the only new thing I see in current dir is t1_n1_s1_v1_0.avro
Copy code
# cat t1_n1_s1_v1_0.avro
Objavro.schema�
               {"type":"record","name":"KafkaRecord","namespace":"none","doc":"","fields":[{"name":"key","type":"string","doc":"Raw bytes of the key"},{"name":"value","type":["null",{"type":"record","name":"Person","namespace":"sink.PulsarVeniceSinkTest","fields":[{"name":"age","type":"int"},{"name":"name","type":["null","string"],"default":null}]}],"doc":"Raw bytes of the value"},{"name":"offset","type":"long","doc":"The offset of this record in Kafka"},{"name":"DELETED_TS","type":["null","long"],"doc":"If the current record is a PUT, this field will be null; if it's a DELETE, this field will be the offset of the record in Kafka","default":null},{"name":"metadata","type":{"type":"map","values":"string"},"doc":"Metadata of the record; currently it contains the schemaId of the record"}]}d�s�s?�*ԥ��T}
or I managed to send data as control messages 😕
s
or I managed to send data as control messages
control messages are part of venice internal communication protocol
SOP(StartOfPush) --> Batch Data --> EOP(EndOfPush) --> TS(TopicSwitch) [signifies start consuming RT] --> ....
StartOfSegment and EndOfSegment(EOS) is for data integrity validation. Stream is split into segments for integrity checks. On servers upon processing EOS, segment is verified for data integrity (with lots of in between steps)
how are you ingesting data? VPJ or something else?
a
I am using VeniceSystemProducer
s
oh ok. May be @Zac Policzer can answer your question then
also try dumping RT as well
a
I had to tweak it to skip D2 configuration https://github.com/dlg99/venice/tree/pulsar-sink - the branch is dirty/needs clean up of experiments + rebase, it is couple of week old. https://github.com/dlg99/venice/blob/pulsar-sink/clients/venice-pulsar/src/main/java/com/linkedin/venice/pulsar/sink/VeniceSink.java os where I send the data.
Thank you so much for your help, really helped me understand how to troubleshoot this
🙏 1
I have to stop for the day (family), will experiment more tomorrow
s
Sounds great!
3 Views
Open in Slack
PreviousNext
Powered by