https://venicedb.org logo
Join Slack
Powered by
Hi all, We are still in the process of deploying t...
# general
d
Hi all, We are still in the process of deploying the change for the fast-client. I would like to ask for some guidance on how I can debug if R2/D2 routing is configured properly, and my app will be able to reach the store with the fast-client. I'll add my current config files to the thread.
Server:
Copy code
admin.port=54096

data.base.path=/opt/venice/rocksdb

disk.full.threshold=0.99

enable.server.allowlist=false

<http://kafka.read.cycle.delay.ms|kafka.read.cycle.delay.ms>=50

listener.port=54093

max.state.transition.thread.number=100

<http://participant.message.consumption.delay.ms|participant.message.consumption.delay.ms>=1000

persistence.type=ROCKS_DB

r2d2Client.zkHosts=venice-zookeeper-headless:2181

rocksdb.block.cache.size.in.bytes=2147483648
rocksdb.options.use.direct.reads=false
rocksdb.plain.table.format.enabled=true
rocksdb.sst.file.manager.delete.rate.bytes.per.second=524288000
rocksdb.sst.file.manager.max.trash.db.ratio=0.25

server.d2.announce.enabled=true
server.d2.announce.host=venice-server
server.d2.service.name=venice-server-cluster0-d2

server.netty.graceful.shutdown.period.seconds=0
server.partition.graceful.drop.time.in.seconds=0
server.promotion.to.leader.replica.delay.seconds=1
server.rest.service.storage.thread.num=4

system.schema.cluster.name=venice-cluster0
router:
Copy code
cluster.name=venice-cluster0
cluster.to.d2=venice-cluster0:venice-router-cluster0-d2
cluster.to.server.d2=venice-cluster0:venice-server-cluster0-d2

kafka.bootstrap.servers=kafka:9092
# kafka.zk.address=venice-zookeeper-headless:2181

listener.port=7777
listener.ssl.port=7778

max.read.capacity=20000000

router.connection.limit=20

router.d2.announce.enabled=true
router.d2.announce.host=venice-router

router.enable.ssl=false

router.http.client.pool.size=2
router.httpasyncclient.connection.warming.low.water.mark=1
router.http2.inbound.enabled=true

router.io.worker.count=4

router.max.outgoing.connection.per.route=2
router.max.outgoing.connection=10

router.netty.graceful.shutdown.period.seconds=0

router.storage.node.client.type=APACHE_HTTP_ASYNC_CLIENT

sslToStorageNodes=false

system.schema.cluster.name=venice-cluster0

zookeeper.address=venice-zookeeper-headless:2181
controller:
Copy code
admin.port=5555
admin.secure.port=5556

admin.topic.replication.factor=1

child.cluster.allowlist=

cluster.name=venice-cluster0
cluster.to.d2=venice-cluster0:venice-router-cluster0-d2
cluster.to.server.d2=venice-cluster0:venice-server-cluster0-d2

concurrent.init.routines.enabled=true

controller.add.version.via.admin.protocol=true
controller.name=venice-controller
controller.parent.mode=false
controller.ssl.enabled=false
controller.system.schema.cluster.name=venice-cluster0
controller.zk.shared.davinci.push.status.system.schema.store.auto.creation.enabled=true
controller.zk.shared.metadata.system.schema.store.auto.creation.enabled=true

default.partition.count=1
default.partition.max.count=3
default.partition.size=100
default.replica.factor=1

<http://delay.to.rebalance.ms|delay.to.rebalance.ms>=0

enable.hybrid.push.ssl.whitelist=false
enable.leader.follower.as.default.for.all.stores=true
enable.offline.push.ssl.whitelist=false

kafka.admin.class=com.linkedin.venice.kafka.admin.ApacheKafkaAdminAdapter
kafka.bootstrap.servers=kafka:9092
<http://kafka.linger.ms|kafka.linger.ms>=0
kafka.replication.factor=1
# kafka.zk.address=venice-zookeeper-headless:2181

min.active.replica=1

<http://offline.job.start.timeout.ms|offline.job.start.timeout.ms>=60000

persistence.type=ROCKS_DB

push.status.store.enabled=true

ssl.to.kakfa=false

storage.engine.overhead.ratio=0.85

topic.cleanup.send.concurrent.delete.requests.enabled=false
<http://topic.cleanup.sleep.interval.between.topic.list.fetch.ms|topic.cleanup.sleep.interval.between.topic.list.fetch.ms>=30000
<http://topic.creation.throttling.time.window.ms|topic.creation.throttling.time.window.ms>=100

zookeeper.address=venice-zookeeper-headless:2181
At the moment we've the following error:
Copy code
- Venice client - retrieves multiple keys
    com.linkedin.venice.client.exceptions.ServiceDiscoveryException: Failed to find d2 service for test-store after 10 attempts
        at com.linkedin.venice.client.store.D2ServiceDiscovery.find(D2ServiceDiscovery.java:100)
        at com.linkedin.venice.fastclient.meta.RequestBasedMetadata.discoverD2Service(RequestBasedMetadata.java:230)
        at com.linkedin.venice.fastclient.meta.RequestBasedMetadata.start(RequestBasedMetadata.java:198)
        at com.linkedin.venice.fastclient.DispatchingAvroGenericStoreClient.start(DispatchingAvroGenericStoreClient.java:753)
        at com.linkedin.venice.fastclient.DelegatingAvroStoreClient.start(DelegatingAvroStoreClient.java:115)
        at com.linkedin.venice.fastclient.factory.ClientFactory.getAndStartGenericStoreClient(ClientFactory.java:95)
        at com.linkedin.venice.fastclient.factory.ClientFactory.getAndStartGenericStoreClient(ClientFactory.java:32)
        at com.liveintent.venice.VeniceClient$.$anonfun$storeClient$3(VeniceClient.scala:103)
        at zio.ZIOCompanionVersionSpecific.$anonfun$attempt$1(ZIOCompanionVersionSpecific.scala:98)
I also tried to enforce 
router.d2.announce.host
and 
server.d2.announce.host
being unique by setting it as 
<pod_name>.<pod_namespace>
. But it seems it doesn't work also.
And this is output for `describe-store`:
Copy code
{
  "cluster": "venice-cluster0",
  "name": "client-id-to-email-hash",
  "error": null,
  "errorType": null,
  "store": {
    "name": "client-id-to-email-hash",
    "owner": "",
    "currentVersion": 6,
    "coloToCurrentVersions": null,
    "reservedVersion": 0,
    "partitionCount": 1000,
    "lowWatermark": 0,
    "enableStoreWrites": true,
    "enableStoreReads": true,
    "versions": [
      {
        "storeName": "client-id-to-email-hash",
        "number": 6,
        "createdTime": 1773916943711,
        "pushJobId": "1773916681699_failed_to_obtain_execution_url",
        "partitionCount": 1000,
        "partitionerConfig": {
          "partitionerClass": "com.linkedin.venice.partitioner.DefaultVenicePartitioner",
          "partitionerParams": {},
          "amplificationFactor": 1
        },
        "status": "ONLINE",
        "isDavinciHeartbeatReported": false,
        "blobDbEnabled": "NOT_SPECIFIED",
        "blobTransferInServerEnabled": "NOT_SPECIFIED",
        "blobTransferEnabled": false,
        "activeActiveReplicationEnabled": false,
        "nativeReplicationSourceFabric": "",
        "replicationFactor": 1,
        "pushStreamSourceAddress": "",
        "nativeReplicationEnabled": false,
        "separateRealTimeTopicEnabled": false,
        "incrementalPushEnabled": false,
        "rmdChunkingEnabled": false,
        "chunkingEnabled": false,
        "compressionStrategy": "NO_OP",
        "globalRtDivEnabled": false,
        "targetSwapRegion": "",
        "keyUrnCompressionEnabled": false,
        "keyUrnFields": [],
        "previousCurrentVersion": -1,
        "pushType": "BATCH",
        "versionSwapDeferred": false,
        "minActiveReplicas": 0,
        "useVersionLevelIncrementalPushEnabled": true,
        "hybrid": false,
        "hybridStoreConfig": null,
        "useVersionLevelHybridConfig": true,
        "dataRecoveryVersionConfig": null,
        "targetSwapRegionWaitTime": 60,
        "repushSourceVersion": -1,
        "repushTtlSeconds": -1,
        "leaderFollowerModelEnabled": true,
        "timestampMetadataVersionId": 1,
        "viewConfigs": {}
      }
    ],
    "storageQuotaInByte": 10000000000000,
    "hybridStoreOverheadBypass": false,
    "readQuotaInCU": 1800,
    "hybridStoreConfig": null,
    "accessControlled": true,
    "chunkingEnabled": false,
    "rmdChunkingEnabled": false,
    "singleGetRouterCacheEnabled": false,
    "batchGetRouterCacheEnabled": false,
    "batchGetLimit": -1,
    "largestUsedVersionNumber": 6,
    "largestUsedRTVersionNumber": 0,
    "incrementalPushEnabled": false,
    "compressionStrategy": "NO_OP",
    "clientDecompressionEnabled": true,
    "numVersionsToPreserve": 0,
    "migrating": false,
    "writeComputationEnabled": false,
    "replicationMetadataVersionId": -1,
    "readComputationEnabled": false,
    "bootstrapToOnlineTimeoutInHours": 24,
    "nativeReplicationEnabled": false,
    "pushStreamSourceAddress": "",
    "backupStrategy": "DELETE_ON_NEW_PUSH_START",
    "schemaAutoRegisterFromPushJobEnabled": false,
    "superSetSchemaAutoGenerationForReadComputeEnabled": false,
    "latestSuperSetValueSchemaId": -1,
    "hybridStoreDiskQuotaEnabled": false,
    "etlStoreConfig": {
      "etledUserProxyAccount": "",
      "regularVersionETLEnabled": false,
      "futureVersionETLEnabled": false,
      "etlstrategy": "EXTERNAL_SERVICE"
    },
    "partitionerConfig": {
      "partitionerClass": "com.linkedin.venice.partitioner.DefaultVenicePartitioner",
      "partitionerParams": {},
      "amplificationFactor": 1
    },
    "backupVersionRetentionMs": 604800000,
    "replicationFactor": 1,
    "migrationDuplicateStore": false,
    "nativeReplicationSourceFabric": "",
    "storeMetadataSystemStoreEnabled": false,
    "storeMetaSystemStoreEnabled": false,
    "daVinciPushStatusStoreEnabled": false,
    "activeActiveReplicationEnabled": false,
    "kafkaBrokerUrl": "<kafka>:9092",
    "viewConfigs": {},
    "storageNodeReadQuotaEnabled": true,
    "storeDeadStatusReasons": [],
    "isStoreDead": false,
    "compactionEnabled": true,
    "compactionThreshold": -1,
    "minCompactionLagSeconds": -1,
    "maxCompactionLagSeconds": -1,
    "maxRecordSizeBytes": 104857600,
    "maxNearlineRecordSizeBytes": -1,
    "unusedSchemaDeletionEnabled": false,
    "blobTransferEnabled": false,
    "blobDbEnabled": "NOT_SPECIFIED",
    "nearlineProducerCompressionEnabled": true,
    "nearlineProducerCountPerWriter": 1,
    "targetRegionSwap": "",
    "targetRegionSwapWaitTime": 60,
    "isDavinciHeartbeatReported": false,
    "globalRtDivEnabled": false,
    "enumSchemaEvolutionAllowed": false,
    "storeLifecycleHooks": [],
    "keyUrnCompressionEnabled": false,
    "keyUrnFields": [],
    "flinkVeniceViewsEnabled": false,
    "previousCurrentVersion": -1,
    "blobTransferInServerEnabled": "NOT_SPECIFIED",
    "ttlrepushEnabled": false,
    "latestVersionPromoteToCurrentTimestamp": 1773928197225
  },
  "exceptionType": null
}
hi @Zac Policzer , can you or your colleagues maybe guide me with it?
z
Whoops, this slipped us by apologies
So the way I would typically debug this is I would look at the routing state in the d2 zk for the ednpoint I was trying to hit. The ednpoint should be under d2://venice-cluster0, but for fast client I 'believe' it uses a separate storage node address. @Xun Yin would remember better then I think. Have you access to your zk? Internal to linkedin we often use zooinspector for this kind of thing.
Essentially the steps to debug should be: • Make sure the d2 endpoint is in zk and available • Make sure the uri's under the endpoint are non empty and make sense (have http/https and appropriate port information that matches your config). • If that all looks good, check the configuration, make sure SSL or not SSL is what you'd expect. In the d2 service config in zk there will be a note about prioritized schemes, see if it has http/https in that list and that those endpoints are indeed available and that your client is correctly configured
x
Sorry @Dmytro Prokhorenkov and @Zac Policzer I was on vacation but here are the corresponding d2 endpoints you can hit to verify your setup: Hit the discovery endpoint with curl d2://venice-discovery/discover_cluster/test-store-name and you should get some output like
Copy code
{
  "cluster": "cluster-0",
  "name": "test-store-name",
  "error": null,
  "errorType": null,
  "d2Service": "venice-cluster-0",
  "serverD2Service": "venice-server-cluster-0",
  "zkAddress": "<http://xxx.xxx.xxx:2622/venice|xxx.xxx.xxx:2622/venice>",
  "kafkaBootstrapServers": "<http://kafka.venice.xxx:16637|kafka.venice.xxx:16637>",
  "exceptionType": null
}
The FC will use the same discovery call and 
serverD2Service
is what it will then use to attempt to fetch metadata from. The payload will be in avro but you can still hit it to see if you can get back response so something like 
<d2://serverD2Service/metadata/><storeName>
d
Hi, I'm off till Monday. Can you send me an example of how to run this curl to D2 path? Or is it just the router address as a URL?
x
ah good catch, I think i was using some linkedin flavor of curl. It should work with regular curl and a router URL, let me try to figure that out. Alternatively you can also write your own admin tool endpoint to perform this and it should be much easier programmatically to create a D2TransportClient and hit the 
discover_cluster
endpoint on the 
venice-discovery
d2 service
z
Honestly someone should just vibe code the d2 friendly version of curl that just hits ZK and figures out an address
x
Given that you know the port number and have some similar ssl configuration this should work:
Copy code
~ curl -ivk "<https://xxx.linkedin.com:8100/discover_cluster/><storeName>" --key identity.key --cert identity.cert
I was able to get the output:
Copy code
* Request completely sent off
< HTTP/2 200 
HTTP/2 200 
< content-type: application/json
content-type: application/json
< content-length: 310
content-length: 310
< 

* Connection #0 to host <http://xxx.linkedin.com|xxx.linkedin.com> left intact
{"cluster":"xxx","name":"xxx","error":null,"errorType":null,"d2Service":"cluster-0","serverD2Service":"venice-server-cluster-0","zkAddress":"<http://xxx.linkedin.com:2622/venice|xxx.linkedin.com:2622/venice>","kafkaBootstrapServers":"<http://xxx.linkedin.com:16637|xxx.linkedin.com:16637>","exceptionType":null}%
d
I found a few issues with addressing zookeeper in our setup. Right now I've such response from router:
Copy code
/ # curl -s 10.233.133.100:7777/discover_cluster/<storename> | jq -r '.'
{
  "cluster": "venice-cluster0",
  "name": "<storename>",
  "error": null,
  "errorType": null,
  "d2Service": "venice-router-cluster0-d2",
  "serverD2Service": "venice-server-cluster0-d2",
  "zkAddress": "venice-zookeeper-headless.venice-staging:2181",
  "kafkaBootstrapServers": "<kafka-address>:9092",
  "exceptionType": null
}
I've a few questions: • should 
zkAddress
be the same to ZK address defined for venice components? In our case for 
zkAddress
it has k8s service name and namespace. ZK itself is running in the same namespace as venice components. • should server-side and client-side versions be equal, or could there be a slight drift? In application atm we've an error saying 
Wrapped by: com.linkedin.r2.RemoteInvocationException: Failed to get response from server for URI <http://10.233.131.21:7777/discover_cluster/><storename>
. But we'll have a debug session with Dejan on Thursday. So I'll have update after it.
z
should zkAddress be the same to ZK address defined for venice components? In our case for zkAddress it has k8s service name and namespace. ZK itself is running in the same namespace as venice components.
In our integration tests and in 'most' likely your case yet, it'll all be the same zk. That said, it doesn't HAVE to be. In production at LinkedIn there are seperate zk's entirely for d2/venice/kafka, mostly by virtue of the teams supporting those platforms being seperate.
should server-side and client-side versions be equal, or could there be a slight drift?
I'm not sure I understand this question fully (perhaps I'm up too early). Do you mean what zk address server and client side see? They should be equal.
d
Sorry. I mean venice version. Like i've 0.4.798 in client and 0.4.799 in cluster
z
Ah I see. So, we're generally very careful about keeping things compatible across release versions. So it's not strictly required they're the same. And in fact, I'd wager in production they're almost never the same.
x
+1 on the above, we have partner teams eager to try the latest client versions for new features/optimization while also having partner teams that won't pick up a new client version until the one they are on is deprecated or forced to upgrade. While all this is happening we are deploying the backend (cluster) for bug fixes and improvements with regular release cadence.
3 Views
Open in Slack
PreviousNext
Powered by