Hi, we are slowly but steadily continue with testing Venice. We are trying now with the bigger setup and trying to push 4TB store but we fail. We use 0.4.613 version with the k8s setup through EKS. We are pushing with 100 partitions and 12 venice-servers. It seems that some partitions fail to allocate replicas. The root cause seems to be the failure of opening RocksDB metadata database on some venice-server pods which manifests in such exceptions:

Caused by: org.rocksdb.RocksDBException: While lock file: /opt/venice/rocksdb/rocksdb/client-id-to-email-hash_v26/client-id-to-email-hash_v26_1000000000/LOCK: Resource temporarily unavailable

I am attaching VPJ, leader controler and one of the venice-server logs. It seems that venice-server tries to open the metadata database from multiple threads and the locking failures are not retried. Is it a known issue? Are we doing anything wrong? 🤔

Is there only 1 venice-server pod per node? If there's multiple, it could run into this issue.

yes - this is actually the case. @Dmytro Prokhorenkov is adjusting the setup and we will continue testing. Thank you!

Checking to see if that change fixed this issue?

Thanks, all fixed now. It was my bad 🙂

No worries. Glad to hear that the issue is resolved

Thank you @Koorous Vargha

No problem

Hey @Koorous Vargha. We didn't run in any further problems yet but we are also not running it yet on production. @Dejan Mijic and his team are currently load testing. The plan is to have it on production by the end of the year.

Gotcha. Thank you for the update! cc: @Zac Policzer @Felix GV

That’s very exciting!

Btw, I understand you've real servers with x86_64 CPU 🙂 . Just curious - have you tried to build venice for arm64?

*laughs in java

We're still running on x86 in our internal Kubernetes clusters. Does AWS still offer x86 for the available node pools?

The Java part should work just fine… the question is whether RocksJava packs that architecture within it or not. I suspect it probably does, but IDK.

Looks like this got discussed at one point in rocksdb community: https://github.com/facebook/rocksdb/issues/5559

Is there any advantage to using arm VMs in AWS? Or is it because you have some sort of edge use case in mind where it has to be arm?

I'm pretty sure they're cheaper

Sounds refreshing to care about cost without needing to do the optimizations under the table 😅

Price was main factor for older instances. Latest version is slightly expensive. We're running our Voldemort's on arm machines for the almost two years.