https://venicedb.org logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
e
java.util.concurrent.ExecutionException: com.linkedin.venice.client.exceptions.VeniceClientHttpException: http status: 400, Online writes are only supported for hybrid stores that have ACTIVE_ACTIVE or NON_AGGREGATE data replication policy.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.linkedin.venice.producer.online.OnlineVeniceProducer.lambda$executeRouterRequest$1(OnlineVeniceProducer.java:100)
at net.jodah.failsafe.Functions.lambda$get$0(Functions.java:48)
at net.jodah.failsafe.RetryPolicyExecutor.lambda$supply$0(RetryPolicyExecutor.java:62)
at net.jodah.failsafe.Execution.executeSync(Execution.java:129)
at net.jodah.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
... 17 more
Caused by: com.linkedin.venice.client.exceptions.VeniceClientHttpException: http status: 400, Online writes are only supported for hybrid stores that have ACTIVE_ACTIVE or NON_AGGREGATE data replication policy.
at com.linkedin.venice.client.store.transport.TransportClientCallback.completeFuture(TransportClientCallback.java:39)
at com.linkedin.venice.client.store.transport.HttpTransportClient$HttpTransportClient
f
Thanks for trying it out Enrico! Could you help out with this @Nisarg Thakkar?
e
One interesting thing is that this OnlineProducer shouldn’t be much different from the SystemProducer that we use from the PulsarVeniceSink, I mean there we don’t depend on ZK discovery. I image that in the end we will end up in moving the PulsarVeniceSink to using the OnlineProducer in order to reduce code complexity/duplication
f
Nisarg has a plan for consolidation with the goal that the Pulsar sink stops depending on Samza. Pulsar, Samza and the OnlineProducer would all depend on a shared building block.
e
makes great sense
f
He has a draft PR somewhere for that, but wanted to merge the OP first.
e
@Nisarg Thakkar it seems that if I remove that assertion everything works well (at least I am able to read the data from the router), with the default configuration of the store. I have just enabled incremental push.
Copy code
java -jar $jar --new-store --url $url --cluster $clusterName  --store $storeName --key-schema-file $keySchema --value-schema-file $valueSchema --hybrid-data-replication-policy NON_AGGREGATE


# enable incremental push
java -jar $jar --update-store --url $url --cluster $clusterName  --store $storeName --storage-quota -1 --incremental-push-enabled true  --read-quota 1000000

# disable read quota

# create the first version of the store
java -jar $jar --empty-push --url $url --cluster $clusterName --store $storeName --push-id init --store-size 1000
n
Hi @Enrico Olivelli. 
AGGREGATE
mode is deprecated and we do not wish to support it any more as it leads to extra cross-colo data movement and has a SPoF problem I believe the steps you mentioned in the last message should have worked. Did they not?
Oh, I see what’s wrong. 
--hybrid-data-replication-policy NON_AGGREGATE
is accepted by 
--update-store
and not 
--new-store
. So, you need to set it in the 
# enable incremental push
command
e
It doesn't work. Could you please try? I simply start Venice standalone locally, and create the store with the script. Then I get that error when I try to connect
My understanding is that the currentVersion has NONE as strategy but the store has the correct value
I saw that test but it doesn't reproduce the problem. I am not sure that it is following the same code paths
n
I see. So this is due to setting incremental push without setting other hybrid configs.
Copy code
/**
   * This enum value is used in 2 cases:
   *   1. batch-only stores since it has no real-time data replication.
   *   2. incremental push stores that do not have Samza jobs.
   */
  NONE(2),
This javadoc needs to be updated, but Samza job = Nearline producer = Online producer So, you should be specifying some hybrid configs with your 
--update-store
. At LinkedIn, we use 
--hybrid-rewind-seconds 86400 --hybrid-offset-lag 1000
by default for users
e
I will try it out tomorrow. Thank you very much
What do those parameters mean?
z
They configure the behavior of how Venice behaves between pushes when hybrid is enabled. 
hybrid-rewind-seconds 86400
means that when the next push happens, the messages produced to nearline within the last 86400s will be applied to the new version. -
-hybrid-offset-lag 1000
basically means that after the new version is pushed, each replica should be at least 1000 offsets from the tail of nearline data before declaring itself ready to serve traffic
e
And if I have only 1 version? And I never create v2?
z
Then doesn't really matter
Those parameters just have to be provided to enable nearline producing from incremental push or samza producer or the online client
Basically, for all other write paths other then batch push
e
Tomorrow I will check out the code. Thanks
f
We should probably fill in those defaults on behalf of the user if the user doesn't care... unless we believe it is impossible to use Venice properly without understanding these parameters (I doubt that's the case, but someone could argue for it I suppose).
z
No arguing from me.
n
Glancing over the code, it looks like they should have gotten set as the default values
I'll try an integration test to see what's up
Looks like the intention was there to set default values, but some update-store code paths don’t handle this very well. We should fix this, but for the time being, we need to specify the hybrid configs in 
update-store
call.
Copy code
@Test
public void testIncPushDRP() {
  String storeName = Utils.getUniqueString("hybrid-store");
  sharedVenice.useControllerClient(cc -> {
    NewStoreResponse newStoreResponse = cc.createNewStore(storeName, "default", "\"string\"", "\"string\"");
    Assert.assertFalse(newStoreResponse.isError());

    UpdateStoreQueryParams updateStoreQueryParams = new UpdateStoreQueryParams()
        .setStorageQuotaInByte(1)
        .setIncrementalPushEnabled(true)
        .setReadQuotaInCU(1000000)
        .setHybridDataReplicationPolicy(DataReplicationPolicy.NON_AGGREGATE)
        .setHybridRewindSeconds(86400)
        .setHybridOffsetLagThreshold(1000);

    ControllerResponse updateStoreResponse = cc.updateStore(storeName, updateStoreQueryParams);
    Assert.assertFalse(updateStoreResponse.isError());

    cc.sendEmptyPushAndWait(storeName, Utils.getUniqueString(storeName), 1000, 10000);

    StoreResponse storeResponse = cc.getStore(storeName);
    StoreInfo storeInfo = storeResponse.getStore();
    Assert.assertEquals(storeInfo.getVersion(1).get().getHybridStoreConfig().getDataReplicationPolicy(), DataReplicationPolicy.NON_AGGREGATE);
  });
}
e
it works !
🎉 2
4 Views
Open in Slack
PreviousNext
Powered by