This message was deleted venice #general

Join Slack

This message was deleted.

# general

Slackbot

03/16/2023, 11:50 AM

This message was deleted.

Enrico Olivelli

03/16/2023, 11:51 AM

logs on the controller (only one instance of each service types is running)

Copy code

2023-03-16 11:49:12 INFO [VeniceWriter [83dbf6a57f234d4ab69426be7a1061eb]] [qtp1830701273-135] Successfully broadcast START_OF_PUSH Control Message for topic: test-store-persons_v1
2023-03-16 11:49:12 INFO [VeniceWriter [83dbf6a57f234d4ab69426be7a1061eb]] [qtp1830701273-135] Successfully broadcast END_OF_PUSH Control Message for topic: test-store-persons_v1
2023-03-16 11:49:12 WARN [VeniceWriter [83dbf6a57f234d4ab69426be7a1061eb]] [qtp1830701273-135] endSegment(partition 0) called but currentSegment.ended == true. Ignoring.
2023-03-16 11:49:12 WARN [VeniceWriter [83dbf6a57f234d4ab69426be7a1061eb]] [qtp1830701273-135] endSegment(partition 1) called but currentSegment.ended == true. Ignoring.
2023-03-16 11:49:12 WARN [VeniceWriter [83dbf6a57f234d4ab69426be7a1061eb]] [qtp1830701273-135] endSegment(partition 2) called but currentSegment.ended == true. Ignoring.
2023-03-16 11:49:12 INFO [ApacheKafkaProducerAdapter] [qtp1830701273-135] Flushed all the messages in producer before closing
2023-03-16 11:49:12 INFO [KafkaProducer] [qtp1830701273-135] [Producer clientId=test-store-persons_v1] Closing the Kafka producer with timeoutMillis = 30000 ms.
2023-03-16 11:49:12 INFO [AdminSparkServer] [qtp1830701273-135] [AUDIT] SUCCESS POST <http://10.204.5.9:5555/empty_push> {push_job_id=init, cluster_name=venice-cluster0, hostname=localhost, store_name=test-store-persons, store_size=1
2023-03-16 11:49:12 INFO [AdminSparkServer] [qtp1830701273-133] [AUDIT] GET <http://venice-controller:5555/leader_controller> {cluster_name=venice-cluster0, hostname=localhost}

Enrico Olivelli

03/16/2023, 11:52 AM

what is the client waiting for ?

Enrico Olivelli

03/16/2023, 11:55 AM

I have found an error on Kafka, it complains that this topic doesn’t exist. test-store-persons_rt (my understanding is that that’s the topic for incremental push jobs)

Felix GV

03/16/2023, 12:06 PM

The RT topic should get created automatically. Is this using Kafka or Pulsar underneath? I'm wondering if the admin operation to create a topic might have failed due to some differences in the admin endpoints between Kafka and KoP?

Enrico Olivelli

03/16/2023, 1:15 PM

usually everything works well, it is on my k8s cluster, maybe I still have some problems with the DNS names

Enrico Olivelli

03/16/2023, 1:15 PM

I will keep you posted

Enrico Olivelli

03/16/2023, 1:28 PM

can this be the problem ?

Copy code

venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266680094+01:00 com.linkedin.venice.exceptions.VeniceException: Failed to add version: 1 to store: test-store-persons-3 in cluster: venice-cluster0. Detailed stack trace: com.linkedin.venice.exceptions.VeniceException: After waiting for 60000ms, resource assignment for: test-store-persons-3_v1 timed out, strategy=WAIT_ALL_REPLICAS, replicationFactor=1, reason=not yet in EXTERNALVIEW
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266686164+01:00 	at com.linkedin.venice.controller.VeniceHelixAdmin.waitUntilNodesAreAssignedForResource(VeniceHelixAdmin.java:4412)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266690724+01:00 	at com.linkedin.venice.controller.VeniceHelixAdmin.addVersion(VeniceHelixAdmin.java:2368)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266694844+01:00 	at com.linkedin.venice.controller.VeniceHelixAdmin.incrementVersionIdempotent(VeniceHelixAdmin.java:2484)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266698854+01:00 	at com.linkedin.venice.controller.Admin.incrementVersionIdempotent(Admin.java:185)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266702764+01:00 	at com.linkedin.venice.controller.server.CreateVersion.lambda$emptyPush$6(CreateVersion.java:652)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266715134+01:00 	at spark.RouteImpl$1.handle(RouteImpl.java:72)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266718934+01:00 	at spark.http.matching.Routes.execute(Routes.java:61)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266722714+01:00 	at spark.http.matching.MatcherFilter.doFilter(MatcherFilter.java:134)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266726534+01:00 	at spark.embeddedserver.jetty.JettyHandler.doHandle(JettyHandler.java:50)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266730344+01:00 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1598)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266734174+01:00 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266738044+01:00 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266741994+01:00 	at org.eclipse.jetty.server.Server.handle(Server.java:516)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266746794+01:00 	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266750804+01:00 	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266754824+01:00 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266758824+01:00 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266762614+01:00 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266767174+01:00 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266771094+01:00 	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266775044+01:00 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266778934+01:00 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266782804+01:00 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266790824+01:00 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266794744+01:00 	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266798604+01:00 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266802424+01:00 	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266806344+01:00 	at java.base/java.lang.Thread.run(Thread.java:829)
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266810114+01:00 
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266813824+01:00 	at com.linkedin.venice.controller.VeniceHelixAdmin.addVersion(VeniceHelixAdmin.java:2414) ~[venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266817964+01:00 	at com.linkedin.venice.controller.VeniceHelixAdmin.incrementVersionIdempotent(VeniceHelixAdmin.java:2484) ~[venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266822204+01:00 	at com.linkedin.venice.controller.Admin.incrementVersionIdempotent(Admin.java:185) ~[venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266826564+01:00 	at com.linkedin.venice.controller.server.CreateVersion.lambda$emptyPush$6(CreateVersion.java:652) ~[venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266830674+01:00 	at spark.RouteImpl$1.handle(RouteImpl.java:72) ~[venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266834524+01:00 	at spark.http.matching.Routes.execute(Routes.java:61) ~[venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266838454+01:00 	at spark.http.matching.MatcherFilter.doFilter(MatcherFilter.java:134) ~[venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266842434+01:00 	at spark.embeddedserver.jetty.JettyHandler.doHandle(JettyHandler.java:50) ~[venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266846534+01:00 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1598) ~[venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266850554+01:00 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266855034+01:00 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266859024+01:00 	at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266863064+01:00 	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266867024+01:00 	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) [venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266870784+01:00 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) [venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266874824+01:00 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266878914+01:00 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266887274+01:00 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266891464+01:00 	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266895554+01:00 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266899644+01:00 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266903644+01:00 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266907724+01:00 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266911824+01:00 	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) [venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266915974+01:00 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266920184+01:00 	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266947874+01:00 	at java.lang.Thread.run(Thread.java:829) [?:?]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266954584+01:00 Caused by: com.linkedin.venice.exceptions.VeniceException: After waiting for 60000ms, resource assignment for: test-store-persons-3_v1 timed out, strategy=WAIT_ALL_REPLICAS, replicationFactor=1, reason=not yet in EXTERNALVIEW
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266959454+01:00 	at com.linkedin.venice.controller.VeniceHelixAdmin.waitUntilNodesAreAssignedForResource(VeniceHelixAdmin.java:4412) ~[venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266963774+01:00 	at com.linkedin.venice.controller.VeniceHelixAdmin.addVersion(VeniceHelixAdmin.java:2368) ~[venice-controller-all.jar:unspecified]
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.266967904+01:00 	... 26 more
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.268215044+01:00 2023-03-16 13:24:03 INFO [InstanceMessagesCache] [HelixController-pipeline-task-venice-cluster0-(50b07205_TASK)] START: InstanceMessagesCache.refresh()
venice-controller-6854bffcc-cjrxp venice 2023-03-16T14:24:03.268295634+01:00 2023-03-16 13:24:03 INFO [InstanceMessagesCach

Felix GV

03/16/2023, 1:29 PM

This is definitely a problem… it means Helix failed to assign the partition-replicas of the new resource (i.e. the new store-version).

Enrico Olivelli

03/16/2023, 1:30 PM

Hold on, in this case the only venice-server was down 🤦

✅ 1

Enrico Olivelli

03/16/2023, 1:30 PM

I am starting it

Felix GV

03/16/2023, 1:30 PM

We see that every once in a while when Helix or ZK are overloaded. If it happens consistently then it may be a connectivity issue.

Enrico Olivelli

03/16/2023, 1:31 PM

I had upgraded the docker images versions, but the venice-server docker images failed to land into the registry, so the pod never started

Enrico Olivelli

03/16/2023, 1:34 PM

Now that the server is up and running the operation completes 👏

🚀 1

Enrico Olivelli

03/16/2023, 1:34 PM

Next step is to write data 😁

Felix GV

03/16/2023, 1:49 PM

Woohoo!

Felix GV

03/16/2023, 1:50 PM

I’m always interested in improving error messages. While the message you got is accurate, it would be interesting to think of a way to detect the condition where there are no servers at all, and maybe replace or supplement the error message with that extra detail. It’s just fine tuning but anyway.

Enrico Olivelli

03/16/2023, 2:29 PM

Writing to Venice works also on K8S 🍾 (incremental writes)

🎉 2

🍾 2

💖 2

Enrico Olivelli

03/16/2023, 2:29 PM

I am going to publish my code within the end of the week

Felix GV

03/16/2023, 2:35 PM

Nice! Congrats on getting this far 😄

Felix GV

03/16/2023, 2:35 PM

Would be really great if you can share your k8s/helm stuff when you’re ready. We’re also interested in trying that out internally.

Enrico Olivelli

03/16/2023, 2:38 PM

sure thing!

Enrico Olivelli

03/16/2023, 2:38 PM

I still have everything on a temporary private repo, the plan for next week is to open a github.com/datastax/venice-helm-chart repo

Felix GV

03/16/2023, 2:39 PM

Wow, super cool!

Felix GV

03/16/2023, 2:39 PM

We’ll link to it when it’s out.

Enrico Olivelli

03/16/2023, 8:43 PM

Please hold on. It is still not mature, and it uses some docker images baked by myself (I will send new patches tomorrow)

Enrico Olivelli

03/16/2023, 8:43 PM

also basic k8s stuff is not configured, like volumes

Enrico Olivelli

03/16/2023, 8:43 PM

I will ping here when we can advertise about it

Felix GV

03/16/2023, 9:07 PM

Yeah SG, let's hold on until it's in good shape. Thank you!

2 Views

Open in Slack

Previous Next