https://pinot.apache.org/ logo
Join Slack
Powered by
# troubleshooting
  • c

    Cesar

    10/15/2020, 7:40 PM
    I'm doing some profiling to understand what's the bottleneck of the current setup - which AFAICT is 'similar' to the one used by a client.
  • c

    Cesar

    10/15/2020, 7:40 PM
    I don't know if this latency that I'm observing is something that I should expect given the current setup of if I'm misconfiguring something
  • m

    Mayank

    10/15/2020, 7:41 PM
    It does not seem out of the ballpark, but there's definitely room for improvement
  • m

    Mayank

    10/15/2020, 7:43 PM
    BTW, IMO, if you have your client's setup (data/query), I think that would be more helpful in profiling/optimizing
  • c

    Cesar

    10/15/2020, 7:43 PM
    With this setup if I setup a harness client to send 10 queries concurrently I start seeing couple of seconds of latency. I was really surprised by that.
  • c

    Cesar

    10/15/2020, 7:44 PM
    Fully agree with that.. I'm trying to get that.
  • m

    Mayank

    10/15/2020, 7:46 PM
    Yeah, we have various optimizations for different cases. For example in your case of
    sum() on all records
    , star-tree index would work perfectly (as it will pre-compute some of the cubes).
  • b

    Buchi Reddy

    10/15/2020, 7:48 PM
    @Cesar FWIW, I've noticee the max GC pause to be 200ms in your settings. I've recently done some optimization around that by setting it to lower value like 20ms and that has improved query latencies. It depends on overall GC patterns though. If you want to confirm, analyze GC logs
  • c

    Cesar

    10/15/2020, 7:49 PM
    Thanks Buchi. I'll play with that.
  • x

    Xiang Fu

    10/21/2020, 4:10 PM
    @Sri Surya lets move the discussion here. What’s the environment you are running right now?
  • k

    Kishore G

    10/27/2020, 11:37 PM
    Does the new region have access to deep store?
  • e

    Elon

    10/27/2020, 11:37 PM
    Yes
  • k

    Kishore G

    10/27/2020, 11:38 PM
    What you suggested works
  • k

    Kishore G

    10/27/2020, 11:38 PM
    As long the instance ids are the same
  • k

    Kishore G

    10/27/2020, 11:39 PM
    If you want to be smart, you just need to copy zookeeper directory
  • k

    Kishore G

    10/27/2020, 11:39 PM
    And start same number of Pinot servers in the new cluster
  • k

    Kishore G

    10/27/2020, 11:39 PM
    And it will download the segments from gcs
    👍 1
  • e

    Elon

    10/27/2020, 11:40 PM
    Would we need downtime to copy zookeeper disks?
  • e

    Elon

    10/27/2020, 11:42 PM
    Or is it possible to add nodes to the zookeeper cluster in the new region and then remove nodes in the old region?
  • e

    Elon

    10/27/2020, 11:44 PM
    @Mayank was saying we can add new pinot servers in the new region and remove old pinot servers once the new ones are replicated to. Does that involve tagging?
  • m

    Mayank

    10/27/2020, 11:51 PM
    Yes, this ^^ will get you zero downtime, if that is important for you
  • e

    Elon

    10/28/2020, 12:13 AM
    Sounds good! Is that as simple as going to the zk explorer page and adding the tags?
  • e

    Elon

    10/29/2020, 8:00 PM
    Another migration question: our kafka cluster will be moving and offsets will be reset. How can we ensure that pinot keeps ingesting, is there a way to do this with no downtime?
  • m

    Mayank

    10/29/2020, 8:02 PM
    How many tables do you have in your cluster?
  • k

    Kishore G

    10/29/2020, 8:03 PM
    do you have offline table?
  • e

    Elon

    10/29/2020, 8:05 PM
    Yes
  • e

    Elon

    10/29/2020, 8:06 PM
    We have ~15 tables, all hybrid except 2 that are realtime only
  • e

    Elon

    10/29/2020, 8:06 PM
    This is staging, so we can test it out
  • e

    Elon

    10/29/2020, 8:11 PM
    I will see if we can set the offsets on the new cluster.
  • e

    Elon

    10/29/2020, 8:12 PM
    Is it possible to update a realtime table def to change the kafka broker url with no issues?
1...136137138...166Latest