https://pinot.apache.org/ logo
Join Slack
Powered by
# troubleshooting
  • e
    qq - I can increase the number of servers in k8s and then call rebalance. How can I decrease the number of servers and move data off of servers, i.e. halving the cluster instead of doubling?
  • k
    same
  • k
    untag the servers
  • e
    How do you untag them? Is it an api call?
  • n
    i dont think there’s an api call to untag servers. but you can go to zooinspector and under PinotCluster/CONFIGS/PARTICIPANT/ServerName and change the tag name. then call rebalance
  • n
    oh there is one 
    /instances/{instanceName}"
  • n
    check under Instances in swagger
    👍 1
  • d
    Hello everybody! I found a good server with 32 cores and 128 gb of ram, it is not the definitive solution because I need another one in case of failover, but for now it's okay. Working on a single machine like the one I mentioned, how many brokers / servers should I start?
  • p
    Hi, I believe I asked this question previously, sorry I don’t have my slack history unfortunately. TEXT_MATCH/REGEXP_LIKE with NOT seems to be not working (I am using code from master) for example:
    Copy code
    select *
from table 
where not TEXT_MATCH(column, 'term')
    returns following exception
    Copy code
    pinot error: {'errorCode': 150, 'message': 'PQLParsingError:\n' 'java.lang.IllegalArgumentException: No enum constant ' 'org.apache.pinot.pql.parsers.pql2.ast.FilterKind.NOT\n' '\tat java.lang.Enum.valueOf(Enum.java:238)\n' '\tat ' 'org.apache.pinot.pql.parsers.pql2.ast.FilterKind.valueOf(FilterKind.java:21)\n' '\tat ' 'org.apache.pinot.parsers.utils.ParserUtils.getFilterKind(ParserUtils.java:83)\n' '\tat ' 'org.apache.pinot.pql.parsers.PinotQuery2BrokerRequestConverter.traverseFilterExpression(PinotQuery2BrokerRequestConverter.java:250)\n' '\tat ' 'org.apache.pinot.pql.parsers.PinotQuery2BrokerRequestConverter.convertFilter(PinotQuery2BrokerRequestConverter.java:178)\n' '\tat ' 'org.apache.pinot.pql.parsers.PinotQuery2BrokerRequestConverter.convert(PinotQuery2BrokerRequestConverter.java:58)\n' '\tat ' 'org.apache.pinot.sql.parsers.CalciteSqlCompiler.compileToBrokerRequest(CalciteSqlCompiler.java:34)\n' '\tat ' 'org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:186)\n' '\tat ' 'org.apache.pinot.broker.api.resources.PinotClientRequest.processSqlQueryPost(PinotClientRequest.java:155)\n' '\tat sun.reflect.GeneratedMethodAccessor11.invoke(Unknown ' 'Source)\n' '\tat ' 'sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n' '\tat java.lang.reflect.Method.invoke(Method.java:498)\n' '\tat ' 'org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)\n' '\tat ' 'org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)'}
    is there a alternative to NOT?
  • p
    (Created this for tracking, so I don’t loose the info again: https://github.com/apache/incubator-pinot/issues/5797)
  • s
    we can use NOT as part of search expression inside text_match
  • p
    This didn’t seem to work
  • k
    Can you paste the query with NOT in the expression
  • s
    it looks like lucene syntax doesn't support NOT with a single term. So, a better way to support what @Pradeep wants to do is to NOT TEXT_MATCH
  • s
    Currently SQL (and even PQL) only handle NOT IN AFAIK
  • p
    Just as an example, so this didn’t work 
    text_match(column, 'NOT term2')
    but this does 
    text_match(column, 'term1 AND  NOT term2')
  • s
    yeah that's because NOT is not supported with single terms or phrases
  • s
    you should use text_match(column, 'term1 NOT term2')
  • s
    you can also do text_match(column, '/.*/ NOT term') -- but this will be slow so a better way to handle single term NOT expressions is to handle NOT text_match(....) syntax
  • p
    makes sense thanks @Sidd I think I can go ahead with this workaround for now. Yeah, but would be nice to have NOT expression support for regexp_like/text_match
  • s
    Yes. I will add soon
  • p
    thanks a lot
  • o
    hi, Is 
    isSingleValueField
    value of 
    BYTES
    data type is always true?
  • a
    Copy code
    java.lang.IllegalArgumentException: Illegal memory allocation -4 for segment wallet_success_rate__0__3__20200805T1112Z column wallet_success_rate__0__3__20200805T1112Z:amount.sv.unsorted.fwd
	at shaded.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
	at org.apache.pinot.core.io.readerwriter.RealtimeIndexOffHeapMemoryManager.allocate(RealtimeIndexOffHeapMemoryManager.java:77) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
	at org.apache.pinot.core.realtime.impl.forward.FixedByteSVMutableForwardIndex.addBuffer(FixedByteSVMutableForwardIndex.java:208) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
	at org.apache.pinot.core.realtime.impl.forward.FixedByteSVMutableForwardIndex.<init>(FixedByteSVMutableForwardIndex.java:77) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
	at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.<init>(MutableSegmentImpl.java:263) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
	at org.apache.pinot.tools.realtime.provisioning.MemoryEstimator.getMemoryForConsumingSegmentPerPartition(MemoryEstimator.java:280) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
	at org.apache.pinot.tools.realtime.provisioning.MemoryEstimator.estimateMemoryUsed(MemoryEstimator.java:223) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
	at org.apache.pinot.tools.admin.command.RealtimeProvisioningHelperCommand.execute(RealtimeProvisioningHelperCommand.java:227) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
	at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:156) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
	at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:168) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
  • d
    I want to eventually support backfilling pinot data for specific customers. I'd like to not have to backfill all data in the time period. Is there an easy way to modify the SegmentNameGenerator to support this? I see how to do it if I implement my own SegmentNameGenerator (basically by adding in my customer id to segments).
  • k
    yeah, thats a smart solution.
  • e
    Is there any performance difference in realtime vs offline segments if the segments have star tree, text, inverted and sorted indexes? Or do realtime segments not have all of those indexes?
  • e
    I see the realtime text and star tree indexes
  • m
    Flushed realtime segments share the same format as offline segments. Consuming realtime segments also share the same format for the most part, but I can't think of a reason that could lead to big perf degradation.
    👍 1
  • m
    One way offline can perform better is if you pre-aggregate in your offline pipeline
1...124125126...166Latest