https://pinot.apache.org/ logo
Join Slack
Powered by
# pinot-dev
  • g
    Hello! I’m using the pinot client and am getting this exception
    Copy code
    Caused by: java.lang.NumberFormatException: For input string: "20619889534"
	at java.lang.NumberFormatException.forInputString(Unknown Source) ~[?:?]
	at java.lang.Integer.parseInt(Unknown Source) ~[?:?]
	at java.lang.Integer.parseInt(Unknown Source) ~[?:?]
	at org.apache.pinot.client.AbstractResultSet.getInt(AbstractResultSet.java:58) ~[pinot-java-client-0.7.1.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
	at org.apache.pinot.client.PinotResultSet.getInt(PinotResultSet.java:263) ~[pinot-jdbc-client-0.7.1.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
    and I believe it it because 20,619,889,534 is larger than the maximum integer: 2,147,483,647 Is there a solution on my side so I can get around this?
    m
    • 2
    • 6
  • s
    Hi Team, I got to know that Pinot provides support for encryption of data stored in deepstore(HDFS). A crypter needs to be provided for this. But I am not able to figure out where to provide the crypter details , can anyone help ?
    k
    t
    • 3
    • 6
  • g
    Hi! I’m using the pinot jdbc client and I’m getting a timeout:
    Copy code
    java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Request timed out to {broker url} of 60000 ms
	at org.apache.pinot.client.JsonAsyncHttpPinotClientTransport$BrokerResponseFuture.get(JsonAsyncHttpPinotClientTransport.java:173) ~[pinot-java-client-0.7.1.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
	at org.apache.pinot.client.JsonAsyncHttpPinotClientTransport$BrokerResponseFuture.get(JsonAsyncHttpPinotClientTransport.java:152) ~[pinot-java-client-0.7.1.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
	at org.apache.pinot.client.JsonAsyncHttpPinotClientTransport$BrokerResponseFuture.get(JsonAsyncHttpPinotClientTransport.java:123) ~[pinot-java-client-0.7.1.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
	at org.apache.pinot.client.JsonAsyncHttpPinotClientTransport.executeQuery(JsonAsyncHttpPinotClientTransport.java:102) ~[pinot-java-client-0.7.1.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
	at org.apache.pinot.client.Connection.execute(Connection.java:127) ~[pinot-java-client-0.7.1.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
	at org.apache.pinot.client.Connection.execute(Connection.java:96) ~[pinot-java-client-0.7.1.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
	at org.apache.pinot.client.PreparedStatement.execute(PreparedStatement.java:72) ~[pinot-java-client-0.7.1.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
	at org.apache.pinot.client.PinotPreparedStatement.executeQuery(PinotPreparedStatement.java:193) ~[pinot-jdbc-client-0.7.1.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
	at org.apache.pinot.client.PinotPreparedStatement.execute(PinotPreparedStatement.java:160) ~[pinot-jdbc-client-0.7.1.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
    Where is the 
    60000
    ms being set? Can I increase it? Thanks!
    x
    k
    • 3
    • 24
  • x
    Hi Pinot devs, I’ve finished this issue https://github.com/apache/pinot/issues/7309 to clean up our code repo to reformat the code and fix the checkstyle errors and enabled checkstyle maven plugin. There are a few slight changes of the coding style, please re-import the intellij checkstyle xml file. Still working on eclipse xml.
    👍 2
    k
    • 2
    • 3
  • k
    @User - I’m wondering how thoroughly the classloader change (for Java9+ compatibility) was tested with Hadoop/Spark segment building code, when running with Java 8. Asking because we (and others) have run into class-not-found issues (for plugins) when trying to build segments with 0.8.0 using Hadoop and Spark on Java 8, and I see this change to 
    PluginClassLoader.java
    that I think you made:
    Copy code
    < import java.lang.reflect.Method;
39,47d37
<     Method method = null;
<     try {
<       method = URLClassLoader.class.getDeclaredMethod("addURL", URL.class);
< 
<     } catch (NoSuchMethodException e) {
<       //this should never happen
<       ExceptionUtils.rethrow(e);
<     }
<     method.setAccessible(true);
50c40,47
<         method.invoke(classLoader, url);
---
>         /**
>          * ClassLoader in java9+ does not extend URLClassLoader.
>          * If the class is not found in the parent classloader,
>          * it will be found in this classloader via findClass().
>          *
>          * @see <https://community.oracle.com/tech/developers/discussion/4011800/base-classloader-no-longer-from-urlclassloader>
>          */
>         addURL(url);
    @User I think you wrote the original code here, right?
    e
    k
    x
    • 4
    • 10
  • k
    thats how it is right?
    k
    • 2
    • 8
  • c
    Hello, Im looking into writing a custom kafka connector (used for connecting with a proprietary kafka service). What would be the best way of doing this and also being able to pull changes from the public repo? My first solution was to duplicate the pinot-kafka-2.0 module and adapt that, but this results in a lot of duplicate files and if the original module changes i'll have to reimplement those changes in mine. My changes aren't that big, only to KafkaPartitionLevelConnectionHandler and KafkaPartitionLevelStreamConfig, and a version bump for the kafka client to 2.5
    m
    • 2
    • 1
  • a
    Folks, I need some help. My latest PR adds JMH benchmarks, hence generates relevant classes. However, Pre commit is failing for the same with checkstyle failures. I thought checkstyle is excluded for /target? https://github.com/apache/pinot/pull/7405/checks?check_run_id=3595648311
    x
    • 2
    • 13
  • k
    We noticed that our automated running of the batch ingestion job (via 
    bin/pinot-admin.sh
    ) wasn’t stopping when batch ingest failed. In looking at 
    PinotAdministrator
    , it seems like you have to set the 
    pinot.admin.system.exit
    System property to 
    true
    for this to work. Any reason why 
    pinot-admin.sh
    shouldn’t be setting this to true if 
    JAVA_OPTS
    isn’t specified? E.g. something like
    Copy code
    if [ -z "$JAVA_OPTS" ] ; then
  ALL_JAVA_OPTS="-Xms4G -Dlog4j2.configurationFile=conf/log4j2.xml -Dpinot.admin.system.exit=true"
else
  ALL_JAVA_OPTS=$JAVA_OPTS
fi
    k
    m
    x
    • 4
    • 15
  • a
    Where will I find the code style sheet for Pinot in the repo?
    x
    • 2
    • 1
  • d
    @User not sure if youve seen my previous message. I'm going to raise a draft PR in the next couple of days with what alive got so far so you and anyone else can feed back
    k
    • 2
    • 5
  • x
    Copy code
    Hi all,

I plan to cut Pinot 0.9.0 release next Tuesday EOD (Nov 9) from current master branch.

Please let me know if you have any pending PR/feature want to be included.

Best,

Xiang
    a
    r
    • 3
    • 6
  • k
    Hi all - we’ve noticed when doing a big metadata push (1200 segments) that a lot of the time is spent downloading/expanding the segment from deep store (HDFS) to the local machine, so that the metadata file from the segment can be extracted and used to build the request to the controller. It should be possible to open a stream to the file and extract only that bit of the segment. Does this make sense as a reasonable enhancement?
    s
    r
    • 3
    • 20
  • k
    Can the master branch still be built using Java 1.8? Just pulled and tried 
    mvn clean install -DskipTests -Pbin-dist -T 4 -Djdk.version=8
    and got a strange error:
    Copy code
    [ERROR] pinot/pinot-common/src/test/java/org/apache/pinot/common/metrics/PinotMetricUtilsTest.java:[43,88] error: cannot access PinotMetricsRegistry
[ERROR]   class file for PinotMetricsRegistry not found
[ERROR] pinot/pinot-common/src/test/java/org/apache/pinot/common/metrics/PinotMetricUtilsTest.java:[77,76] error: incompatible types: PinotMetricsRegistry cannot be converted to org.apache.pinot.spi.metrics.PinotMetricsRegistry
[ERROR] pinot/pinot-common/src/test/java/org/apache/pinot/common/metrics/PinotMetricUtilsTest.java:[78,36] error: incompatible types: org.apache.pinot.spi.metrics.PinotMetricsRegistry cannot be converted to PinotMetricsRegistry
    x
    • 2
    • 2
  • k
    I’m working on a PR for https://github.com/apache/pinot/issues/7791. I need to modify 
    SegmentPushUtils
    , but noticed that this class is in the 
    pinot-segment-local
    sub-project, and the 
    org.apache.pinot.segment.local.utils
    package, even though it’s used by hadoop and spark batch, minion, etc. Seems like it should be moved someplace else, but not sure where - any advice?
    x
    • 2
    • 3
  • k
    Hi Team, I would like to contribute to Pinot but this is just a start for me in open source space, so can I club with someone and contribute. (Sorry in case this is not the right channel).
    m
    r
    • 3
    • 3
  • a
    Hi Pinot Team, Could you please confirm what is the impact of log4j Vulnerability in Pinot. I can see there are log4j 2.x dependencies also what is the plan to remediate it
    k
    k
    +4
    • 7
    • 14
  • k
    Got it.. i like your idea.. another option is it to make the push job or the controller code to be smarter? If no change in crc, then skip the segment
    k
    v
    s
    • 4
    • 7
  • m
    I think another issue here is that metadata push should not need to download at all. I think that is the case when segment is in final location already (controllers data dir). But it should also be the case when it is not (we should fix this).
    k
    • 2
    • 2
  • k
    Question about the right process for getting docs updated for a PR. I’d created https://github.com/apache/pinot/pull/7222, which contained information about the documentation that would need to be updated. This PR was eventually merged (on Aug 21st, 2021), but the docs were never updated. Was that my responsibility, once it got merged? If so, how would I time the update of the docs with a release that contained the change? E.g. there was a 0.8.0 release in August, but I don’t think it picked up my change, so it wasn’t until the 0.9.0 release (November 2021) that it would have been appropriate to edit the docs, right? Just trying to understand the appropriate process, thanks!
    r
    x
    j
    • 4
    • 12
  • a
    I think the failure started earlier this week -- I just investigated a PR build failure and it seems related? https://github.com/apache/pinot/runs/5091616493?check_suite_focus=true
    r
    • 2
    • 3
  • s
    This message was deleted.
    r
    s
    • 3
    • 2
  • s
    FYI, I'm going to start the process for release 0.10.0 tomorrow based on the commit 
    fd9c58a
    (#8270).
    🙏 8
    👍 3
    l
    • 2
    • 2
  • a
    Hi, in the GroupByKeyGenerator, under some circumstances, raw keys are used (by invoking getInternal method on the dictionary) instead of dictId. This does not go through the datafetcher and hence causes increased latency. Is there a way around this?
    m
    r
    • 3
    • 15
  • k
    Hey a quick question - Is there an Http API to do batch inserts into Pinot? Can we skip Kafka ingestion for ingesting real time batches of data using a direct API?
    m
    • 2
    • 5
  • d
    hi @User, @User thanks for reviewing this PR https://github.com/apache/pinot/pull/7358 I've now rebased it and moved the converters to the schema utils class. I've left a comment with regards regards to config that I thought it'd be easier to discuss over slack
    r
    • 2
    • 6
  • s
    @User @User the master branch should compile just fine now. The required pom changes were merged recently. Beyond that, I just made sure I used the right jdk (official openjdk 11 isn't available as such for M1 yet, so had to use https://www.azul.com/downloads/?version=java-11-lts&amp;os=macos&amp;architecture=arm-64-bit&amp;package=jdk). It compiled for me with this.
    k
    • 2
    • 4
  • d
    hey there, I've raised this issue which I've already started looking into. Once I've have some code to show I'll get back to you on this channel https://github.com/apache/pinot/issues/8527
    m
    • 2
    • 2
  • m
    Hello Team, I have raised a PR to support refreshing realtime segments which are completed. This will allow us to purge/modify the records in purge minion task. Please review the PR https://github.com/apache/pinot/pull/8587
    m
    • 2
    • 3
  • k
    https://github.com/apache/pinot/pull/8578 can someone help in reviewing this PR? Pending for last few days.
    🙏 1
    👍 1
    r
    • 2
    • 1
12345...30Latest