Hi dev-team, I have moved this discussion from the general channel but I wanted to follow up on this issue discussion - particularly about the amount of effort involved in supporting Apache Datasketches in Pinot from storage, aggregation, indexing and query. Our use case relies on us being able to ingest pre-built sketches as well as create sketches from raw data during pre-aggregation (rollup in Druid). As a newcomer, I am willing to help with this should the changes be approachable.

Hi Folks, I have a small fix for pinot-spark-connector here. Can someone please take a look? Thanks

Hi ! we use the pulsar connector to feed several realtime tables on Pinot. We find that the connector creates a lot of subscriptions on Pulsar which are never cleaned (hundreds or thousands per day per topic, depending on the events flow). We are forced to configure Pulsar to clean up unused subscriptions quickly, but I don't think that's a good solution. Is this behavior normal? Thank you !

Hi Folks, is there a direct way( or a query) to check if a subset of values exist in a pinot multi value column without checking each element separately? eg. select * from a where test_tags in (‘a’, ‘b’) I want both (‘a’,‘b’) to exist in the column. test_tags can contain more values as well but, (‘a’, ‘b’) should be a subset.

A Bug? There are two similar methods in org.apache.pinot.plugin.filesystem.HadoopPinotFS

@Override
  public void copyFromLocalFile(File srcFile, URI dstUri)
      throws Exception {
    if (srcFile.isDirectory()) {
      throw new IllegalArgumentException(srcFile.getAbsolutePath() + " is a direactory");
    }
    _hadoopFS.copyFromLocalFile(new Path(srcFile.toURI()), new Path(dstUri));
  }

  public void copyFromLocalDir(File srcFile, URI dstUri)
      throws Exception {
    Path srcPath = new Path(srcFile.toURI());
    if (!_hadoopFS.isDirectory(srcPath)) {
      throw new IllegalArgumentException(srcFile.getAbsolutePath() + " is not a directory");
    }
    _hadoopFS.copyFromLocalFile(srcPath, new Path(dstUri));
  }

I plan to cut a new Pinot release on Monday, let me know if any pending pr or features you want to get in

hey folks, I’m trying to get my dev env set up for the first time and following the instructions here: https://docs.pinot.apache.org/developers/developers-and-contributors/code-setup#maven When running the

mvn

command to get things bootstrapped, I keep running into an issue with checkstyle:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.1.2:check (checkstyle) on project pinot-common: You have 2 Checkstyle violations. -> [Help 1]

This is being run on

apache/pinot:master

. Is there a maven command/routine set up to auto-fix violations? I did try a few things I found online for auto-fixing checkstyle[1] but it hasn’t been fruitful. This is blocking me from building from source and developing locally. Any suggestions? [1] https://docs.openrewrite.org/tutorials/running-recipes/automatically-fix-checkstyle-violations

Hello, we are planning to use Pinot with Snowflake. This is the use case: We have a service that is intensively querying Snowflake, to get similar data in near real time, which is costing, and we would like to Add Pinot, to avoid those intensives queries. Is Pinot the proper solution to do that ? Is there any Snowflake connector to start playing with it ? Thank you

Hi 😉 Quick question about interest about a PR or not , and if not a way to easly integrate my changes with future master build. I’ve modified tranformFunction JsonPathString(Object,String,String) from

@ScalarFunction(nullableParameters = true)
public static String jsonPathString(@Nullable Object object, String jsonPath, String defaultValue) {
  try {
    Object jsonValue = jsonPath(object, jsonPath);
    if (jsonValue instanceof String) {
      return (String) jsonValue;
    }
    return jsonValue == null ? defaultValue : JsonUtils.objectToString(jsonValue);
  } catch (Exception ignore) {
    return defaultValue;
  }
}

to

@ScalarFunction(nullableParameters = true)
public static String jsonPathString(@Nullable Object object, String jsonPath, String defaultValue) {
  try {
    Object jsonValue = jsonPath(object, jsonPath);
    if ((jsonValue instanceof String) && (!String.valueOf(jsonValue).isBlank())) {
      return (String) jsonValue;
    }
    return jsonValue == null || String.valueOf(jsonValue).isBlank()
        ? defaultValue : JsonUtils.objectToString(jsonValue);
  } catch (Exception ignore) {
    return defaultValue;
  }
}

Doing that allow me to set a default value on the blank string also. Thats can be helpfull on many use cases. Is that interesting to get that as a PR ? If not do I have a way to surcharge this with a provided jar or something easy to integrate ?

Is it possible to run pinot integrations tests in Intellij? I am able to run them successfully with mvn, but when running via the Intellij test runner they fail while starting the controller due to

java.lang.reflect.InaccessibleObjectException: Unable to make field private static final long java.util.Collections$SingletonList.serialVersionUID accessible: module java.base does not "opens java.util" to unnamed module @78e94dcf

Hey folks, I’m trying to run tests in a debugger (via intellij), but I’m getting a specific conflict about jackson JSON module coming from pinot-connectors (not what I’d like to test, but presumably part of what’s being built in order to run the tests:

java: incompatible types: org.apache.pinot.shaded.com.fasterxml.jackson.databind.JsonNode cannot be converted to com.fasterxml.jackson.databind.JsonNode

Any thoughts on a fix?

hello folks, I followed https://docs.pinot.apache.org/basics/getting-started/public-cloud-examples/aws-quickstart to create an EKS cluster but the version proposed for creating the cluster (1.16), is not supported.

Error: invalid version, 1.16 is no longer supported, supported values: 1.20, 1.21, 1.22, 1.23, 1.24

. Is it necessary to update the documentation ? or is it related to my local environment ?

this one is ready review, this helps with queries which match lots of segments so end up doing a lot of dictionary lookups https://github.com/apache/pinot/pull/10044

A quick question about Flink Pinot connector: Is this only for offline table? Does it support upsert table?

Hi all! Created my 1st PR. It's just a simple fix to the Controller to handle invalid table configs. thanks for the review. https://github.com/apache/pinot/pull/10080

hello all, hope you are doing well! I would like to use aggregation functions in Pinot, such as dense_rank() or rank() but I am not able to find any documentation about that. Is it supported in Pinot ?

Hi, I'm planning to remove the support for legacy padding character

'%'

which can cause unnecessary overhead, and complicates the code. We have enforced the padding to be

'\0'

about 5 years ago in this PR, so this change can only affect the very early user of Pinot (probably only LinkedIn). @Sidd @Subbu Subramaniam Let me know if you have concern on this

Is anybody seeing this type of error in recent builds:

java: cannot find symbol

symbol:   method eq(long)

location: variable rangeBitmap of type org.roaringbitmap.RangeBitmap

This started as I rebased today

Hi quick question of a cleaner way to provide to pinot my custom

SegmentPurgerFactory

Actually I do some edit on the

BaseMinionStarter

and then compile pinot to deploy it. But I would like to get an easy way to integrate my own business logic in the pinot code base. Is there a way to override it using the plugin mecanism like custom UDF ?

Can I get a review for a small fix in pinot-spark-connector: https://github.com/apache/pinot/pull/10209 ? This addresses some memory issue on Spark executor when scanning large segments via GRPC server.

Hey folks, has there ever been consideration for using gradle over maven as a build tool, or to use some form of build caching in maven so that rebuilding the project locally could be done more quickly, therefore allowing one to iterate more quickly? I should say that I may just be doing something wrong. Is there a faster way to rebuild after small changes (ex. a typo in one file)?

are there any guides on how to get the unit tests to run? on my mac I've got an artefact that doesn't resolve (

io.grpc:protoc-gen-grpc-java:exe:osx-aarch_64:1.41.0

) and on linux it's trying to download and run a non-static binary for protoc which breaks because I'm on nixos

could do with some feedback if I'm on the right track with this https://github.com/apache/pinot/pull/10288 adding theta sketch support on startree indexes

hellos, We tried exploring pinot-spark-connector but it does not support https protocol as mentioned here. Can I add a github issue around this and possibly pick up implementation for it, if its not something that is there in roadmap

Hey all, we're testing V2 support and need to be able to pass the useMultistageEngine option in the connection string. I've created a PR (https://github.com/apache/pinot/pull/10292) to allow this.

hey, can someone please approve the workflow for this PR - https://github.com/apache/pinot/pull/10293 It's a short improvement for not getting NPE during quickstart.

I looked at the integration tests for realtime tables and seems like we don't have any integration test for for tables with columns with dataType JSON. Did I miss any test we may be doing for this? Should we file an issue to add support for this?

Hello pinot dev, I was unable to build the project on my arm macbook so I submitting an MR to fix the issue. https://github.com/apache/pinot/pull/10312 CC @Amol Mathur @Mayank

can some contributor allow this PR to run the CI? https://github.com/apache/pinot/pull/10304