Should they match for a successful replacement of the segments? If yes, how do we configure the naming in ``RealtimeToOfflineSegmentsTask` /

MergeRollupTask

?

Hi Everyone. Anyone here has connected Tableau to Pinot?

is there a quicker way to create schema from data files. we have about 1200 columns and wondering if there is any jar that automatically creates a schema file for pinot

Hi, just wondering, are there any plans to support table aliases/views in pinot? Or the ability to rename tables?

Hey, how can I create an array from a list of values in Pinot? Calcite SQL grammar has this value constructor:

ARRAY '[' value [, value ]* ']'	Creates an array from a list of values.

Hi, I have a table

{
  "schemaName": "balance",
  "dimensionFieldSpecs": [
    {
      "name": "account_id",
      "dataType": "STRING"
    },
    {
      "name": "token_id",
      "dataType": "STRING"
    },
    {
      "name": "balance",
      "dataType": "LONG"
    }
  ],
  "dateTimeFieldSpecs": [
    {
      "name": "consensus_timestamp",
      "dataType": "LONG",
      "format": "1:NANOSECONDS:EPOCH",
      "granularity": "1:NANOSECONDS"
    }
  ]
}

I am trying to find out the most recent balance for each account_id & token_id pairs. I can't seem to see a easy way to do that without many queries?

Need help to build project locally

pranay.sankpal@ITIN000509-MAC pinot % java -version                             
openjdk version "1.8.0_302"
OpenJDK Runtime Environment Corretto-8.302.08.1 (build 1.8.0_302-b08)
OpenJDK 64-Bit Server VM Corretto-8.302.08.1 (build 25.302-b08, mixed mode)

pranay.sankpal@ITIN000509-MAC pinot % mvn -version                              
Apache Maven 3.8.5 (3599d3414f046de2324203b78ddcf9b5e4388aa0)
Maven home: /usr/local/Cellar/maven/3.8.5/libexec
Java version: 18.0.1, vendor: Homebrew, runtime: /usr/local/Cellar/openjdk/18.0.1/libexec/openjdk.jdk/Contents/Home
Default locale: en_IN, platform encoding: UTF-8
OS name: "mac os x", version: "10.15.7", arch: "x86_64", family: "mac"

pranay.sankpal@ITIN000509-MAC pinot % mvn install package -DskipTests -Pbin-dist
[ERROR] Failed to execute goal com.diffplug.spotless:spotless-maven-plugin:2.9.0:check (default) on project pinot-spi: Execution default of goal com.diffplug.spotless:spotless-maven-plugin:2.9.0:check failed: java.lang.reflect.InvocationTargetException: class com.google.googlejavaformat.java.RemoveUnusedImports (in unnamed module @0x687d31a9) cannot access class com.sun.tools.javac.util.Context (in module jdk.compiler) because module jdk.compiler does not export com.sun.tools.javac.util to unnamed module @0x687d31a9 -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.diffplug.spotless:spotless-maven-plugin:2.9.0:check (default) on project pinot-spi: Execution default of goal com.diffplug.spotless:spotless-maven-plugin:2.9.0:check failed: java.lang.reflect.InvocationTargetException
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:306)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:211)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:165)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:157)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:121)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:127)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:294)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:104)
    at java.lang.reflect.Method.invoke (Method.java:577)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution default of goal com.diffplug.spotless:spotless-maven-plugin:2.9.0:check failed: java.lang.reflect.InvocationTargetException
[ERROR] 
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] <http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException>
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :pinot-spi

i'm new to Java and this project, small help will be appreciated 🙂

Is there a way of doing MAX() on a long timestamp column returning a long not double?

Is there a index that helps performance for LASTWITHTIME() ?

Is it possible to configure minions to upload segments directly to the segment store like you can for servers as described here https://docs.pinot.apache.org/operators/operating-pinot/decoupling-controller-from-the-data-path ? We are seeing timeouts when our minions try to upload segments to the controller during the realtime to offline job

Hi Team. Is there a namespace concept in Pinot? We would like to park tables under different namespaces for access control. If namespaces are not available, may I know how this can be achieved? Should be use some metadata tagging?

Hi, every day a docker image is built and published from the latest snapshot (e.g. 0.11.0-SNAPSHOT-2fe8953-20220803). I wonder if there is a place where the snapshot tar.gz-s are published. Does anyone know?

Hellos, is there a feature/issue which is being worked on which handles SQL injection vulnerability in Pinot as mentioned here https://blog.doyensec.com/2022/06/09/apache-pinot-sqli-rce.html#pinot-sql-syntax--injection-basics

is release 0.11 out, see updated docs but no binary

Hi team, we're looking at dynamic filtering for the Presto-Pinot connector and see that we can generate proper logical plans to pushdown the dynamic filters, specifying

dynamicFilterAssignments

, but doesn't look like the plan is able to take into account or pass this information into the connector. Is there a way to order the source nodes or pass this information down, or what would we need to build in support for this?

Pinot is creating all these directories for the segments of about 450 MB each. Do we need to keep it while spark submit load job is still in progress or can i delete them before job completion.. wondering if its still needed in further steps.. its consuming too much space on disk especially for local mode run

I also see a issue where if the previously executed spark submit job is terminated before completion the staging directory is not deleted and if we rerun the job old staged files are also loaded in tables creating bad segments.. shouldn’t staging directory be deleted each time before job runs? Also deleting the table doesnt delete the underlying data directory causing data duplication when you recreate the table and load it

what's a good channel to discuss more system / platform design oriented questions? here? https://www.startree.ai/blogs/real-time-analytics-at-scale-solving-the-trade-off-problem read this last night, i have questions - i am intrigued but skeptical that things will pan out well with the approach proposed here where we eschew data modelling / preparation and just rely on indexing in Pinot

What is the best practice on to dictionary or not to dictionary string id columns with billions of unique values? I have a non-dictionary column of id strings like

"0.0.89748-1612413398-472232185"

and

select * where id = '...' performance

is horrible. Just times out after 10 seconds. I assume I need to add a reverse index, the question is should it be dictionary or non-dictionary?

Hi everyone, We are happy to announce that the new multi-stage query engine (called “v2” engine from now on) is entering Beta testing phase. The new v2 engine comes with multi-stage DAG query plan execution capability and many new query features have been added such as inner join, multiple joins, and mix executions with joins and group-bys. Please feel free to try it out by following the instructions below: 1. build Pinot normally on latest master. 2. run Pinot with the v2 engine quickstart:

bin/pinot-admin.sh Quickstart -type MULTI_STAGE

3. run queries by checking the v2 engine selection box (see attachments) We would like to also invite you to join the #C03Q4A11GC9 to discuss the future of the new v2 engine to • share real-world use cases and stories; • file bug reports and other issues/comments during try-out; • collaborate to add more SQL semantic features & performance improvement patches. Thank you!

Hi everyone! 📣 As our community grows and expending around the globe, we're going to have more events to bring us all together, to meet and share knowledge with each other. In the next few months, we'll be having meetups in EMEA, NAMAR, LATIN AMERICA, and INDIA. 🌎 I'd LOVE to see new faces speaking! I'm encouraging you warmly to step forward and do your first step in the direction of speaking at events if you are thinking about it*.* If you're facing Imposter Syndrome (like many of us), or not sure what to speak about- just start with the first move and send me a DM. The rest will follow ❤️

Can the partial upsert be conditional? For example, set the column’s partialUpsertStrategies as OVERWRITE, but only overwrite the existing value when the event current being ingested has a non null value for the column.

I'm curious about how segments are organized internally. From what I've heard, a segment is stored as a single binary file that includes all of the columns and indices for a particular table. I've also heard that StarTree indices are stored as separate binary files (1 file per StarTree Index) but are still considered to be part of the segment. Does anyone have any additional information or resources on how the binary file is organized or used?

Hey everyone👋, I’m Johnny and I lead the Apache Pinot project team at Slack S Cross posting this job opportunity here for your reference: https://apache-pinot.slack.com/archives/C034Z8216JJ/p1660076901294319

hi , is there a way to set

count(a) / count(b)

to 0 when

count(b) = 0

instead of

NaN

or

Infinity

?

Nan

and

Infinity

isn’t really sort friendly.

org.apache.pinot.shaded.org.reflections.ReflectionsException: could not get type for name org.apache.bsf.util.BSFEngineImpl at org.apache.pinot.shaded.org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-dcedf480af9fb63fb66b07bdbef9a099c653dbdd] at org.apache.pinot.shaded.org.reflections.Reflections.expandSuperTypes(Reflections.java:381) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-dcedf480af9fb63fb66b07bdbef9a099c653dbdd] at org.apache.pinot.shaded.org.reflections.Reflections.<init>(Reflections.java:126) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-dcedf480af9fb63fb66b07bdbef9a099c653dbdd] at io.swagger.jaxrs.config.BeanConfig.classes(BeanConfig.java:288) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-dcedf480af9fb63fb66b07bdbef9a099c653dbdd] at io.swagger.jaxrs.config.BeanConfig.scanAndRead(BeanConfig.java:250) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-dcedf480af9fb63fb66b07bdbef9a099c653dbdd] at io.swagger.jaxrs.config.BeanConfig.setScan(BeanConfig.java:231) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-dcedf480af9fb63fb66b07bdbef9a099c653dbdd] at org.apache.pinot.minion.MinionAdminApiApplication.setupSwagger(MinionAdminApiApplication.java:93) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-dcedf480af9fb63fb66b07bdbef9a099c653dbdd] at org.apache.pinot.spi.utils.PinotReflectionUtils.runWithLock(PinotReflectionUtils.java:113) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-dcedf480af9fb63fb66b07bdbef9a099c653dbdd] at org.apache.pinot.minion.MinionAdminApiApplication.start(MinionAdminApiApplication.java:77) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-dcedf480af9fb63fb66b07bdbef9a099c653dbdd] at org.apache.pinot.minion.BaseMinionStarter.start(BaseMinionStarter.java:247) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-dcedf480af9fb63fb66b07bdbef9a099c653dbdd] at org.apache.pinot.tools.service.PinotServiceManager.startMinion(PinotServiceManager.java:186) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-dcedf480af9fb63fb66b07bdbef9a099c653dbdd] at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:101) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-dcedf480af9fb63fb66b07bdbef9a099c653dbdd] at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:278) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-dcedf480af9fb63fb66b07bdbef9a099c653dbdd] at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:304) [pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-dcedf480af9fb63fb66b07bdbef9a099c653dbdd] at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:278) [pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-dcedf480af9fb63fb66b07bdbef9a099c653dbdd] Caused by: java.lang.ClassNotFoundException: org.apache.bsf.util.BSFEngineImpl at jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) ~[?:?] at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[?:?] at java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[?:?] at org.apache.pinot.shaded.org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-dcedf480af9fb63fb66b07bdbef9a099c653dbdd] ... 14 more could not get type for name jline.console.completer.CandidateListCompletionHandler from any class loader org.apache.pinot.shaded.org.reflections.ReflectionsException: could not get type for name jline.console.completer.CandidateListCompletionHandler at org.apache.pinot.shaded.org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-dcedf480af9fb63fb66b07bdbef9a099c653dbdd] at org.apache.pinot.shaded.org.reflections.Reflections.expandSuperTypes(Reflections.java:381) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-dcedf480af9fb63fb66b07bdbef9a099c653dbdd] at org.apache.pinot.shaded.org.reflections.Reflections.<init>(Reflections.java:126) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-dcedf480af9fb63fb66b07bdbef9a099c653dbdd] at io.swagger.jaxrs.config.BeanConfig.classes(BeanConfig.java:288) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-dcedf480af9fb63fb66b07bdbef9a099c653dbdd] at io.swagger.jaxrs.config.BeanConfig.scanAndRead(BeanConfig.java:250) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-dcedf480af9fb63fb66b07bdbef9a099c653dbdd]

Hi fellows, Have you guys seen any documentation about using PySpark to ingest into an offline table? if yes would be able to send me the link?

it seems like pinot does not have google pub/sub as a streaming source because it doesn't fit into the stream model pinot uses with offsets. however, google also offers pubsub lite that is compatible with the kafka api. i think it would require some refactoring since the kafkaconsumerfactory doesn't construct the kafka consumer directly, but would something like this be possible?

Getting

failed to index value with DICTIONARY

when ingesting avro-formatted data from Kafka. Any ideas? Using KafkaConfluentSchemaRegistryAvroMessageDecoder.

Hi! Question: What would be the consequences to creating many small segments, say via the segment upload API, say one every 10 seconds, for many months?