Is that contributing to the long runtime (specially of integ tests) also, or is that separate issue?

that’s might be separate

long time integration may come from more integration tests

Long unit test is caused by connection timeout and stuck; long integration test is caused by running more queries with non-empty result: https://github.com/apache/incubator-pinot/pull/6181

Are we ok with the runtime though? And is it worth the additional coverage it is providing?

Do we have an increase in coverage due to this @User ?

We can further reduce the number of queries from 500 to 100, but AFAIK most of the time those integration tests took more time on spinning up the local cluster, especially when the hardware resource is limited

Any thoughts?

fix for current kafka tests: https://github.com/apache/incubator-pinot/pull/6786

Thanks to @User, we have a PR(https://github.com/apache/incubator-pinot/pull/6424) for java 11 upgrade. This is a backward incompatible upgrade which means we have to drop java8 support from here. Wanna collect feedback on this as well, Github Issue: https://github.com/apache/incubator-pinot/issues/6689

Sorry for spamming in various channels, but this is the last one 🙂 Any interest from Pinot-dev to present in ApacheCon: https://www.apachecon.com/acah2021/

Hi, I am running a query which involves AND , OR and with some filters on string and long values. It has basically 34Million rows , and querying(selecting few columns for an ID) takes almost 2 sec and numEntriesScannedInFilter(89Million) & numEntriesScannedPostFilter are bigger values. Can someone help me to understand, how come this many entries scanned in filter, where i am using Inverted index...?

@User, did we get any consensus on the 0.8.0 release timeline?

Here is the changelog since the last release from the master’s branch. I’m trying to come up with the list of new major features. If anyone has the feature to highlight, please update to this thread.

Updated the java11 PR to make the code compilable from both java 8 and java 11. Make tests to use java 11 and quickstart can be built and ran by java 8/10 https://github.com/apache/incubator-pinot/pull/6424

Hi, I have started exploring apache pinot, there are few query regarding schema of apache pinot. I want to understand how apache pinot work with Kafka topic that has AVRO schema (schema includes nested object, array of object etc..) because i didn't find any resource or example that shows how we can inject data with Kafka that has avro schema with it. As per my understanding apache pinot we have to provide flat schema or other option for nested Json object we can use transform function. Avro schema

{
  "namespace": "my.avro.ns",
  "name": "MyRecord",
  "type": "record",
  "fields": [
    {
      "name": "uid",
      "type": "int"
    },
    {
      "name": "somefield",
      "type": "string"
    },
    {
      "name": "options",
      "type": {
        "type": "array",
        "items": {
          "type": "record",
          "name": "lvl2_record",
          "fields": [
            {
              "name": "item1_lvl2",
              "type": "string"
            },
            {
              "name": "item2_lvl2",
              "type": {
                "type": "array",
                "items": {
                  "type": "record",
                  "name": "lvl3_record",
                  "fields": [
                    {
                      "name": "item1_lvl3",
                      "type": "string"
                    },
                    {
                      "name": "item2_lvl3",
                      "type": "string"
                    }
                  ]
                }
              }
            }
          ]
        }
      }
    }
  ]
}

Kafka Avro Message:

{
  "uid": 29153333,
  "somefield": "somevalue",
  "options": [
    {
      "item1_lvl2": "a",
      "item2_lvl2": [
        {
          "item1_lvl3": "x1",
          "item2_lvl3": "y1"
        },
        {
          "item1_lvl3": "x2",
          "item2_lvl3": "y2"
        }
      ]
    }
  ]
}

🎉

If yes how much time will it take to release this feature ?

I’ve merged java 11 upgrade PR(#6424) today. So recommended to upgrade to java 11 for dev. Right now we still need to support java8 compilation for existing users to upgrade, so don’t use java11 code features until further notice when we drop the java 8 support. For devs still on java8, please use below command to build your project.

mvn clean install -DskipTests -Pbin-dist -T 4  -Djdk.version=8

Thanks a lot to @User for making this happen!

or 8.0 tar

Saw this paper on automatically extracting columnar information from JSON, and thought about how that could be interesting for Pinot: https://db.in.tum.de/people/sites/durner/papers/json-tiles-sigmod21.pdf

quickstart jdk12 is failing intermittently on PR github actions. Did anything change recently?

Hi folks, need some help

I have added a new parameter in QueryExecutorConfig, which I wish to set in a new unit test I am writing

Is there a pattern which I can follow to set and unset the parameter in the test only?

use the serve.conf if you want to control this as start/stop of service, you can use query options if you want it more dynamic on a per query basis

Is there an example? I want to localise this parameter's value to the test class only