Apache Druid

Please review small PR to correctly throw parse exception for multi-valued numeric dims:
<https://github.com/apache/druid/pull/12953>

PSA: I suspect any open PR needs to merge master in order to pass CI, due to the recent module changes from <https://github.com/apache/druid/pull/12368>. `it-tools` is gone, so the older .travis.yml in some PR branches won't build any longer. I tried retriggering builds and it didn't help, so I suspect the merge from master -&gt; branch needs to be done manually by contributors

Sorry! This is due to a rename of an IT module. Looks like the build is trying to use the old Travis file with the new code.

The solution is to rebase on master to pick up the new travis file.

documentation of org.apache.calcite.rel.core.Window.RexWinAggCall :smile:

```/**
   * A call to a windowed aggregate function.
   *
   * &lt;p&gt;Belongs to a {@link Window.Group}.
   *
   * &lt;p&gt;It's a bastard son of a {@link org.apache.calcite.rex.RexCall}; similar
   * enough that it gets visited by a {@link org.apache.calcite.rex.RexVisitor},
   * but it also has some extra data members.
   */```

<https://github.com/apache/druid/pull/12968>

i guess this one isn’t really specific to json, <https://github.com/apache/druid/pull/12971>

<https://github.com/apache/druid/pull/12972>

merged 12972; reviewed 12971 and 12968 with comments

I could use a review on <https://github.com/apache/druid/pull/12975>

tldr: it's looking like we'll need to avoid certain features (like `Memory.map` and `WritableMemory.allocateDirect`) if we want to support 8 + 11 + 17 on a single build

i just commented on this suggestion about scan query ordering: <https://github.com/apache/druid/issues/12958#issuecomment-1227904609>

curious if i'm missing anything. seems pretty useful

Is the version numbering system changed now?
Previously it was 0.23.0, right? and now is 24.0.0?

yep, we just got rid of the leading 0., everything else is the same; see <https://lists.apache.org/thread/xcfjoq696hn5w85hdgpxfb9hx4nsj4h1>

resolve the last remaining incompatibility i'm aware of: <https://github.com/apache/druid/pull/12987>

add "java 17 is experimental" to docs, along with tips: <https://github.com/apache/druid/pull/12990>

remove old, no longer needed, &amp; also problematic-on-java17 dependency since JvmMonitor was updated to use MXBeans: <https://github.com/apache/druid/pull/12989>

i've seen good results with ext4 + noatime

Fixing a flakey test: <https://github.com/apache/druid/pull/13004>

are you talking about the byRow mode?

might it make sense to tracks nulls there in some cases? like we want to add 1 for (a, null) 

yes.

do we? MySQL would exclude this tuple from the count.
<http://sqlfiddle.com/#!9/faf2f/38171>
• Select count(distinct FName, LName) from Employees - Returns 4
• Select count(distinct FName, LName, ManagerId) from Employees - Returns 3 

hmm: two thoughts on that

1) if it's standard behavior for COUNT DISTINCT with multiple args then we should follow the standard

2) but i don't find it very useful! for me my main use case of counting unique tuples is to know how many rows we would have if we rolled up on those dimensions: so counting nulls is valuable

putting 1 + 2 together: perhaps we want two functions so the second one can behave the way that's useful for rollup estimation?

yeah. something like `count(unique)` described here  - <https://www.ibm.com/docs/en/informix-servers/12.10?topic=expressions-count-distinct-count-unique-functions>

Screenshot 2022-09-18 at 7.18.01 PM.png

While the SQL Fiddle returns 3 for the query, I think the wording in the above IBM’s spec says:
&gt; If the *COUNT DISTINCT* function encounters NULL values, it ignores them unless every value in the specified column is NULL. If every column value is NULL, the *COUNT DISTINCT* function returns zero (0).
So, IMO a row like (null, null, null) should not be counted and else  like (’A, null, null) should be

This might not be a reliable source of information but FWIW <https://www.sqlshack.com/overview-of-sql-count-distinct-function/|here>, the output includes the row with NULL value in a single column (attached screenshot).

I didn’t find much documentation on handling of the null values esp with multiple columns.

Hi Didip - is this really a bug though? I see the current filter matching similar as `find`. That is, the filter is on the filename but we would look through all the files that are in the folder specified through prefix.

good question. From the perspective of end users crafting the ingestion spec, they expect the glob to behave like a posix file system, so they do think that it is a bug.

See: <https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1662065702857259|https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1662065702857259>

Without being able to stop at a certain subfolder, the utility of the glob syntax is limited.

my 2¢ on github: <https://github.com/apache/druid/pull/13027#issuecomment-1239058349>

long story short I do think we should make a change, but, a different change

<@U0344FW86DD> I added a comment trying to clarify what you are saying.

Will take another look this week! Sorry for the delay, spent a lot of time last week on the 24.0 release :slightly_smiling_face:

No problemo! I know that y’all have your hands full with 24.0.0

Looking at this again now. Sorry for the delay, been hectic the last couple of weeks :smile:

that's from autogenerated code; if your IDE isn't doing the generation on its own, you can do it with maven by running `mvn compile`

you should just have to do that once and then you can use IDE like normal

ooo gotcha, thank you for your explanation.

I got this error when running `mvn compile` .
```[ERROR] /path/to/druid/sql/src/main/java/org/apache/druid/sql/calcite/parser/DruidSqlParserImplFactory.java:[33,16] cannot access org.apache.druid.sql.calcite.parser.DruidSqlParserImpl
  bad class file: /path/to/druid/sql/target/classes/org/apache/druid/sql/calcite/parser/DruidSqlParserImpl.class
    unable to access file: java.nio.file.NoSuchFileException: /path/to/druid/sql/target/classes/org/apache/druid/sql/calcite/parser/DruidSqlParserImpl.class
    Please remove or make sure it appears in the correct subdirectory of the classpath.```
But that `.class` file exists.

i find sometimes maven gets confused and a clean helps

When I did `mvn clean &amp;&amp; mvn compile` , I got this error:
```[ERROR] Failed to execute goal on project druid-processing: Could not resolve dependencies for project org.apache.druid:druid-processing:jar:25.0.0-SNAPSHOT: Failure to find org.apache.druid:druid-core:jar:tests:25.0.0-SNAPSHOT in <https://repository.apache.org/snapshots> was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots has elapsed or updates are forced -&gt; [Help 1]```
But if I run that several more times, the compilation actually failed in a different place.
```[ERROR] Failed to execute goal org.apache.maven.plugins:maven-clean-plugin:2.5:clean (default-clean) on project druid-server: Failed to clean project: Failed to delete /Users/didip/oss/druid-didip/server/target -&gt; [Help 1]```
Am I doing something wrong? Why compile task is not idempotent?

hmm. for the first one you might need to do `mvn install` instead of `mvn compile`. (if it's trying to read from your local repository, `install` would put it there.)

for the second one, i'm not sure. seems odd for `mvn clean` to fail and i have not encountered that before

Ideally the tests should set it up themselves. There are xyz-sample-data.sql that set these up when you run the tests using a mvn command like:

xyz IMO is resolved by the -Dgroups=query parameter, so in this case it will resolve to query-sample-data.sql. IIRC the command that you shared with me offline doesn’t have this parameter and therefore might be one of the reasons why the data is not getting setup locally when you run the test case.

Sometimes, queries do not consider data due to incorrect core partition count being set during segment allocation in the same interval.
This usually occurs when core partitions are dropped for an interval and data is appended in the presence of stale pending segment entries.
 <https://github.com/apache/druid/pull/13070> attempts to fix it

where can we see the results of the cron tests?

i did not even really register that they were different from PR tests until reading this

last question, is there a way to run them on-demand on release branches?

seems like the most relevant use for a separated set

asked in the PR too, since that seems more appropriate

Thank you, I've tried to address your comment.
Is there a better alternative to prevent contention on the lock once supervisor stop has begun?

travis might break on any open PRs, <https://github.com/apache/druid/pull/13083> should fix

<https://github.com/apache/druid/pull/13092> is an attempt to make http based TaskRunner, LoadQueue and ServerInventoryView as the default in Druid

wonder if they're trying to get us to migrate to github actions

Hello, Here is a PR to introduce idle feature in Seekable stream supervisors if stream is inactive for a while. Please provide your feedback. <https://github.com/apache/druid/pull/13144>

<https://github.com/apache/druid/pull/13151> - Minor fix for a sql planning bug.

I believe this is related: <https://github.com/facebook/docusaurus/issues/6432>

I don't know what algolia account to log in with? is this something you have access to, Vad?

I believe I solved it here: <https://github.com/apache/druid/pull/13164>

it looks like something is messed up with the build?