Apache Druid

To install Apache Druid, follow the detailed steps recommended by the community members:

1. Download the Apache Druid software from the official website and extract the contents of the downloaded file into a directory of your choosing.
2. To start Apache Druid after installation, you can utilize scripts provided in the `bin` folder e.g.`./bin/start-single-server-small` which is suitable for a small server. This would also automatically set server properties based on available memory and CPU on your server.
3. After starting Druid, you should be able to access the Druid console on router port 8888 by default (e.g., localhost:8888).
4. If you are running Druid 25 or later, it is suggested to utilize the `bin/start-druid` command for more practical and automatic configurations.
5. If you encounter issues while installing, you can refer to the logs found in the same directory where you installed Druid. This can guide you in troubleshooting and understanding any potential problems during the installation process.

Remember, these setup steps form the basis for a simple configuration -- depending on your specific requirements, you may need to adjust the configuration accordingly.&lt;br&gt;

References:&lt;br&gt;
- <https://druid.struct.ai/kb/t/2K2b17>
- <https://druid.struct.ai/kb/t/2K2aca>



<@U031A9D19NU> wdyt? :slightly_smiling_face:

set the channel topic: Ask <https://struct.ai|Struct.AI> bot anything. Knowledge Base: <https://druid.struct.ai>

To run queries with large result size efficiently in Druid, there are a handful of recommended techniques that can be implemented:

1. Increase the number of brokers and merge buffers to handle more concurrent requests and conglomerate results from multiple queries without overwhelming the system <https://druid.struct.ai/kb/t/2K2a4e>

2. Adjust the number of threads (`numThreads`) and connections (`numConnections`) to optimize resource utilization and balance load <https://druid.struct.ai/kb/t/2K2a4e>

3. Increase buffer size by adjusting the `druid.processing.buffer.sizeBytes` in the Broker runtime properties, as this provides more space for storing intermediate query results <https://druid.struct.ai/kb/t/2K2adb>

4. Reduce the number of segments needing to be scanned by range partitioning, compacting to reduce fragmentation, and configuring secondary partitioning for faster scanning &lt;@2K3510, @2K2a52&gt;

5. Improve schema design according to best practices and optimize the order of dimensions in the index spec &lt;@2K29eb, @2K3510&gt;

6. Boost resources by adding more data servers, which has the potential to alleviate bottlenecks at the Broker &lt;@2K29eb, @2K2a52&gt;

References:
- <https://druid.struct.ai/kb/t/2K2a4e>
- <https://druid.struct.ai/kb/t/2K29eb>
- <https://druid.struct.ai/kb/t/2K2adb>
- <https://druid.struct.ai/kb/t/2K3510>
- <https://druid.struct.ai/kb/t/2K2a52>



The error that you're getting suggests that in Druid's SQL, the UNION operation expects simple table scans, without any additional filters or aliasing. Also, it requires the column types of tables being unioned to be of the same type. The query contains a LIMIT clause on the tables being unioned, which isn't a simple table scan, so this could be the reason behind the error.

However, this behavior is a limitation in Druid and not something usually expected in SQL dialects. In standard SQL, your query would run without issues. To resolve this, you may need to have simple table scans in your UNION operation or change the SQL planning strategies in Druid. 

Additionally, this problem could also be due to differences in the dataset's nature or its underlying format, such as Parquet or JSON, or complications with its transformation, ingestion, or querying. It is crucial to ensure that the data processed is appropriately arranged and all the runtime properties are accurately set. Therefore, it would be helpful also to check on these additional details, although they were not mentioned specifically in your question. 

References:
- <https://druid.struct.ai/kb/t/2K2a95>
- <https://druid.struct.ai/kb/t/2K2a9f>
- <https://druid.struct.ai/kb/t/2K29be>
- <https://druid.struct.ai/kb/t/2K3510>
- <https://druid.struct.ai/kb/t/2K2b64>



Hmm. <@U05FTJ3T08K> - The answers are quite off the mark.

Did you check the sources / are they relevant?

Yes. They are not relevant. The problem in the question is a functional one, being misdiagnosed as a performance issue.

<#C05GUBF9ZBQ|>  I would like to ingest data from an external postgres database. However, the examples on the website regarding input sources do not lead me anywhere (I am lost.). Can someone provide a sample json file and the corresponding “curl command” to upload it? THat would be awesome. Any help is highly appreciated