This message was deleted.
# generals
Slackbot
09/19/2023, 5:49 AMThis message was deleted.
b
Bibek Sahoo
09/19/2023, 5:56 AMwe should use
segmentGranularitymaxRowsPerSegmentBibek Sahoo
09/19/2023, 5:56 AMBibek Sahoo
09/19/2023, 5:57 AMFor existing records in druid, you can use compaction to optimally resize segments
https://druid.apache.org/docs/latest/data-management/compaction/
s
Shithanshu Mishra
09/19/2023, 5:58 AM@Bibek Sahoo Thank you for the response.
I wanted to know if using API to insert via node is a good option to ingest data into druid or there are some better industry standards
b
Bibek Sahoo
09/19/2023, 6:00 AMIngesting into druid, you can use the druid console and submit job and API does the same process. So, if we are comfortable with API, That should be good, Through API we provide the ingestion SPEC through API, so we need to make sure we do have all the parameters specified as per standard.
Here is an sample SPEC, also you can find more suggestions on the guide a s well as per your source.
https://druid.apache.org/docs/latest/tutorials/tutorial-ingestion-spec/#final-spec
s
Shithanshu Mishra
09/19/2023, 6:01 AM@Bibek Sahoo Thank you will try it out and get back top you.
Shithanshu Mishra
09/19/2023, 6:15 AM@Bibek Sahoo i have kept the following config
Copy code
{
type: "index",
spec: {
dataSchema: {
dataSource: process.env.DRUID_TABLE_NAME,
timestampSpec: {
column: "log_time",
format: "auto",
},
dimensionsSpec: {},
metricsSpec: [],
granularitySpec: {
segmentGranularity: "day",
queryGranularity: "none",
rollup: false
}
},
ioConfig: {
type: "index",
inputSource: {
type: "inline",
data: JSON.stringify(logRequest),
},
inputFormat: {
type: "json",
},
appendToExisting: true,
},
tuningConfig: {
type: "index",
},
},
};b
Bibek Sahoo
09/19/2023, 6:19 AMIs it like the API request sends one one record on each run?
s
Shithanshu Mishra
09/19/2023, 6:19 AMYes
b
Bibek Sahoo
09/19/2023, 6:20 AMthen you might need to run compaction so that all the segments can be optimally resized.
s
Shithanshu Mishra
09/19/2023, 6:20 AMYou can consider it as an Insert API created on Node to ingest Data in druid
b
Bibek Sahoo
09/19/2023, 6:21 AMsomething like below:
Copy code
{
"type": "compact",
"dataSource": "process.env.DRUID_TABLE_NAME",
"ioConfig": {
"type": "compact",
"inputSpec": {
"type": "interval",
"interval": "2015-01-01/2024-01-01"
}
},
"granularitySpec": {
"segmentGranularity": "Day",
"queryGranularity": "none"
}
}s
Shithanshu Mishra
09/19/2023, 6:22 AMSo this compaction task will be called after every API Call of index insertion ?
b
Bibek Sahoo
09/19/2023, 6:23 AMyes, this can be a manual process or or you can enable auto compaction and druid will take care of comapction.
https://druid.apache.org/docs/latest/data-management/automatic-compaction/
s
Shithanshu Mishra
09/19/2023, 6:23 AMThank you
b
Bibek Sahoo
09/19/2023, 6:24 AMCompaction is one strategy you can use to optimize segment size for your Druid database. Compaction tasks read an existing set of segments for a given time interval and combine the data into a new “compacted” set of segments.
s
Shithanshu Mishra
09/19/2023, 6:25 AMI had one more concern
When i am calling APIs with 20 request/second then the druid server stops responding for a while,
The server configuration is is around 128 Cores of CPU and 128 GB of RAM
is this because of creation of new segment for each record insertion ?
b
Bibek Sahoo
09/19/2023, 6:26 AMNot sure on this, might be worth looking at logs or any other suggestion, its worth starting a new thread and let someone answer it having good expericence on same
s
Shithanshu Mishra
09/19/2023, 6:27 AM@Bibek Sahoo Thank you for the answers
👍 1
k
Karan Kumar
09/19/2023, 6:57 AM@Shithanshu Mishra if each api call wants to insert a record in druid, then you should thinking about putthing the record in kafka and then use the realtime kafka ingestion to druid.
Karan Kumar
09/19/2023, 7:00 AMThen you can pretty much scale upto 2 million RPS
s
Shithanshu Mishra
09/19/2023, 7:02 AM@Karan Kumar so you are suggesting to put data into Kafka on an API call and then connect Kafka to druid?
k
Karan Kumar
09/19/2023, 7:03 AM@Shithanshu Mishra Yes
s
Sergio Ferragut
09/19/2023, 9:41 PM@Shithanshu Mishra to add to Karan's suggestion which is definitely the right way to go for this, streaming ingestion makes rows available to query as soon as it consumes them from kafka. Kafka partitions are used to increase the throughput of a given stream and Druid ingestion is also parallelized across multiple tasks. This setup allows you to scale the real-time throughput while making the data available to query almost immediately after you call your API (the one that publishes to Kafka).