I have a very overloaded question that will be hard to answer. I know druid excels as analytical storage. The usecase we have is that we need a timeseries database. This will be the primary database and be treated as source of truth. We got regulatory requirement to ensure data correctness, integrity and durability. It will host billions of rows, per day, up to 10 years. Most solution require a very deep dive to uncover some of the limitations which has (very large) cost. Given the scenario, where would the major pain points be if this was druid? For example, my understanding is that there is no WAL, so how does druid recover from a crash?

i think you're right, it will be tough to answer a question like this without doing a deep dive, for any system. i will try to give you a pointer about where to look at as you evaluate Druid

If you are looking for a big data solution to store timeseries data, I would suggest that you also look at HBase if you have access to a Hadoop cluster. We store timeseries data in it (millions of points per day) and it works fantastically well. It even supports multi-cluster replication for failover/resiliency. You can look at OpenTSDB as an example of how to store timeseries data in HBase.

Thank you @Gian Merlino for taking the time to reply.

when it comes to correctness / integrity / durability on the ingestion and storage side, the main thing to recognize is that all data ingested into Druid is stored in "segment" files on object storage (e.g. S3, HDFS, etc) with pointers to those files in a "metadata store" (typically relational database like cloud-managed MySQL or PostgreSQL).

In case when S3 is the storage mechanism, wouldn't writing every segment create a TPS bottleneck? In the official documentation it is mentioned that segments are pushed to deep storage (eg s3) after they are constructed. I cannot find where they are stored before they are pushed to S3. This is the part that I'm most concerned about in terms of durability

@HEPBO3AH Are you thinking of using batch ingestion or streaming ingestion? For batch ingestion, while writing the segments, they are not queryable so the storage (while writing the segment) doesn’t matter. They will only be queryable once they are pushed to S3 and Druid historical nodes download them. For realtime ingestion, while writing the segments, they are stored on the ingestion tasks (Peon, K8 pod, etc) and are available for query right away. You can increase durability by setting the

replicas

setting on SupervisorSpec.

Loss of segments before pushing to S3 doesn't affect long-term durability, because: • for batch ingest, the expectation is whatever is driving the ingest (Airflow, etc) will retry any failed batch ingest jobs. • for streaming (realtime) ingest, we will re-read whatever is lost from upstream (Kafka or Kinesis) automatically.

Another thing to add... You can also set replication per each datasource on the Retention (Load/Drop) rules. This can replicate the segments on more than one Historical nodes. This make sure that if a Historical node goes down, your data is still available. (Historical node pulls down data from S3 and make them available for query). However, one pain point we have (@Gian Merlino Please let me know if I am overlooking something) is that if many historical nodes are down due to OOM caused by a query, the replication may not be enough. For example, some topN, or timeseries data can caused a wide spread oom. When this happens, query will still return successful querying whatever data is left on the remaining UP historical nodes. This can be a data integrity issue as we do not know that data is missing (query still returns 200). There was a PR to address this: https://github.com/apache/druid/pull/13967 but it was closed (@Gian Merlino do you know why we abandon the PR?)

hmm, i'm not sure what happened with that one… it looks like it lost steam

restored when the historicals are back online

That can take some time (~30mins for us with our 4-7TB per historical). During that time, you would have query returning incorrect query results (missing data, gaps, etc.). You can setup alert from Druid metrics to detect when Historicals are down and/or segments are unavailable though.

yeah, that's true. i am just trying to be clear about whether any particular data consistency/integrity issue is temporary or permanent

yep. agree

> • for batch ingest, the expectation is whatever is driving the ingest (Airflow, etc) will retry any failed batch ingest jobs. Would you know if the

flush & fsync

is called before the 200 is returned to ensure that the data in the buffer is written to the disk?

for Druid, durability is achieved via object storage, not local storage (local storage is always assumed by Druid to be non-durable)