This message was deleted.

The broker is not generally involved with ingestion, its does parse SQL Based Ingestion commands to convert them into native ingestion instructions, but then the work is done by the Overlord and MM/Indexers. Look for problems in the overlord and MM/indexer/task logs. It could also be a problem with coordinator at the time of handoff to the historicals. How are you determining that only half the data was ingested?

Hi Sergio, thanks for the reply. We determined only half of the total rows being ingested is becoming available in the datasource by referencing the ingestion task report of the multiple ingestion tasks that complete and save data into the datasource in question. The sum of the processed metrics for all the corresponding ingestion task reports was equal to exactly double of the number of rows shown in the resulting data source

Yes. That would help. Particularly the task logs. But back to the question, how do you know that data is missing. Is is the result of a query? What’s the query? Is it by row count? If so, are you using roll up at ingestion?

Yea, we have a data connector that our devs built that connects an internal visualization tool to the Druid server. One of the internal stakeholders noticed that the visualizations were off so I looked into it and ran some basic queries which verified that there was data leakage

A common mistake with rollup ingestion is to compare the count of rows processed at ingestion to the count of rows actually ingested. Since there is aggregation, these ingestions usually calculate a “count” column that indicates how many source rows were rolled up into the target row. So a sum(“count”) is used to verify the rows ingested. Don’t know if this is your case…yes, I’m around.

Gotcha, but also verified that there's missing rows by running the queries like I mentioned. Its product event data so looked for basics like daily active users which was clearly off which wasn't the case previously

Not unless there is a bug. So, maxRowsInMemory/maxBytesInMemory control the real-time ingestion buffer. Once this limit is reached, the ingestion does a local temporary persist and continues ingesting into a new buffer. I don't think it would have a direct effect. After the segment level thresholds are reached, the ingestion task merges all the temporary persists into segment files and publishes them. Let's take a look at the logs and then also check that the coordinator/historicals are loading all the segments that the ingestion produced. Do you see full availability in the data sources view?

yea 100% availability

did you say this was a rollup ingestion? or not rollup?

Ingestion task 1:

To close out this thread... we did a bit of troubleshooting offline and discovered that there were two ingestion jobs for the same data source running one after the other with

appendToExisting:false

and covering the same overall timeframe. This caused the second job to overwrite any segments (most of them) that covered the same time interval, by switching

appendToExisting:true

, the second job added segments instead of replacing them and then we got the full row count.