Hi Team, i am trying to store daily aggregated dat...
# troubleshooting
v
Hi Team, i am trying to store daily aggregated data. However i see multiple rows coming up instead of single. Schema: { "schemaName": "AggregationTest", "enableColumnBasedNullHandling": false, "dimensionFieldSpecs": [ { "name": "request_metadata.request.account", "dataType": "STRING", "notNull": false } ], "metricFieldSpecs": [ { "name": "TotalCallVolume", "dataType": "LONG", "notNull": false } ], "dateTimeFieldSpecs": [ { "name": "aggregation_created_at", "dataType": "STRING", "notNull": false, "format": "1DAYSSIMPLE_DATE_FORMAT:yyyy-MM-dd", "granularity": "1:DAYS" } ] } Transform: "aggregationConfigs": [ { "columnName": "TotalCallVolume", "aggregationFunction": "COUNT(*)" } ], "transformConfigs": [ { "columnName": "aggregation_created_at", "transformFunction": "DATETRUNC('DAY',DATETIMECONVERT(created_at, '1MILLISECONDSEPOCH', '1DAYSEPOCH', '1:DAYS'))" } ] Output that i see :
image.png
Aso if i use 15 minutes bucket or something similar, it does not show new aggregated_created_at and aggregates in the initial aggreated_created at value itself
x
this aggregates is time bucket based
so you may have multiple rows
v
Hi @Xiang Fu problem is all data is going into same time (aggregation_created_at). if i go for 15 min/hourly aggregation still there is only one time
x
Are you using Kafka for ingestion?
How many Kafka partitions you have?
Do you have sample Kafka message
v
Hi @Xiang Fu Able to figure out the problem and resolve. Thanks. One more question. how can i avoid duplicates in ingestion aggregation? We have possibility of duplicates in Kafka. I need to be able to show the total usage and cannot afford duplicates as we are going to take the usage and do metered billing.
x
what do you mean by the duplicates and how do you distinguish the duplicates?
v
@Xiang Fu I can have duplicate records in Kafka and my message has a unique ID to distinguish them. Now when i use ingestion aggregation, is there a way to ignore them into aggregation.
producer_message_id is my unique key. This screenshot is without using ingestion aggregation
x
afaik, there is no way to have dedup+ ingestion agg together
you will lose the primary key when doing ingestion agg