Has anyone had trouble configuring ingestion aggre...
# troubleshootings
Stuart Millholland
07/13/2022, 4:22 PMHas anyone had trouble configuring ingestion aggregations via the UI? I add the section and it saves, but it doesn't stick.
Stuart Millholland
07/13/2022, 4:22 PMHere's the section I add:
Stuart Millholland
07/13/2022, 4:22 PM"ingestionConfig": {
"transformConfigs": [],
"aggregationConfigs": [
{
"columnName": "sum_activity_count",
"aggregationFunction": "SUM(activity_count)"
},
{
"columnName": "sum_file_size",
"aggregationFunction": "SUM(file_size)"
},
{
"columnName": "sum_req_size",
"aggregationFunction": "SUM(req_size)"
},
{
"columnName": "sum_resp_size",
"aggregationFunction": "SUM(resp_size)"
},
{
"columnName": "sum_event_count",
"aggregationFunction": "SUM(event_count)",
"defaultNullValue": 1
},
{
"columnName": "max_user_risk_score",
"aggregationFunction": "MAX(user_risk_score)"
},
{
"columnName": "max_user_raw_risk_score",
"aggregationFunction": "MAX(user_raw_risk_score)"
},
{
"columnName": "max_user_threat_score",
"aggregationFunction": "MAX(user_threat_score)"
}
]
},
Stuart Millholland
07/13/2022, 4:23 PMThe table saves fine but then I see this
Stuart Millholland
07/13/2022, 4:23 PM"ingestionConfig": {},
x
Xiaobing
07/13/2022, 5:23 PM👀
Xiaobing
07/13/2022, 5:26 PMDid you see errors while saving the changes via UI. As there are a few requirements to enable aggregation during ingestion and the changes to table config might be invalid. (trying on my side in the meanwhile)
s
Stuart Millholland
07/13/2022, 5:55 PMI saw no errors in the UI, it reported success
Stuart Millholland
07/13/2022, 5:55 PMI added this to the tableIndexConfig section as well: "noDictionaryColumns": ["sum_activity_count", "sum_file_size", "sum_req_size", "sum_resp_size", "sum_event_count", "max_user_risk_score", "max_user_raw_risk_score", "max_user_threat_score"],
Stuart Millholland
07/13/2022, 5:59 PMAhhhh sorry @Xiaobing I am on 0.10.0 and it looks like support for ingestion aggregations was added in 0.11.0 correct?
Stuart Millholland
07/13/2022, 5:59 PMIf so, I'll upgrade and try it.
Stuart Millholland
07/13/2022, 6:02 PMI don't see Ingestion Aggregations here
Stuart Millholland
07/13/2022, 6:03 PMBut I do see it here
x
Xiaobing
07/13/2022, 6:07 PMoh, yes, this new features was landed after 0.10.0 per the commit history
fyi, https://github.com/apache/pinot/pull/8611
s
Stuart Millholland
07/13/2022, 6:07 PMok now it all makes sense 🙂
Stuart Millholland
07/13/2022, 6:08 PMAny idea of the official 0.11.0 release date?
Stuart Millholland
07/13/2022, 7:39 PMOK @Xiaobing I got it all setup locally w/ 0.11.0 and it tries to do the aggregation but I get this error: Metrics aggregation cannot be turned ON in presence of multi-value dimension columns, eg: external_recipients
Stuart Millholland
07/13/2022, 7:39 PMSo looks like no good for me. Though I'm curious why offline managed flows (rollup) works fine on those multi-valued columns and ingestion aggregation does not. Maybe something for future support?
m
Mayank
07/13/2022, 7:40 PM
We are working on some pre-requisites for 0.11.0, so I’d say a few weeks
❤️ 1
x
Xiaobing
07/13/2022, 8:23 PMMetrics aggregation cannot be turned ON …not so sure about this. I see both places actually use different ValueAggregators. But for Aggregator used in rollups, I didn’t find handling logic on MV columns 🤔
Xiaobing
07/13/2022, 10:30 PMlooks like for rollups during offline managed flows, the aggregation is done on SingleValue metrics columns. MV columns must be dimension columns. Is this behavior consistent with what you saw on your side?
✅ 1
s
Stuart Millholland
07/14/2022, 1:25 PMSo the offline rollups work fine on our mv dimension fields. Just to state it, the mv fields we use are dimension and ingestion aggregation doesn't seem to like that, but offline managed rollup is ok with it.
Stuart Millholland
07/14/2022, 2:05 PMSo to reproduce what we are seeing you can do a small table w/ an MV dimension and a measure and add offline rollups to that and see it work. Then try ingestion aggregation and it should fail.
Stuart Millholland
07/14/2022, 2:05 PMand to clarify, just a single field measure.
Stuart Millholland
07/14/2022, 2:18 PM@Bobby Richard for visibility.
2 Views