https://linen.dev logo
Join Slack
Powered by
Is there a way to ingest specific column as metric...
# general
p
Is there a way to ingest specific column as metric in MSQ ingestion without enabling the rollup (so without group by clause)? I cannot come up with MSQ equivalent of:
Copy code
{
  "type": "index_parallel",
  "spec": {
    "dataSchema": {
      "dataSource": "test_2r_1d_1m_no_rollup",
      "timestampSpec": {
        "column": "ts",
        "format": "auto"
      },
      "dimensionsSpec": {
        "dimensions": [
          "category"
        ]
      },
      "metricsSpec": [
        {
          "type": "longSum",
          "name": "sum_val",
          "fieldName": "val"
        }
      ],
      "granularitySpec": {
        "queryGranularity": "hour",
        "rollup": false
      }
    },
    "ioConfig": {
      "type": "index_parallel",
      "inputSource": {
        "type": "inline",
        "data": "{\"ts\":\"2026-04-15T10:00:00Z\", \"category\":\"A\", \"val\":10}\n{\"ts\":\"2026-04-15T10:00:00Z\", \"category\":\"A\", \"val\":5}"
      },
      "inputFormat": {
        "type": "json"
      }
    }
  }
}
b
What is the reason it needs to be a metric if you don't want rollup? Why not just ingest as-is, or cast to long if you like?
I'm not sure of the difference between a dimension and a metric that doesn't do any aggregation.
p
I guess it boils down to being explicit - if I want to ingest data as it is (without rollup) but I know I would never filter or group by this column I wanted to ingest it as metric.
Also: • string dimension will create index automatically -> more space usage (can be turned off when ingesting if I remember correctly); for numeric types it does not apply I think • (guess) maybe aggregations run on metrics are faster than on dimension because code path is different?
b
true about strings vs numeric and indexing. Personally, I'm not sure that druid definitively distinguishes between dimensions and metrics internally. It will keep track of the aggregation if you use, eg, longSum, so that it can rollup properly later if needed (compaction, etc).
If you just ingest a number as a long (cast to integer if necessary), are you translating the MSQ in to native somehow, and seeing something different? Or is the end result different somehow?
Open in Slack
PreviousNext
Powered by