https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
h
Can you share the spec for compaction?
k
Copy code
{
  "dataSource": "networks",
  "taskPriority": 25,
  "inputSegmentSizeBytes": 100000000000000,
  "maxRowsPerSegment": null,
  "skipOffsetFromLatest": "PT1H",
  "tuningConfig": {
    "maxRowsInMemory": null,
    "appendableIndexSpec": null,
    "maxBytesInMemory": null,
    "maxTotalRows": null,
    "splitHintSpec": null,
    "partitionsSpec": {
      "type": "dynamic",
      "maxRowsPerSegment": 5000000,
      "maxTotalRows": null
    },
    "indexSpec": null,
    "indexSpecForIntermediatePersists": null,
    "maxPendingPersists": null,
    "pushTimeout": null,
    "segmentWriteOutMediumFactory": null,
    "maxNumConcurrentSubTasks": null,
    "maxRetry": null,
    "taskStatusCheckPeriodMs": null,
    "chatHandlerTimeout": null,
    "chatHandlerNumRetries": null,
    "maxNumSegmentsToMerge": null,
    "totalNumMergeTasks": null,
    "forceGuaranteedRollup": false,
    "type": "index_parallel"
  },
  "granularitySpec": null,
  "dimensionsSpec": null,
  "metricsSpec": null,
  "transformSpec": null,
  "ioConfig": null,
  "taskContext": null
}
k
What was your original ingestion spec that you used to separate the dimensions and metrics?
g
I wonder if this is happening specifically for the time range where you made the cutover
During compaction, Druid constructs a new dimensions and metrics list based on taking the union of what's there in existing segments
During the cutover time range, certain columns would be in both dimensions (older segments) and metrics (newer segments) hence the conflict
It's an interesting situation to be in, since mixing nonrolledup and rolledup data in the same datasource doesn't always make sense (for example, typically you want a 
count
column in your rolled-up data so you can sum it at query time, but you don't need this in nonrolledup data)
If this is in fact your situation, then the right thing to do depends on the specifics of what change you made when you switched from nonrolledup to rolledup
In some cases it will be difficult to reconcile and you may need to either create a new datasource, or backfill from the original data if possible
In other cases it will be possible to reconcile with a manual compaction, which you could run to set things straight
Btw, a good way to think about this is: how do you query this data? Generally, the aggregators you use at query time would inform what you want to do with a manual compaction
2 Views
Open in Slack
PreviousNext
Powered by