This message was deleted.
# generals
Slackbot
08/20/2023, 9:33 PMThis message was deleted.
b
Ben Krug
08/21/2023, 7:02 PMCan you attach the spec?
e
El Hakan
08/21/2023, 9:01 PM Copy code
}
]
}
},
"useEarliestOffset": true
},
"tuningConfig": {
"type": "kafka"
},
"dataSchema": {
"dataSource": "enviro_forecast_table",
"timestampSpec": {
"column": "payload.after.timeSec",
"format": "posix"
},
"transformSpec": {
"transforms": [
{
"type": "expression",
"name": "timeSec",
"expression": "__time"
}
]
},
"granularitySpec": {
"queryGranularity": "minute",
"rollup": false,
"segmentGranularity": "hour"
},
"dimensionsSpec": {
"useSchemaDiscovery": true,
"dimensionExclusions": []
}
}
}
}El Hakan
08/21/2023, 9:03 PM@Ben Krug i added the timeSec columns during the ingestion by extracting the values from the __time expression. When publishing the task i can see the column defined in the transformspec.transforms.type but for somereason it does not appear in my table
b
Ben Krug
08/21/2023, 9:40 PMI'm having trouble testing atm, but I tried two things - one to just copy the timestamp, which will give you a number (milliseconds since epoch iirc), or using timestamp_format to get a string with the date & time. I got this syntax:
Ben Krug
08/21/2023, 9:40 PM{
"type": "expression",
"name": "time_plain",
"expression": "\"__time\""
},
{
"type": "expression",
"name": "time_format",
"expression": "timestamp_format(\"__time\")"
}
e
El Hakan
08/22/2023, 5:43 AM@Ben Krug that is also similar to yours what I did but the issue is when i create the new column time_plain for example copy the values using the expression __time and reach the apply the configure schema , the new created column gets deleted even though it appears in my json spec just like you have it above. Somehow druid is seeing it as a duplicate column (but with a different name ) and removing it
s
Sergio Ferragut
08/22/2023, 6:01 PMHave you tried just adding the source field in the dimensions list, in addition to its mapping in the timestampSpec?
b
Ben Krug
08/22/2023, 9:09 PM@El Hakan if that doesn't work, please post the entire spec and we can see if there's anything else that's affecting it.
k
Kiran Kumar Puttakota
09/07/2023, 6:46 AMHello. It's very simple. what ever is the column name, Please click on Transform button and add column option and give the exact column name which is already there then click ok then you can find your old time column.