This message was deleted.
# troubleshootings
Slackbot
06/15/2023, 2:20 PMThis message was deleted.
s
Sergio Ferragut
06/16/2023, 2:56 PMCan you share the rest of the spec? What version of Druid is this?
l
Lionel Mena
06/16/2023, 3:01 PMForgot to include the version, 25.0.0
I can share the dataSchema part of the Kafka supervisor spec. The relevant field is
OptimalResolution Copy code
"dataSchema": {
"dataSource": "sdk-americas-realtime",
"timestampSpec": {
"column": "timestamp",
"format": "posix",
"missingValue": "2010-01-01T00:00:00Z"
},
"transformSpec": {
"transforms": [
{
"type": "expression",
"name": "TotalDownlinkThroughput",
"expression": "cast(CdnDownlinkThroughput, 'LONG') + cast(P2pDownlinkThroughput, 'LONG')"
},
{
"type": "expression",
"name": "latitude_mirror",
"expression": "Latitude"
},
{
"type": "expression",
"name": "longitude_mirror",
"expression": "Longitude"
},
{
"type": "expression",
"name": "UplinkCapacity",
"expression": "cast(regexp_extract(UplinkCapacity, '(\\\\d+)\\\\.', 1), 'LONG')"
},
{
"type": "expression",
"name": "UplinkCapacityStatus",
"expression": "cast(concat('0.', regexp_extract(UplinkCapacity, '\\\\.(\\\\d+)', 1)), 'DOUBLE')"
},
{
"type": "expression",
"name": "OptimalResolution",
"expression": "if( (((ViewportHeight - VideoResolutionHeight) / case_simple(VideoResolutionHeight, 0.0, 1.0, VideoResolutionHeight)) < 0.01) && (((ViewportWidth - VideoResolutionWidth) / case_simple(VideoResolutionWidth, 0.0, 1.0, VideoResolutionWidth)) < 0.01), 'true', 'false')"
}
],
"filter": null
},
"dimensionsSpec": {
"dimensions": [
{
"type": "string",
"name": "CustomerId",
"createBitmapIndex": true
},
{
"type": "string",
"name": "ChannelId",
"createBitmapIndex": true
},
{
"type": "string",
"name": "StreamId",
"createBitmapIndex": true
},
{
"type": "string",
"name": "Bitrate",
"createBitmapIndex": true
},
{
"type": "string",
"name": "DeviceId",
"createBitmapIndex": true
},
{
"type": "string",
"name": "NodeId",
"createBitmapIndex": true
},
{
"type": "string",
"name": "SessionId",
"createBitmapIndex": true
},
{
"type": "string",
"name": "RepresentationId",
"createBitmapIndex": true
},
{
"type": "string",
"name": "ReceiveFromNodeId",
"createBitmapIndex": true
},
{
"type": "string",
"name": "NatType",
"createBitmapIndex": true
},
{
"type": "string",
"name": "ConnectionType",
"createBitmapIndex": true
},
{
"type": "string",
"name": "CurrentHost",
"createBitmapIndex": false
},
{
"type": "string",
"name": "ResponseCdn",
"createBitmapIndex": false
},
{
"type": "string",
"name": "SdkVersion",
"createBitmapIndex": true
},
{
"type": "long",
"name": "ViewportWidth",
"createBitmapIndex": false
},
{
"type": "long",
"name": "ViewportHeight",
"createBitmapIndex": false
},
{
"type": "long",
"name": "VideoResolutionWidth",
"createBitmapIndex": false
},
{
"type": "long",
"name": "VideoResolutionHeight",
"createBitmapIndex": false
},
{
"type": "string",
"name": "OptimalResolution",
"createBitmapIndex": true
},
{
"type": "string",
"name": "OsName",
"createBitmapIndex": true
},
{
"type": "string",
"name": "OsVersion",
"createBitmapIndex": true
},
{
"type": "string",
"name": "BrowserName",
"createBitmapIndex": true
},
{
"type": "string",
"name": "BrowserVersion",
"createBitmapIndex": true
},
{
"type": "string",
"name": "PlayerName",
"createBitmapIndex": true
},
{
"type": "string",
"name": "PlayerVersion",
"createBitmapIndex": true
},
{
"type": "string",
"name": "AppName",
"createBitmapIndex": true
},
{
"type": "string",
"name": "AppVersion",
"createBitmapIndex": true
},
{
"type": "string",
"name": "HrefLocation",
"createBitmapIndex": true
},
{
"type": "string",
"name": "Source",
"createBitmapIndex": true
},
{
"type": "string",
"name": "PlayerState",
"createBitmapIndex": true
},
{
"type": "string",
"name": "StreamState",
"createBitmapIndex": true
},
{
"type": "string",
"name": "ConnectionStatus",
"createBitmapIndex": true
},
{
"type": "string",
"name": "MaskedStreamUri",
"createBitmapIndex": true
},
{
"type": "string",
"name": "InboundProtocol",
"createBitmapIndex": true
},
{
"type": "string",
"name": "InboundPeeringType",
"createBitmapIndex": true
},
{
"type": "string",
"name": "Ip",
"createBitmapIndex": true
},
{
"type": "string",
"name": "AsName",
"createBitmapIndex": true
},
{
"type": "long",
"name": "Asn",
"createBitmapIndex": false
},
{
"type": "string",
"name": "CountryCode",
"createBitmapIndex": true
},
{
"type": "double",
"name": "Latitude",
"createBitmapIndex": false
},
{
"type": "double",
"name": "Longitude",
"createBitmapIndex": false
}
],
"dimensionExclusions": [
"timestamp"
],
"spatialDimensions": [
{
"dimName": "Coordinates",
"dims": [
"latitude_mirror",
"longitude_mirror"
]
}
]
},
"metricsSpec": [
{
"fieldName": "RoundTripTime",
"name": "RoundTripTime",
"type": "longMax"
},
{
"fieldName": "BufferHealth",
"name": "BufferHealth",
"type": "longSum"
},
{
"fieldName": "DroppedFramesPerSecond",
"name": "DroppedFramesPerSecond",
"type": "longMax"
},
{
"fieldName": "TimeToFirstFrame",
"name": "TimeToFirstFrame",
"type": "longMax"
},
{
"fieldName": "StartingTime",
"name": "StartingTime",
"type": "longMax"
},
{
"fieldName": "SeekingTime",
"name": "SeekingTime",
"type": "longMax"
},
{
"fieldName": "BufferingTime",
"name": "BufferingTime",
"type": "longMax"
},
{
"fieldName": "PlayingTime",
"name": "PlayingTime",
"type": "longMax"
},
{
"fieldName": "PauseTime",
"name": "PauseTime",
"type": "longMax"
},
{
"fieldName": "BufferingEvents",
"name": "BufferingEvents",
"type": "longMax"
},
{
"fieldName": "AggregateBufferFillingRate",
"name": "AggregateBufferFillingRate",
"type": "floatSum"
},
{
"fieldName": "CdnDownlinkThroughput",
"name": "CdnDownlinkThroughput",
"type": "longSum"
},
{
"fieldName": "P2pDownlinkThroughput",
"name": "P2pDownlinkThroughput",
"type": "longSum"
},
{
"fieldName": "DownlinkSpeed",
"name": "DownlinkSpeed",
"type": "longMax"
},
{
"fieldName": "JoinResponseTime",
"name": "JoinResponseTime",
"type": "longMax"
},
{
"fieldName": "InboundConnectionCount",
"name": "InboundConnectionCount",
"type": "longMax"
},
{
"fieldName": "UplinkCapacity",
"name": "UplinkCapacity",
"type": "longSum"
},
{
"fieldName": "UplinkCapacityStatus",
"name": "UplinkCapacityStatus",
"type": "floatMax"
},
{
"fieldName": "AliveTime",
"name": "AliveTime",
"type": "longMax"
},
{
"name": "RollupCount",
"type": "count"
},
{
"fieldName": "TotalDownlinkThroughput",
"name": "TotalDownlinkThroughput",
"type": "longSum"
}
],
"granularitySpec": {
"type": "uniform",
"segmentGranularity": "hour",
"queryGranularity": "none",
"rollup": false
}
},s
Sergio Ferragut
07/07/2023, 1:42 AMFinally had some time to look at this. It seems to me like there is an issue with how Druid is interpreting data types in this case. I wonder if the fact that OptimalResolution is a string is what makes it run into this. Can you try making it long and changing the expression to return 0 or 1. If not as a workaround, at least to verify that this is the reason and then create a bug report on it.
l
Lionel Mena
07/07/2023, 1:26 PMThanks for getting back to me Sergio! As you suspected, there is a bug for strings fields. It's not a workaround for me because numeric types are not indexable.
g
Gian Merlino
07/07/2023, 4:48 PMDoes it work if you cast
ViewportHeightcast(ViewportHeight, 'long')VideoResolutionHeightGian Merlino
07/07/2023, 4:49 PMthe transforms are applied before
dimensionsSpecGian Merlino
07/07/2023, 4:49 PMit would use the types that are seen in the source data
Gian Merlino
07/07/2023, 4:50 PMis it possible some
ViewportHeightVideoResolutionHeightGian Merlino
07/07/2023, 4:50 PMbtw, your example doesn't have the
inputFormattsvcsvtsvcsvl
Lionel Mena
07/12/2023, 2:07 PMHi Gian, this spec is part of a Kafka Supervisor spec. The input format is a protobuf and the fields
ViewportHeightVideoResolutionHeightViewportWidthVideoResolutionWidthuint322 Views