Venkatesh Damasthapur
06/26/2024, 12:43 PMdimensionOverrideMap are not rendered (i.e, though the lookup returned "null" (as string) it is not replaced with "" (empty) string. Any issue in specifying extractionFn?
{
"type": "extraction",
"dimension": "acc_id",
"outputName": "AgencyName",
"outputType": "STRING",
"extractionFn": {
"type": "registeredLookup",
"lookup": "agency_lookup",
"retainMissingValue": false,
"replaceMissingValueWith": "null",
"injective": false,
"optimize": true,
"valueColumn": "business_name",
"dimensionOverrideMap": {
"null": "",
"-1": "N/A",
"-3": "N/A"
},
"useQueryLevelCache": false
}
}Tim Wolfe
08/29/2024, 10:32 PMdimensionOverrideMap.
Here are a few points to consider regarding your current configuration:
1. retainMissingValue: Since you have set this property to false, it means that any missing values will not be retained. This is useful if you want to clean up null or missing values but might lead to unexpected behavior if combined with the replaceMissingValueWith option.
2. replaceMissingValueWith: You have specified "null" (as a string) to replace missing values. If your expectation is to replace these with an empty string "", you should set this parameter to "" instead.
3. dimensionOverrideMap: You are defining how specific values should be overridden. However, the key "null" in JavaScript (JSON compatible) isn't quite accurate for distinguishing between actual null values and the string "null". Make sure the values in this map correspond precisely to how values are returned from the lookup.
4. lookup returned values: If the lookup returns an actual null (which is different from the string "null"), it will not trigger replacements as specified in your dimensionOverrideMap. Ensure that your lookup mapping correctly handles these cases.
5. debugging: For effective debugging, evaluate the outputs of your lookups. Check whether the values being returned match what you expect. You might also want to confirm that the lookup table being referenced (agency_lookup) is configured correctly and holds the correct mappings.
Make necessary adjustments based on the feedback above and consider testing your extraction function configuration using various input cases to ensure that it behaves as intended.
Sources:
Extraction Function Details: filters.md (internal document)
Druid SQL Compatibility Changes: upgrade-notes.md (internal document)
Data Types and Null Handling: sql-data-types.md (internal document)