Hi , Your guidance and help are much appreciated. My situation is that: I am submitting an ingestion task with

metricsSpec

"metricsSpec": [{
                    "type": "doubleSum",
                    "name": "quantity",
                    "fieldName": "quantity"
                },
                {
                    "type": "doubleSum",
                    "name": "ponderation",
                    "fieldName": "ponderation"
                },
                {
                    "type": "longMax",
                    "name": "updated_at",
                    "fieldName": "updated_at"
                }
            ]

And `inputFormat`:

"inputFormat": {
                "type": "csv",
                "findColumnsFromHeader": true
            },

And my question is: The source system drops all of the

dimensions

,

metrics (quantity, ponderation, updated_at)

enclosed in double quotes. Druid was able to deserialize the data and aggregation over the quantity yields correct results. Do you find any risks in accepting the source system to enclose everything in double-quotes? Are there different recommendations or best practices that i should follow? Thanks in advance, Omar

Enclosing data in double quotes can lead to certain considerations: 1. Data Parsing: If the source system consistently uses double quotes, ensure that the serialization and deserialization processes handle this appropriately. Druid's CSV input format usually handles quoted fields without issues, but it's important to verify that the CSV parsing settings correctly interpret those quotes. 2. Null Values: If any field contains null values that are also quoted (e.g.,

""

), ensure that your data processing logic treats these as nulls rather than empty strings. 3. Field Boundaries: Consistent usage of double quotes can help avoid issues with commas in data fields (if they exist), defining boundaries for each field clearly. 4. Best Practices: ◦ Validation: Always validate incoming data to ensure that it adheres to the expected formats and types. ◦ Testing: Implement tests on ingestion jobs to ensure that they behave as expected with quoted data. ◦ Documentation: Maintain clear documentation on the expected input formats for anyone working with the data pipeline to reduce confusion. While the ability of Druid to handle these cases well is advantageous, confirm by running tests. If issues arise, modifying the ingestion specs or adjusting the source system’s output format may be necessary.