Hi all, I try to parse the following data in csv f...
# general
a
Hi all, I try to parse the following data in csv format: "datetime";"field1int";"field2string" "31.12.2023 123456";"220";"simple string" "31.12.2023 123457";"330";"string with ; symbol" but got a problems: 1. all values are in double quotes, how to remove them? 2. the value "string with ; symbol" parsed as a "string with" - so how to fix it, I use tsv input format, the full spec is the following: { "type": "index_parallel", "spec": { "ioConfig": { "type": "index_parallel", "inputSource": { "type": "inline", "data": "\"datetime\";\"field1int\";\"field2string\"\n\"31.12.2023 123456\";\"220\";\"simple string\"\n\"31.12.2023 123457\";\"330\";\"string with ; symbol\"" }, "inputFormat": { "type": "tsv", "findColumnsFromHeader": true, "skipHeaderRows": 0, "delimiter": ";", "listDelimiter": "\u0001" }, "appendToExisting": true }, "tuningConfig": { "type": "index_parallel", "partitionsSpec": { "type": "dynamic" }, "logParseExceptions": true }, "dataSchema": { "dataSource": "test_data", "timestampSpec": { "column": "datetime", "format": "d.M.yyyy Hmmss" }, "transformSpec": {}, "dimensionsSpec": { "dimensions": [ { "name": "field1int", "type": "long" }, "field2string" ] }, "granularitySpec": { "queryGranularity": "none", "rollup": false } } } }
the solution was 1. change delimeter in file from ; to , 2. change "inputFormat"."type": "csv" in ingestion spec now all "test sting" parsed as expected. It's strange for me that tsv input time does't supports double quotes for the values