https://flink.apache.org/ logo
Join Slack
Powered by
Hi All! Using Flink SQL abilities, i need to sink...
# troubleshooting
m
Hi All! Using Flink SQL abilities, i need to sink to kinesis from my source data this structure:
Copy code
{
  "str1": 970089079,
  "str2": 1688400374,
  "str3": {
    "str4": {
      "column1": {
        grouped_column2: 1,
        grouped_column2: 1,
        ....
      }
    }
  }
}
my approaches are in thread below…
str1-str4 are fixed constants, column1 and grouped_column2 are varying columns values within a tumble window. I found 2 approaches. First is to generate “str3” as a combination of JSON_OBJECTAGG and JSON_OBJECT. Managed to do this, but when i sink it with kinesis sink + json format, JSON_OBJECT part is escaped additionally which i don’t need; this is expected, but i can’t see how can i turn it off without writing my own connector. Second approach is nested complex datatype using ROW, MAP, ARRAY; i didn’t manage to achieve what i want as i didnt find a way to get this done -
Copy code
"column1": {
        grouped_column2: 1,
        grouped_column2: 1,
        ....
      }
What am i missing? Is there any good information how to define and select such complicated nested object with FLINK SQL?
a
I would recommend second approach with ROW datatype. You can define a SQL table with format=json. ROWs will be automatically encoded into proper JSON text
Copy code
select "a" as str1, "b" as str2, 
ROW(
  ROW (
     ROW( 
          1, 1
     )
   )
) as str3 from sourceTable
m
ok, with second approach i managed to do what i need to certain stage, ROWs actually work fine. But how can i achieve this -
Copy code
"column1": {
        id1: '1',
        id2: '1',
        ....
      }
? Within one grouped window, i have single “column1" value and several ids - id1, id2, … . All of them should be keys with constant value ‘1’ I tried this:
Copy code
SELECT
      a,
      b,       
      (
           [column1, 
                [ id?, '1' ]
           ]
      ) as str3
But it says that ‘id’ should be grouped, which is definitely not what i want. I struggle finding any real examples of using MAP for nested structures
a
I guess you would need to do select-from-select, if you want to encode grouped_column2 inside of the column1. The example I put above is more for Single Row Transformation type of queries, i.e. when no aggregations are used
2 Views
Open in Slack
PreviousNext
Powered by