Ari Huttunen
03/08/2023, 10:15 AMsave_sample_data = f"""
CREATE TABLE save_sample_data
PARTITIONED BY (EVENT_DAY, EVENT_HOUR)
WITH (
'connector' = 'filesystem',
'path' = '{EGRESS_S3_PATH_FOR_SAMPLES}',
'format' = 'parquet'
)
AS SELECT * FROM input_data
WHERE MINUTE(ts) = 1 AND HOUR(ts) = 10;
"""
table_env.execute_sql(save_sample_data)
I tried to put the PARTITIONED.. at every location. π
I also would have wanted to sample a percentage of the input data.Martijn Visser
03/08/2023, 10:38 AMAri Huttunen
03/08/2023, 10:38 AMMartijn Visser
03/08/2023, 10:39 AM