Hi everyone! I’ve met a strange one and i need to ...
# troubleshooting
m
Hi everyone! I’ve met a strange one and i need to ask you guys if im not missing anything. I have an issue with parsing gzipped json in plain file, but i’m cutting this to much more simple case: I have filesystem raw source, and simple sql which counts lines. For non-compressed test file of 1k lines, i get 1k as result of count. for same file, gzipped with terminal, i get 12 as a result. Docs says gzipped files should be decompressed on-the-fly, based on extension, but it doesn’t happen to me. There is no error, result is just wrong. More details in thread
Strangiest thing is that if applied to json log file (that’s my initial task), Flink actually parses PART of json objects from gzipped file. This is my SQL:
Copy code
def main():
    table_env.execute_sql(f"""
        CREATE TABLE logs_source (
            raw_row STRING
        ) WITH (
            'connector' = 'filesystem',
            'path' = '{logs_path}',
            'source.monitor-interval' = '10',
            'format' = 'raw'
        )
    """)

    table_env.execute_sql("""
        CREATE TABLE print_sink (
            ip_number BIGINT NOT NULL
        ) WITH (
            'connector' = 'print'
        )
    """)
    table_env.execute_sql(f"""
            INSERT INTO print_sink
                SELECT
                    COUNT(raw_row)         
                FROM logs_source
    """).wait()
I searched for any options or parameters to enable parsing for gzipped files specifically - but i’ve failed... Flink version 1.16.0, im using pyflink, Python 3.9 Where should i search for an answer? Thanks for any ideas!
m
m
It’s exactly that one. Thank you very much. Will wait for fix…