Hi Folks! I was configuring the flink and flink sq...
# troubleshooting
f
Hi Folks! I was configuring the flink and flink sql cli, to use a s3 compatible storage. The last version that i've used is the 1.19.2, as i would need the jdbc connection and the 1.20.1 or 2.0-preview doesn't have the jdbc connectors. For each step, i executed a sql script to garantee that the configuration and libraries are correct (@rmoff thanks for article "Flink SQL and the Joy of JARs"). I've get the needed hadoop libraries (i put it in the lib folder of the flink), configured the config.yaml from flink and wrote a script to be called from sql-cli.sh. The libraries that i've downloaed to the lib folder are: • hadoop-common-3.3.4.jar • hadoop-shaded-guava-1.1.1.jar • hadoop-mapreduce-client-core-3.3.4.jar • hadoop-hdfs-3.3.4.jar • commons-logging-1.1.3.jar • stax2-api-4.2.2.jar • woodstox-core-5.3.0.jar One of my tests script, was to write a table in the s3 bucket, in the json format, and that was perfectly executed. When i changed to csv, the same script failed. The error message in the sql-cli console was:
Copy code
[ERROR] Could not execute SQL statement. Reason:
java.io.IOException: Stream closed
I don't have a clue of what could be! I've already tried another hadoop set of libraries like: • Hadoop 2.10: hadoop-common-2.10.2.jar, hadoop-hdfs-2.10.2.jar, hadoop-mapreduce-client-core-2.10.2.jar, commons-logging-1.1.3.jar, guava-11.0.2.jar, stax2-api-4.2.2.jar, woodstox-core-5.3.0.jar • Hadoop 3.3.6: hadoop-common-3.3.6.jar, hadoop-hdfs-3.3.6.jar, hadoop-mapreduce-client-core-3.3.6.jar, hadoop-shaded-guava-1.1.1.jar, commons-logging-1.1.3.jar, stax2-api-4.2.1.jar, woodstox-core-5.4.0.jar • Hadoop 3.4.1: commons-configuration2-2.10.1.jar, hadoop-common-3.4.1.jar, hadoop-hdfs-3.4.1.jar, hadoop-mapreduce-client-core-3.4.1.jar, hadoop-shaded-guava-1.3.0.jar, stax2-api-4.2.2.jar, woodstox-core-5.4.0.jar I've tried the combination of the libraries hadoop-aws and aws-java-sdk-core, but no success: • Hadoop 3.3.4: hadoop-aws-3.3.4.jar, aws-java-sdk-bundle-1.12.262.jar • Hadoop 3.3.6: hadoop-aws-3.3.6.jar, aws-java-sdk-bundle-1.12.367.jar • Hadoop 3.4.1: hadoop-aws-3.4.1.jar, aws-java-sdk-bundle-1.12.720.jar (tried too the bundle-2.24.6.jar from https://mvnrepository.com/artifact/software.amazon.awssdk/bundle) None of this actions changed the error i got. I've created a clean environment to test this, that is a VM with debian 12, java sdk 11, running on a VMWare on AMD64. The name "baian", is the hostname of the VM. The SQL Script that i've used is attached here, as well as the configuration and the logs generated (sql-cli, jobmanager and taskmanager). Please, anyone could give me light, where to go from here now ?
p
Hey there Flavio! Checking your logs it looks like there is some instability with s3 connection. Have you tried to configure some restart strategies on the job?
f
Hi Pedro! Thanks for your time! I tried to stick to simplest possible implementation. It's a install from scratch for MinIO, Flink and libraries. The S3 is the MinIO docker image running in the same host! The difference is that MinIO is a docker image and flink is direct/native install. How i could assert and investigate that instability ? Thanks