Amit
07/11/2024, 5:43 AMHadoop FS is not available (not packaged with this application): NoClassDefFoundError : "org/apache/hadoop/conf/Configuration". - HadoopFSDelegationTokenProvider.init
I dont need hdfs support, just want to properly start that MiniCluster.
What to do to bypass that?
Thanks Alot!! πD. Draco O'Brien
07/11/2024, 9:05 AMimport org.apache.flink.configuration.Configuration;
Configuration config = new Configuration();
config.setBoolean("fs.hdfs.enabled", false); // disable Hadoop filesystem support
config.setString("high-availability", "none"); // disable high availability mode which might also look for HDFSD. Draco O'Brien
07/11/2024, 9:08 AMimport org.apache.flink.api.common.JobExecutionResult;
import org.apache.flink.api.java.ExecutionEnvironment;
import org.apache.flink.runtime.minicluster.MiniCluster;
import org.apache.flink.runtime.minicluster.MiniClusterConfiguration;
MiniClusterConfiguration miniClusterConfig = new MiniClusterConfiguration.Builder()
.setConfiguration(config) // Use the custom configuration
.setNumTaskManagers(1)
.setNumSlotsPerTaskManager(1)
.build();
try (MiniCluster miniCluster = new MiniCluster(miniClusterConfig)) {
miniCluster.start();
// execute your job or
//
} catch (Exception e) {
e.printStackTrace();
}Amit
07/11/2024, 11:06 AM