Hello guys, I'm trying to run a job that reads dat...
# troubleshooting
c
Hello guys, I'm trying to run a job that reads data from Kafka, maps it to an entity, then maps it again to another DTO entity, and finally writes the data to Iceberg. At the moment, I have communication between Flink and Iceberg, but when I try to run the job from a .jar file, it shows me this error (Flink 1.18.1, Hive 3.1.3, Iceberg 1.5.2):
Copy code
org.apache.flink.runtime.rest.handler.RestHandlerException: Could not execute application. at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$1(JarRunHandler.java:114) at java.base/java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.CompletionException: org.apache.flink.util.FlinkRuntimeException: Could not execute application. at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ... 2 more Caused by: org.apache.flink.util.FlinkRuntimeException: Could not execute application. at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:88) at org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:70) at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:108) ... 2 more Caused by: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Failed to get table schema from deserializer at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105) at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84) ... 4 more Caused by: org.apache.flink.table.catalog.exceptions.CatalogException: Failed to get table schema from deserializer at org.apache.flink.table.catalog.hive.client.HiveShimV110.getFieldsFromDeserializer(HiveShimV110.java:116) at org.apache.flink.table.catalog.hive.util.HiveTableUtil.getNonPartitionFields(HiveTableUtil.java:730) at org.apache.flink.table.catalog.hive.util.HiveTableUtil.extractHiveTableInfo(HiveTableUtil.java:218) at org.apache.flink.table.catalog.hive.util.HiveTableUtil.createSchema(HiveTableUtil.java:117) at org.apache.flink.table.catalog.hive.HiveCatalog.instantiateCatalogTable(HiveCatalog.java:761) at org.apache.flink.table.catalog.hive.HiveCatalog.getTable(HiveCatalog.java:468) at org.apache.flink.table.catalog.CatalogManager.getPermanentTable(CatalogManager.java:667) at org.apache.flink.table.catalog.CatalogManager.getTable(CatalogManager.java:561) at org.apache.flink.table.catalog.CatalogManager.getTableOrError(CatalogManager.java:618) at org.apache.flink.table.api.internal.TableImpl.insertInto(TableImpl.java:435) at org.apache.flink.table.api.internal.TableImpl.insertInto(TableImpl.java:425) at org.apache.flink.table.api.Table.executeInsert(Table.java:1064) at flink.Main.main(Main.java:142) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ... 7 more Caused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.apache.flink.table.catalog.hive.client.HiveShimV110.getFieldsFromDeserializer(HiveShimV110.java:109) ... 24 more Caused by: MetaException(message:java.lang.AbstractMethodError Receiver class org.apache.iceberg.mr.hive.HiveIcebergSerDe does not define or inherit an implementation of the resolved method 'abstract void initialize(org.apache.hadoop.conf.Configuration, java.util.Properties)' of abstract class org.apache.hadoop.hive.serde2.AbstractSerDe.) at org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:98) at org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77) ... 29 more
I think the problem is
Copy code
MetaException(message:java.lang.AbstractMethodError Receiver class org.apache.iceberg.mr.hive.HiveIcebergSerDe does not define or inherit an implementation of the resolved method 'abstract void initialize(org.apache.hadoop.conf.Configuration, java.util.Properties)' of abstract class org.apache.hadoop.hive.serde2.AbstractSerDe.) at org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:98) at org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77) ... 29 more
After some research, I found that I need
iceberg-hive-metastore - 1.5.2
and
iceberg-flink-runtime-1.18 - 1.5.2
. However, there's another problem. I added these dependencies in my pom.xml following Maven Central:
Copy code
dependency
    groupIdorg.apache.iceberg/groupId
    artifactIdiceberg-hive-metastore/artifactId
    version1.5.2/version
/dependency
dependency
    groupIdorg.apache.iceberg/groupId
    artifactIdiceberg-flink-runtime-1.18/artifactId
    version1.5.2/version
/dependency
but when i run the compile stage i get this error:
error reading C:my-location\.m2\repository\org\apache\iceberg\iceberg-hive-metastore\1.5.2\iceberg-hive-metastore-1.5.2.jar; zip END header not found.
I'm creating the hive catalog like this:
Copy code
String catalogName = "catalog"; // Default catalog
        String databaseName = "db"; // Default database
        String hiveConfDir = "/opt/hive/conf";
        String hadoopConfDir = "/opt/hadoop/etc/hadoop";
        String hiveVersion = "3.1.3";
        String hiveWarehouse = "wasbs://x@y.blob.core.windows.net/user/hive/warehouse/";
        String thriftUri = "thrift://uri";
        HiveConf hiveConf = new HiveConf();

        hiveConf.addResource(new Path(hiveConfDir + "/hive-site.xml"));
        hiveConf.addResource(new Path(hadoopConfDir + "/core-site.xml"));
        hiveConf.addResource(new Path(hadoopConfDir + "/hdfs-site.xml"));
        hiveConf.set("hive.metastore.uris", thriftUri);
        hiveConf.set("hive.metastore.warehouse.dir", hiveWarehouse);

        HiveCatalog hiveCatalog = new HiveCatalog(catalogName, databaseName, hiveConf, hiveVersion);
        tableEnv.registerCatalog(catalogName, hiveCatalog);
        tableEnv.useCatalog(catalogName);
        tableEnv.useDatabase(databaseName);
If anyone knows how to solve any of these problems, I would appreciate it.
t
Official Apache Flink 1.19.0 via JARaaS Hybrid RAG - Code current 6/17/2024 Note: Sources at the end of the response It seems you are encountering multiple issues related to compatibility and errors in dependencies when running a Flink job that reads from Kafka and writes to Iceberg through a Hive catalog. Let's break down the issues and address them one by one. ### 1.
MetaException
and
AbstractMethodError
The error message indicates that the
HiveIcebergSerDe
class does not implement the
initialize
method from the
AbstractSerDe
class. This error typically arises due to an incompatibility between different versions of Hive and Iceberg libraries. Solution: Ensure that you are using compatible versions of the Iceberg and Hive libraries. Based on your descriptions, you might be already doing this, but here’s a more detailed way to ensure compatibility: - Verify that all dependencies for Iceberg, Flink, and Hive are compatible. - Double-check that you have the correct versions of the
iceberg-hive-metastore
and
iceberg-flink-runtime
modules. Your
pom.xml
dependencies should look like this, ensuring they point to compatible versions:
Copy code
xml
<dependency>
    <groupId>org.apache.iceberg</groupId>
    <artifactId>iceberg-hive-metastore</artifactId>
    <version>1.5.2</version>
</dependency>
<dependency>
    <groupId>org.apache.iceberg</groupId>
    <artifactId>iceberg-flink-runtime-1.18</artifactId>
    <version>1.5.2</version>
</dependency>
### 2.
zip END header not found
during compile stage The
zip END header not found
error suggests that there might be a corruption in your local Maven repository. Solution: - Delete the problematic jar from your local Maven repository and re-download it.
Copy code
bash
rm C:my-location\.m2\repository\org\apache\iceberg\iceberg-hive-metastore\1.5.2\iceberg-hive-metastore-1.5.2.jar
- Then, re-run your Maven build to download the dependencies again.
Copy code
bash
mvn clean install
### 3. Correct Setup for Hive Catalog To ensure your Hive catalog is set up correctly, make sure you use the correct version of Hive and Flink libraries and configure them properly. Here is your Hive catalog setup for reference:
Copy code
java
String catalogName = "catalog"; // Default catalog
String databaseName = "db"; // Default database
String hiveConfDir = "/opt/hive/conf";
String hadoopConfDir = "/opt/hadoop/etc/hadoop";
String hiveVersion = "3.1.3";
String hiveWarehouse = "<wasbs://x@y.blob.core.windows.net/user/hive/warehouse/>";
String thriftUri = "thrift://uri";

HiveConf hiveConf = new HiveConf();
hiveConf.addResource(new Path(hiveConfDir + "/hive-site.xml"));
hiveConf.addResource(new Path(hadoopConfDir + "/core-site.xml"));
hiveConf.addResource(new Path(hadoopConfDir + "/hdfs-site.xml"));
hiveConf.set("hive.metastore.uris", thriftUri);
hiveConf.set("hive.metastore.warehouse.dir", hiveWarehouse);

HiveCatalog hiveCatalog = new HiveCatalog(catalogName, databaseName, hiveConf, hiveVersion);
tableEnv.registerCatalog(catalogName, hiveCatalog);
tableEnv.useCatalog(catalogName);
tableEnv.useDatabase(databaseName);
### Additional Steps: Ensure Java Compatibility: Make sure your Java version is compatible with the versions of Flink, Hive, and Iceberg you are using. Use Flink Docker Image: If you need to create a custom Docker image, ensure all required dependencies are bundled correctly. An example Dockerfile could be:
Copy code
dockerfile
FROM flink:1.18.1-scala-2.12

# Add your dependencies here
RUN mkdir -p /opt/flink/lib \
    && wget -P /opt/flink/lib https://repo.maven.apache.org/maven2/org/apache/iceberg/iceberg-flink-runtime-1.18/1.5.2/iceberg-flink-runtime-1.18-1.5.2.jar \
    && wget -P /opt/flink/lib https://repo.maven.apache.org/maven2/org/apache/iceberg/iceberg-hive-metastore/1.5.2/iceberg-hive-metastore-1.5.2.jar
### Conclusion By ensuring compatibility between Flink, Hive, and Iceberg versions and correctly setting up your dependencies, you should be able to run your Flink job successfully. If issues persist, further debugging may require checking the full stack trace and ensuring no other version conflicts or missing dependencies. ### Sources Internal Documents: - Upgrading Flink Applications: upgrading.md - First Steps in Flink: local_installation.md - Docker Configuration for Flink: docker.md - Kubernetes Configuration: native_kubernetes.md
c
Thanks man 👍