Camilo Marin
06/12/2024, 9:29 PMorg.apache.flink.runtime.rest.handler.RestHandlerException: Could not execute application. at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$1(JarRunHandler.java:114) at java.base/java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.CompletionException: org.apache.flink.util.FlinkRuntimeException: Could not execute application. at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ... 2 more Caused by: org.apache.flink.util.FlinkRuntimeException: Could not execute application. at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:88) at org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:70) at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:108) ... 2 more Caused by: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Failed to get table schema from deserializer at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105) at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84) ... 4 more Caused by: org.apache.flink.table.catalog.exceptions.CatalogException: Failed to get table schema from deserializer at org.apache.flink.table.catalog.hive.client.HiveShimV110.getFieldsFromDeserializer(HiveShimV110.java:116) at org.apache.flink.table.catalog.hive.util.HiveTableUtil.getNonPartitionFields(HiveTableUtil.java:730) at org.apache.flink.table.catalog.hive.util.HiveTableUtil.extractHiveTableInfo(HiveTableUtil.java:218) at org.apache.flink.table.catalog.hive.util.HiveTableUtil.createSchema(HiveTableUtil.java:117) at org.apache.flink.table.catalog.hive.HiveCatalog.instantiateCatalogTable(HiveCatalog.java:761) at org.apache.flink.table.catalog.hive.HiveCatalog.getTable(HiveCatalog.java:468) at org.apache.flink.table.catalog.CatalogManager.getPermanentTable(CatalogManager.java:667) at org.apache.flink.table.catalog.CatalogManager.getTable(CatalogManager.java:561) at org.apache.flink.table.catalog.CatalogManager.getTableOrError(CatalogManager.java:618) at org.apache.flink.table.api.internal.TableImpl.insertInto(TableImpl.java:435) at org.apache.flink.table.api.internal.TableImpl.insertInto(TableImpl.java:425) at org.apache.flink.table.api.Table.executeInsert(Table.java:1064) at flink.Main.main(Main.java:142) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ... 7 more Caused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.apache.flink.table.catalog.hive.client.HiveShimV110.getFieldsFromDeserializer(HiveShimV110.java:109) ... 24 more Caused by: MetaException(message:java.lang.AbstractMethodError Receiver class org.apache.iceberg.mr.hive.HiveIcebergSerDe does not define or inherit an implementation of the resolved method 'abstract void initialize(org.apache.hadoop.conf.Configuration, java.util.Properties)' of abstract class org.apache.hadoop.hive.serde2.AbstractSerDe.) at org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:98) at org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77) ... 29 more
I think the problem is
MetaException(message:java.lang.AbstractMethodError Receiver class org.apache.iceberg.mr.hive.HiveIcebergSerDe does not define or inherit an implementation of the resolved method 'abstract void initialize(org.apache.hadoop.conf.Configuration, java.util.Properties)' of abstract class org.apache.hadoop.hive.serde2.AbstractSerDe.) at org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:98) at org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77) ... 29 more
After some research, I found that I need iceberg-hive-metastore - 1.5.2 and iceberg-flink-runtime-1.18 - 1.5.2 .
However, there's another problem. I added these dependencies in my pom.xml following Maven Central:
dependency
groupIdorg.apache.iceberg/groupId
artifactIdiceberg-hive-metastore/artifactId
version1.5.2/version
/dependency
dependency
groupIdorg.apache.iceberg/groupId
artifactIdiceberg-flink-runtime-1.18/artifactId
version1.5.2/version
/dependency
but when i run the compile stage i get this error: error reading C:my-location\.m2\repository\org\apache\iceberg\iceberg-hive-metastore\1.5.2\iceberg-hive-metastore-1.5.2.jar; zip END header not found.
I'm creating the hive catalog like this:
String catalogName = "catalog"; // Default catalog
String databaseName = "db"; // Default database
String hiveConfDir = "/opt/hive/conf";
String hadoopConfDir = "/opt/hadoop/etc/hadoop";
String hiveVersion = "3.1.3";
String hiveWarehouse = "wasbs://x@y.blob.core.windows.net/user/hive/warehouse/";
String thriftUri = "thrift://uri";
HiveConf hiveConf = new HiveConf();
hiveConf.addResource(new Path(hiveConfDir + "/hive-site.xml"));
hiveConf.addResource(new Path(hadoopConfDir + "/core-site.xml"));
hiveConf.addResource(new Path(hadoopConfDir + "/hdfs-site.xml"));
hiveConf.set("hive.metastore.uris", thriftUri);
hiveConf.set("hive.metastore.warehouse.dir", hiveWarehouse);
HiveCatalog hiveCatalog = new HiveCatalog(catalogName, databaseName, hiveConf, hiveVersion);
tableEnv.registerCatalog(catalogName, hiveCatalog);
tableEnv.useCatalog(catalogName);
tableEnv.useDatabase(databaseName);
If anyone knows how to solve any of these problems, I would appreciate it.Tim Wolfe
06/17/2024, 11:44 PMMetaException and AbstractMethodError
The error message indicates that the HiveIcebergSerDe class does not implement the initialize method from the AbstractSerDe class. This error typically arises due to an incompatibility between different versions of Hive and Iceberg libraries.
Solution: Ensure that you are using compatible versions of the Iceberg and Hive libraries. Based on your descriptions, you might be already doing this, but here’s a more detailed way to ensure compatibility:
- Verify that all dependencies for Iceberg, Flink, and Hive are compatible.
- Double-check that you have the correct versions of the iceberg-hive-metastore and iceberg-flink-runtime modules.
Your pom.xml dependencies should look like this, ensuring they point to compatible versions:
xml
<dependency>
<groupId>org.apache.iceberg</groupId>
<artifactId>iceberg-hive-metastore</artifactId>
<version>1.5.2</version>
</dependency>
<dependency>
<groupId>org.apache.iceberg</groupId>
<artifactId>iceberg-flink-runtime-1.18</artifactId>
<version>1.5.2</version>
</dependency>
### 2. zip END header not found during compile stage
The zip END header not found error suggests that there might be a corruption in your local Maven repository.
Solution:
- Delete the problematic jar from your local Maven repository and re-download it.
bash
rm C:my-location\.m2\repository\org\apache\iceberg\iceberg-hive-metastore\1.5.2\iceberg-hive-metastore-1.5.2.jar
- Then, re-run your Maven build to download the dependencies again.
bash
mvn clean install
### 3. Correct Setup for Hive Catalog
To ensure your Hive catalog is set up correctly, make sure you use the correct version of Hive and Flink libraries and configure them properly.
Here is your Hive catalog setup for reference:
java
String catalogName = "catalog"; // Default catalog
String databaseName = "db"; // Default database
String hiveConfDir = "/opt/hive/conf";
String hadoopConfDir = "/opt/hadoop/etc/hadoop";
String hiveVersion = "3.1.3";
String hiveWarehouse = "<wasbs://x@y.blob.core.windows.net/user/hive/warehouse/>";
String thriftUri = "thrift://uri";
HiveConf hiveConf = new HiveConf();
hiveConf.addResource(new Path(hiveConfDir + "/hive-site.xml"));
hiveConf.addResource(new Path(hadoopConfDir + "/core-site.xml"));
hiveConf.addResource(new Path(hadoopConfDir + "/hdfs-site.xml"));
hiveConf.set("hive.metastore.uris", thriftUri);
hiveConf.set("hive.metastore.warehouse.dir", hiveWarehouse);
HiveCatalog hiveCatalog = new HiveCatalog(catalogName, databaseName, hiveConf, hiveVersion);
tableEnv.registerCatalog(catalogName, hiveCatalog);
tableEnv.useCatalog(catalogName);
tableEnv.useDatabase(databaseName);
### Additional Steps:
Ensure Java Compatibility: Make sure your Java version is compatible with the versions of Flink, Hive, and Iceberg you are using.
Use Flink Docker Image:
If you need to create a custom Docker image, ensure all required dependencies are bundled correctly. An example Dockerfile could be:
dockerfile
FROM flink:1.18.1-scala-2.12
# Add your dependencies here
RUN mkdir -p /opt/flink/lib \
&& wget -P /opt/flink/lib https://repo.maven.apache.org/maven2/org/apache/iceberg/iceberg-flink-runtime-1.18/1.5.2/iceberg-flink-runtime-1.18-1.5.2.jar \
&& wget -P /opt/flink/lib https://repo.maven.apache.org/maven2/org/apache/iceberg/iceberg-hive-metastore/1.5.2/iceberg-hive-metastore-1.5.2.jar
### Conclusion
By ensuring compatibility between Flink, Hive, and Iceberg versions and correctly setting up your dependencies, you should be able to run your Flink job successfully. If issues persist, further debugging may require checking the full stack trace and ensuring no other version conflicts or missing dependencies.
### Sources
Internal Documents:
- Upgrading Flink Applications: upgrading.md
- First Steps in Flink: local_installation.md
- Docker Configuration for Flink: docker.md
- Kubernetes Configuration: native_kubernetes.mdCamilo Marin
06/18/2024, 2:16 PM