http://starrocks.io logo
Join Slack
Powered by
:wave: Hello, team! So we are interested in using...
# questions-and-troubleshooting
k
👋 Hello, team! So we are interested in using Starrocks with a Nessie external catalog with GCS storage for the iceberg data. Currently Starrocks is deployed in our cluster using the Starrocks operator from here: https://github.com/StarRocks/starrocks-kubernetes-operator We are deploying using a high availability scheme with FE nodes and CN backends. Docker image version: ubuntu:4.0.0-rc02 Nessie is deployed and hooked into GCS. I am capable of creating an external catalog in Starrocks using the following:
Copy code
CREATE EXTERNAL CATALOG 'nessieTestGCS'
COMMENT "Nessie GCS Catalog"
PROPERTIES
(
  "type"="iceberg",
  "iceberg.catalog.type"="rest",
  "iceberg.catalog.uri"="<http://plaid-nessie:19120/iceberg>",
  "iceberg.catalog.warehouse"="tenantDefault"
);
This catalog is created successfully, and I am able to create a database in the catalog. via:
Copy code
set catalog 'nessieTestGCS';
create database testGCS;
use testGCS;
The issue is that once I attempt to create a table, or perform any table operation beyond this point I am getting the following error:
Copy code
java.sql.SQLSyntaxErrorException: Cannot initialize FileIO implementation org.apache.iceberg.gcp.gcs.GCSFileIO: Cannot find constructor for interface org.apache.iceberg.io.FileIO
	Missing org.apache.iceberg.gcp.gcs.GCSFileIO [java.lang.ClassNotFoundException: org.apache.iceberg.gcp.gcs.GCSFileIO]
Do I need to add these packages to the Docker images provided? I am attempting to run these commands via Dbeaver using the Starrocks driver they support. I suppose it is possible that the issue could be with the DBeaver driver also, and have little to do with the database itself. Any support would be appreciated.
👍 1
r
Could you mind to provide more context about the error? In the 
fe/log/fe.log
?
As well. what's the starrocks version you're using?
k
The full error log:
Copy code
java.lang.IllegalArgumentException: Cannot initialize FileIO implementation org.apache.iceberg.gcp.gcs.GCSFileIO: Cannot find constructor for interface org.apache.iceberg.io.FileIO
        Missing org.apache.iceberg.gcp.gcs.GCSFileIO [java.lang.ClassNotFoundException: org.apache.iceberg.gcp.gcs.GCSFileIO]
        at org.apache.iceberg.CatalogUtil.loadFileIO(CatalogUtil.java:386) ~[iceberg-core-1.9.0.jar:?]
        at org.apache.iceberg.rest.RESTSessionCatalog.newFileIO(RESTSessionCatalog.java:983) ~[iceberg-core-1.9.0.jar:?]
        at org.apache.iceberg.rest.RESTSessionCatalog.tableFileIO(RESTSessionCatalog.java:1001) ~[iceberg-core-1.9.0.jar:?]
        at org.apache.iceberg.rest.RESTSessionCatalog.loadTable(RESTSessionCatalog.java:448) ~[iceberg-core-1.9.0.jar:?]
        at com.starrocks.connector.iceberg.rest.IcebergRESTCatalog.getTable(IcebergRESTCatalog.java:147) ~[starrocks-fe.jar:?]
        at com.starrocks.connector.iceberg.CachingIcebergCatalog.getTable(CachingIcebergCatalog.java:202) ~[starrocks-fe.jar:?]
        at com.starrocks.connector.iceberg.IcebergMetadata.getTable(IcebergMetadata.java:410) ~[starrocks-fe.jar:?]
        at com.starrocks.connector.CatalogConnectorMetadata.getTable(CatalogConnectorMetadata.java:148) ~[starrocks-fe.jar:?]
        at com.starrocks.server.MetadataMgr.lambda$getTable$5(MetadataMgr.java:502) ~[starrocks-fe.jar:?]
        at java.util.Optional.map(Optional.java:260) ~[?:?]
        at com.starrocks.server.MetadataMgr.getTable(MetadataMgr.java:502) ~[starrocks-fe.jar:?]
        at com.starrocks.sql.analyzer.QueryAnalyzer.resolveTable(QueryAnalyzer.java:1495) ~[starrocks-fe.jar:?]
        at com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.resolveTableRef(QueryAnalyzer.java:554) ~[starrocks-fe.jar:?]
        at com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSelect(QueryAnalyzer.java:368) ~[starrocks-fe.jar:?]
        at com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSelect(QueryAnalyzer.java:288) ~[starrocks-fe.jar:?]
        at com.starrocks.sql.ast.SelectRelation.accept(SelectRelation.java:232) ~[starrocks-fe.jar:?]
        at com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.process(QueryAnalyzer.java:293) ~[starrocks-fe.jar:?]
        at com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryRelation(QueryAnalyzer.java:308) ~[starrocks-fe.jar:?]
        at com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryStatement(QueryAnalyzer.java:298) ~[starrocks-fe.jar:?]
        at com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryStatement(QueryAnalyzer.java:288) ~[starrocks-fe.jar:?]
        at com.starrocks.sql.ast.QueryStatement.accept(QueryStatement.java:72) ~[starrocks-fe.jar:?]
        at com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.process(QueryAnalyzer.java:293) ~[starrocks-fe.jar:?]
        at com.starrocks.sql.analyzer.QueryAnalyzer.analyze(QueryAnalyzer.java:126) ~[starrocks-fe.jar:?]
        at com.starrocks.sql.analyzer.Analyzer$AnalyzerVisitor.visitQueryStatement(Analyzer.java:472) ~[starrocks-fe.jar:?]
        at com.starrocks.sql.analyzer.Analyzer$AnalyzerVisitor.visitQueryStatement(Analyzer.java:194) ~[starrocks-fe.jar:?]
        at com.starrocks.sql.ast.QueryStatement.accept(QueryStatement.java:72) ~[starrocks-fe.jar:?]
        at com.starrocks.sql.ast.AstVisitor.visit(AstVisitor.java:105) ~[starrocks-fe.jar:?]
        at com.starrocks.sql.analyzer.Analyzer.analyze(Analyzer.java:191) ~[starrocks-fe.jar:?]
        at com.starrocks.sql.StatementPlanner.analyzeStatement(StatementPlanner.java:251) ~[starrocks-fe.jar:?]
        at com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:131) ~[starrocks-fe.jar:?]
        at com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:107) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.StmtExecutor.generateExecPlan(StmtExecutor.java:581) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:663) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:433) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:644) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:1002) ~[starrocks-fe.jar:?]
        at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:71) ~[starrocks-fe.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: java.lang.NoSuchMethodException: Cannot find constructor for interface org.apache.iceberg.io.FileIO
        Missing org.apache.iceberg.gcp.gcs.GCSFileIO [java.lang.ClassNotFoundException: org.apache.iceberg.gcp.gcs.GCSFileIO]
        at org.apache.iceberg.common.DynConstructors.buildCheckedException(DynConstructors.java:256) ~[iceberg-common-1.9.0.jar:?]
        at org.apache.iceberg.common.DynConstructors$Builder.buildChecked(DynConstructors.java:214) ~[iceberg-common-1.9.0.jar:?]
        at org.apache.iceberg.CatalogUtil.loadFileIO(CatalogUtil.java:383) ~[iceberg-core-1.9.0.jar:?]
        ... 39 more
        Suppressed: java.lang.ClassNotFoundException: org.apache.iceberg.gcp.gcs.GCSFileIO
                at jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) ~[?:?]
                at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) ~[?:?]
                at java.lang.ClassLoader.loadClass(ClassLoader.java:525) ~[?:?]
                at java.lang.Class.forName0(Native Method) ~[?:?]
                at java.lang.Class.forName(Class.java:467) ~[?:?]
                at org.apache.iceberg.common.DynConstructors$Builder.classForName(DynConstructors.java:227) ~[iceberg-common-1.9.0.jar:?]
                at org.apache.iceberg.common.DynConstructors$Builder.impl(DynConstructors.java:149) ~[iceberg-common-1.9.0.jar:?]
                at org.apache.iceberg.CatalogUtil.loadFileIO(CatalogUtil.java:382) ~[iceberg-core-1.9.0.jar:?]
                at org.apache.iceberg.rest.RESTSessionCatalog.newFileIO(RESTSessionCatalog.java:983) ~[iceberg-core-1.9.0.jar:?]
                at org.apache.iceberg.rest.RESTSessionCatalog.tableFileIO(RESTSessionCatalog.java:1001) ~[iceberg-core-1.9.0.jar:?]
                at org.apache.iceberg.rest.RESTSessionCatalog.loadTable(RESTSessionCatalog.java:448) ~[iceberg-core-1.9.0.jar:?]
                at com.starrocks.connector.iceberg.rest.IcebergRESTCatalog.getTable(IcebergRESTCatalog.java:147) ~[starrocks-fe.jar:?]
                at com.starrocks.connector.iceberg.CachingIcebergCatalog.getTable(CachingIcebergCatalog.java:202) ~[starrocks-fe.jar:?]
                at com.starrocks.connector.iceberg.IcebergMetadata.getTable(IcebergMetadata.java:410) ~[starrocks-fe.jar:?]
                at com.starrocks.connector.CatalogConnectorMetadata.getTable(CatalogConnectorMetadata.java:148) ~[starrocks-fe.jar:?]
                at com.starrocks.server.MetadataMgr.lambda$getTable$5(MetadataMgr.java:502) ~[starrocks-fe.jar:?]
                at java.util.Optional.map(Optional.java:260) ~[?:?]
                at com.starrocks.server.MetadataMgr.getTable(MetadataMgr.java:502) ~[starrocks-fe.jar:?]
                at com.starrocks.sql.analyzer.QueryAnalyzer.resolveTable(QueryAnalyzer.java:1495) ~[starrocks-fe.jar:?]
                at com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.resolveTableRef(QueryAnalyzer.java:554) ~[starrocks-fe.jar:?]
                at com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSelect(QueryAnalyzer.java:368) ~[starrocks-fe.jar:?]
                at com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSelect(QueryAnalyzer.java:288) ~[starrocks-fe.jar:?]
                at com.starrocks.sql.ast.SelectRelation.accept(SelectRelation.java:232) ~[starrocks-fe.jar:?]
                at com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.process(QueryAnalyzer.java:293) ~[starrocks-fe.jar:?]
                at com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryRelation(QueryAnalyzer.java:308) ~[starrocks-fe.jar:?]
                at com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryStatement(QueryAnalyzer.java:298) ~[starrocks-fe.jar:?]
                at com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryStatement(QueryAnalyzer.java:288) ~[starrocks-fe.jar:?]
                at com.starrocks.sql.ast.QueryStatement.accept(QueryStatement.java:72) ~[starrocks-fe.jar:?]
                at com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.process(QueryAnalyzer.java:293) ~[starrocks-fe.jar:?]
                at com.starrocks.sql.analyzer.QueryAnalyzer.analyze(QueryAnalyzer.java:126) ~[starrocks-fe.jar:?]
                at com.starrocks.sql.analyzer.Analyzer$AnalyzerVisitor.visitQueryStatement(Analyzer.java:472) ~[starrocks-fe.jar:?]
                at com.starrocks.sql.analyzer.Analyzer$AnalyzerVisitor.visitQueryStatement(Analyzer.java:194) ~[starrocks-fe.jar:?]
                at com.starrocks.sql.ast.QueryStatement.accept(QueryStatement.java:72) ~[starrocks-fe.jar:?]
                at com.starrocks.sql.ast.AstVisitor.visit(AstVisitor.java:105) ~[starrocks-fe.jar:?]
                at com.starrocks.sql.analyzer.Analyzer.analyze(Analyzer.java:191) ~[starrocks-fe.jar:?]
                at com.starrocks.sql.StatementPlanner.analyzeStatement(StatementPlanner.java:251) ~[starrocks-fe.jar:?]
                at com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:131) ~[starrocks-fe.jar:?]
                at com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:107) ~[starrocks-fe.jar:?]
                at com.starrocks.qe.StmtExecutor.generateExecPlan(StmtExecutor.java:581) ~[starrocks-fe.jar:?]
                at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:663) ~[starrocks-fe.jar:?]
                at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:433) ~[starrocks-fe.jar:?]
                at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:644) ~[starrocks-fe.jar:?]
                at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:1002) ~[starrocks-fe.jar:?]
                at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:71) ~[starrocks-fe.jar:?]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
                at java.lang.Thread.run(Thread.java:840) ~[?:?
Version 8.0.33
@Robert Raharjo I don't want to ping you on this but here is your response.
r
I'm not sure if StarRocks supports the Nessie catalog since it unfortunately has not been tested. One thing you can try is to download this jar file and put it under 
be/lib
& 
fe/lib
and restart the cluster. https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-gcp/1.9.0/iceberg-gcp-1.9.0.jar
p
@Robert Raharjo creating a new build with the jar files appears to have fixed the issue. Any idea why GCP jar files are not included? Now I wonder if Azure will work out of the box too.
@Robert Raharjo just FYI... we opened an issue related this problem: https://github.com/StarRocks/starrocks/issues/64273
Open in Slack
PreviousNext
Powered by