https://datahubproject.io logo
Join Slack
Powered by
Hey, did datahub support hive/trino struct type. I...
# ingestion
d
Hey, did datahub support hive/trino struct type. I got some errors when ingest table with struct field type
Copy code
[2022-01-07 09:36:48,326] ERROR    {datahub.ingestion.run.pipeline:86} - failed to write record with workunit hive.ghtk.table_with_struct_type with ('Unable to emit metadata to DataHub GMS', {'exceptionClass': 'com.linkedin.restli.server.RestLiServiceException', 'stackTrace': 'com.linkedin.restli.server.RestLiServiceException [HTTP Status:422]: com.linkedin.metadata.entity.ValidationException: Failed to validate record with class com.linkedin.entity.Entity: ERROR :: /value/com.linkedin.metadata.snapshot.DatasetSnapshot/aspects/2/com.linkedin.schema.SchemaMetadata/fields/9/jsonProps :: unrecognized field found but not allowed\nERROR :: /value/com.linkedin.metadata.snapshot.DatasetSnapshot/aspects/2/com.linkedin.schema.SchemaMetadata/fields/10/jsonProps :: unrecognized field found but not allowed\nERROR ::
l
Can you please paste the record itself which causes this failure? You can use the file sink to write to a file for this.
cc @hundreds-photographer-13496 can you please take a look? ^
d
@loud-island-88694 Hi, what do y mean the record. The file ingestion .yml or the schema of table hive with struct type
The schema table
Copy code
CREATE TABLE ghtk.table_with_struct_type (   _key STRING,   timestamp DOUBLE,   msg STRING,   logger STRING,   thread STRING,   level STRING,   duration STRING,   endpoint STRING,   request_id STRING,   params STRUCT<address:STRING,call_google_geocoder:STRING,from:STRING,last:STRING,shop_code:STRING,source:STRING,type:STRING,province:STRING,street:STRING,district:STRING,ward:STRING>,   gmap_address STRUCT<id:BIGINT,lat:DOUBLE,lng:DOUBLE,name:STRING,osm_id:BIGINT>,   google_address STRUCT<lat:DOUBLE,lng:DOUBLE,success:STRING,normalized_address:STRING,formatted_address:STRING>,   matching_threshold STRING,   success BOOLEAN,   message STRING ) PARTITIONED BY (   data_date_key INT ) WITH SERDEPROPERTIES ('serialization.format'='1') STORED AS PARQUET LOCATION '<hdfs://nameservice1/user/>' TBLPROPERTIES ('spark.sql.partitionProvider'='catalog')
l
ok this is helpful. @hundreds-photographer-13496 can take a look when she is online
m
Hi @damp-ambulance-34232: you are using a newer client against an older datahub server. Upgrading the server should resolve this issue.
d
Hi @mammoth-bear-12532 Just downgrade my client version, and i can ingest hive table with struct type. Tks u
m
Cool! Yes downgrade will work too. We are working to make the client detect server version and automatically notify the user.
thank you 1
d
The last question, i cannot ingest kudu table in hive. Is there any work around with this issues.
Copy code
OperationalError: (pyhive.exc.OperationalError) TExecuteStatementResp(status=TStatus(statusCode=3, infoMessages=['*org.apache.hive.service.cli.HiveSQLException:Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.ClassNotFoundException Class org.apache.hadoop.hive.kudu.KuduSerDe not found:17:16', 'org.apache.hive.service.cli.operation.Operation:toSQLException:Operation.java:329', 'org.apache.hive.service.cli.operation.SQLOperation:runQuery:SQLOperation.java:258', 'org.apache.hive.service.cli.operation.SQLOperation:runInternal:SQLOperation.java:293', 'org.apache.hive.service.cli.operation.Operation:run:Operation.java:260', 'org.apache.hive.service.cli.session.HiveSessionImpl:executeStatementInternal:HiveSessionImpl.java:505', 'org.apache.hive.service.cli.session.HiveSessionImpl:executeStatement:HiveSessionImpl.java:480', 'org.apache.hive.service.cli.CLIService:executeStatement:CLIService.java:270', 'org.apache.hive.service.cli.thrift.ThriftCLIService:ExecuteStatement:ThriftCLIService.java:508', 'org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1437', 'org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1422', 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39', 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39', 'org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor:process:HadoopThriftAuthBridge.java:605', 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:286', 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1149', 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:624', 'java.lang.Thread:run:Thread.java:748', '*org.apache.hadoop.hive.metastore.api.MetaException:java.lang.ClassNotFoundException Class org.apache.hadoop.hive.kudu.KuduSerDe not found:29:13', 'org.apache.hadoop.hive.metastore.MetaStoreUtils:getDeserializer:MetaStoreUtils.java:456', 'org.apache.hadoop.hive.metastore.MetaStoreUtils:getDeserializer:MetaStoreUtils.java:435', 'org.apache.hadoop.hive.ql.metadata.Table:getDeserializerFromMetaStore:Table.java:280', 'org.apache.hadoop.hive.ql.metadata.Table:getDeserializer:Table.java:273', 'org.apache.hadoop.hive.ql.exec.DDLTask:describeTable:DDLTask.java:3271', 'org.apache.hadoop.hive.ql.exec.DDLTask:execute:DDLTask.java:418', 'org.apache.hadoop.hive.ql.exec.Task:executeTask:Task.java:199', 'org.apache.hadoop.hive.ql.exec.TaskRunner:runSequential:TaskRunner.java:97', 'org.apache.hadoop.hive.ql.Driver:launchTask:Driver.java:2200', 'org.apache.hadoop.hive.ql.Driver:execute:Driver.java:1843', 'org.apache.hadoop.hive.ql.Driver:runInternal:Driver.java:1563', 'org.apache.hadoop.hive.ql.Driver:run:Driver.java:1339', 'org.apache.hadoop.hive.ql.Driver:run:Driver.java:1334', 'org.apache.hive.service.cli.operation.SQLOperation:runQuery:SQLOperation.java:256'], sqlState='08S01', errorCode=1, errorMessage='Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.ClassNotFoundException Class org.apache.hadoop.hive.kudu.KuduSerDe not found'), operationHandle=None)
m
Seems like the hive server instance needs the kuduserde. Talk to your hive administrator?
plus1 1
Open in Slack
PreviousNext
Powered by