I am trying to ingest presto-on-hive, but got foll...
# ingestionc
clean-tomato-22549
09/27/2022, 3:06 AMI am trying to ingest presto-on-hive, but got following error, could anyone help to check it. Thx
Copy code
datahub.ingestion.run.pipeline.PipelineInitError: Failed to configure source (presto-on-hive)
[2022-09-26 05:58:34,312] ERROR {datahub.entrypoints:195} - Command failed:
Failed to configure source (presto-on-hive) due to
'TSocket read 0 bytes'.
Run with --debug to get full stacktrace.clean-tomato-22549
09/27/2022, 3:09 AMMy datahub version is v0.8.44.
the recipe.yml is like
Copy code
source:
type: presto-on-hive
config:
host_port: <http://xxx.presto.com:9106|xxx.presto.com:9106>
database: db
scheme: 'hive'
schema_pattern:
allow:
- "default"
username: hive
options:
connect_args:
http_path: "/hive2"
auth: NOSASL
sink:
type: "datahub-rest"
config:
server: "<http://localhost:8080>"d
dazzling-judge-80093
09/27/2022, 6:11 AMIn Presto on Hive you have to connect to the Metastore db. I think here you tried to connect to something else.
c
clean-magician-86038
09/27/2022, 6:37 AMwhat are you going to ingest, presto or hive? the source type presto-on-hive doesn't exist . it should be hive if your target is hive.
c
clean-tomato-22549
09/27/2022, 8:05 AMInteresting. presto-on-hive is provided by the official doc.
Actually, I want to ingest hive metadata, while using presto query engine ability to do data profiling
c
clean-magician-86038
09/27/2022, 8:12 AMmy mistake. 😂
c
clean-tomato-22549
09/27/2022, 8:58 AMSo, how can I achieve this?
Copy code
Actually, I want to ingest hive metadata, while using presto query engine ability to do data profilingc
clean-magician-86038
09/27/2022, 8:59 AMuse source type hive instead of presto-on-hive, I think.
d
dazzling-judge-80093
09/27/2022, 9:01 AMUnfortunately you can currently only do profiling with the hive source.
Presto on hive source connects directly to the metastore database and that’s why it is much faster to ingest metadata but it can’t access the table data therefore it can’t do profiling.