Hi team, when ingesting hive metadata into datahub...
# troubleshoot
m
Hi team, when ingesting hive metadata into datahub (v0.9.6.1), the execution log confuse me. It shows 42 tables are ingested in about 1min, while it takes 8min to ingest another 3 tables! Anyone can help?
Copy code
source:
  type: hive
  config:
    host_port: localhost:10000
    database_alias: hive
    schema_pattern:
      allow: ["^web_hudi$"]
        
sink:
  type: "datahub-rest"
  config:
    server: ${datahub_server}
    token: ${token}
πŸ“– 1
πŸ” 1
l
Hey there πŸ‘‹ I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request. Let's double check a few things first: βœ… There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? βœ… It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues? Did you find a solution to your issue? ❌ Sorry you weren't able to find a solution. I'm sending you some tips on info you can provide to help the community troubleshoot. Whenever you feel your issue is solved, please react βœ… to your original message to let us know!
a
What’s the size diff on those tables?
it would make sense if the slower three were millions of rows
m
@astonishing-answer-96712 The slower tables have not many rows but millions of partitions. And when I call the yaml file, I also see hive log which shows the process read the partitions. Any workaround to do with it?
a
Yeah, likely this is just an issue with the size of the tables- you might increase read time by allocating more memory to the containers