https://datahubproject.io logo
Join Slack
Powered by
# ingestion
  • l
    wanted to deny table which starts with temp_, can anyone help me with the yml file???
  • p
    Hi, where can we learn more about:
    Copy code
    Data Quality — Metadata Model Support
Data Quality test results are now supported in the DataHub backend metadata model!
    Would love to see how we might be able to integrate our data quality checks
  • i
    Hi, like @cool-iron-6335 on his message : https://datahubspace.slack.com/archives/CUMUWQU66/p1626083174047000, I would like to know if there is a ingestion source or recipe to ingest HDFS files (mostly CSV) into datahub. I have also the same question for Hbase (namespace, table eventually column familly, ... ) I saw that there is a Hive metadata ingestion source, but I would like to also add HDFS ans Hbase ressources to datahub. Thanks in advance for your knowledge.
  • g
    @helpful-optician-78938 Can't slide, click grey button just show this.
  • w
    How do I set the commit policy for the 
    DatahubIngestionCheckpointingProvider
    ? is that configurable somehow? https://github.com/linkedin/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/run/pipeline.py#L235-L243
  • m
    [removed, wrong channel]
  • m
    Hi guys, I am having a problem with UI metadata ingestion, because the PIP we are using locally to install other plug-ins is using the mirror source, using the original connection timed out, causing execution to fail. Are there any good solutions or optimizations in this area ?
  • h
    Hi guys, one of the requirement we have is ingest metadata from: - BI Tools like SAP BO,  - ETL Tools like Informatica/Datastage   - RDBMS like Oracle Exadata - csv/excel/txt files We wonder if these sources are in scope of plugins development in coming roadmaps ( Q1 2022) or better if someone has already developed them by himself. Is it possible? Does anyone else have the same needs as us? Thanks, Mauro
  • w
    I really appreciate any input
  • m
    Is there any requirement on the version of data source for es data source metadata ingestion For example, there are version differences between ES-5 and ES-7
  • s
    Can anyone help me on the above please?
  • l
    Cross-posted as I am not sure where to put this question
  • l
    👋 Hello, team!
  • d
    The error message complains about this as well but not a very user friendly way ->
  • r
    Screenshot 2022-02-28 at 1.10.52 PM.png
  • a
    I have to say that openlineage API endpoint would be really useful. We have implemented openlineage format emitter in our product as well so this would be an awesome way to onboard more users to DataHub
    ➕ 3
  • f
    @square-solstice-69079 did you also install python cx_oracle
  • b
    for glossary term, there is an option to introduce "inherit" and "contain" related terms. Right now, you need to specify the related terms manually for each and every term right, as the respective terms do not know who inherited from them? Is there any depth limitations for glossary terms? Also, whats the functional diff between a glossary node and browsepath "folder", since I don't see a placeholder for the glossaryNode description.
  • n
    @shy-parrot-64120 Where you able to find a fix for this? I see the same issue where the source dataset schema is not ingested which does not match the source ingested from postgres plugin.
  • q
    Is there an easy way to bulk update the fabric from 
    dev
    to 
    prod
    across all urns?
  • f
    Hello. I got a deployment of DataHub in AKS up an running using the default config and running the helm install commands. Now, I am looking to setup ingestion from Databricks via the Hive connector. In the documentation, it says to “Ensure that databricks-dbapi is installed. If not, use 
    pip install databricks-dbapi
    to install”. Any ideas on how I ensure this is installed inside my AKS Cluster?
  • a
    @stocky-midnight-78204 Seeing same things on my side
  • a
    image.png
  • f
    Bumping this up, because I think this could be a bug
  • s
    When I summit one sql job for running four insert sql query and two insert tasks are successfully executed and rest two tasks failed due to below error:
  • s
    Any one faced this issue?
  • b
    Thanks @green-football-43791, it looks like updating worked, but now I'm getting a connection timeout. I believe it's because urllib3 doesn't read the http_proxy variables. Is there a way to pass proxy info via the config or the 
    datahub ingest
    command?
  • k
    Hi All - I am setting up new quickstart. I am now stuck on this error. Connecting to snowflake. Has anybody seen this?
  • c
    any update on the above ? We also have same problem. Currently we are using open source version of delta lake along with a standalone hive metastore service (No hiveserver). For the metadata ingestion, I tried using a spark thrift server but it did not work. So looking for the options. is there any option available currently in datahub to ingest the metadata of delta lake tables from the hive standalone metastore service ?
  • d
    Hello! Are there any transformer to set dataset's domain?
1...135136137...144Latest