https://datahubproject.io logo
Join Slack
Powered by
# ingestion
  • i
    Not at the moment no, I’m simply implementing a cronjob resource in k8s
  • b
    Copy code
    Did not recognize type 'geometry' of column 'geom'
  "Did not recognize type '%s' of column '%s'" % (attype, name)
  • c
    Hi @chilly-spring-43918
  • m
    Just wanted to provide some status on the ingestion connectors, we now have support for the following systems (https://datahubproject.io/docs/metadata-ingestion#sources) • Kafka • MySQL • SQL Server • Hive • Postgres • Snowflake • BigQuery • Athena • Druid • LDAP Thanks for the contributions! Keep them coming! 🚀
    🎉 3
  • m
    @curved-crayon-1929: I just filed this (https://github.com/linkedin/datahub/issues/2280) for MongoDB. We'll get to it soon.
    👍 2
  • c
    Thanks a lot @mammoth-bear-12532
  • l
    welcome @wonderful-area-88986!
    thankyou 1
  • m
    <!here>: wanted to let people know that the much requested AWS Glue support is now in. Please try it out and improve it as needed. Thanks to the folks at Depop and Klarna for working on it! https://datahubproject.io/docs/metadata-ingestion/#aws-glue-glue
  • i
    This is on Ubuntu, if that makes a difference
  • i
    Copy code
    vmadmin@datahub-poc:~$ pip --version
pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)
  • i
    Maybe I should be using pip3?
  • c
    @colossal-nest-20040 has left the channel
  • m
    datahub_docker.sh ingest -c output.txt,datahub ingest -c output.txt
    datahub ingest -c output.txtdatahub_docker.sh ingest -c output.txt
  • m
    Processing was started from datahub/metadata-ingestion folder
  • s
    Hi I'm a little confused if we refer to Postgres as 'postgres' or 'postgresql' If I use Airflow native lineage, Datahub seems to prefer me to refer to postgres as urnlidataPlatform:postgres (that way datahub presents me with Postgres icons in the GUI) e.g native lineage message:
    Copy code
    c.l.m.k.MetadataAuditEventsProcessor - {com.linkedin.metadata.snapshot.DatasetSnapshot={urn=urn:li:dataset:(urn:li:dataPlatform:postgres,myapp.public.target_table_b,PROD), aspects=[{com.linkedin.dataset.UpstreamLineage={upstreams=[{auditStamp={actor=urn:li:corpuser:datahub, time=1619631366909}, type=TRANSFORMED, dataset=urn:li:dataset:(urn:li:dataPlatform:postgres,myapp.public.source_table_c,PROD)}, {auditStamp={actor=urn:li:corpuser:datahub, time=1619631366909}, type=TRANSFORMED, dataset=urn:li:dataset:(urn:li:dataPlatform:postgres,myapp.public.source_table_d,PROD)}]}}]}}
  • i
    Hi, I'm trying to ingest lineage data following the recently updated guide here: https://github.com/linkedin/datahub/tree/master/metadata-ingestion#using-datahubs-airflow-lineage-backend-recommended
  • i
    oops hit enter too quickly, rest of the message to follow...
  • i
    Answered my own question above: it was due to me using Airflow 2.0.1 as opposed to 2.0.2 (it's in the docs but I missed it). Comms between Airflow and Datahub working now
    🙌 4
  • s
    Hi all, I created a PR for ingesting lineage from Kafka Connect. Currently limited only for Debezium source connector. Please have a look and test 🙂 https://github.com/linkedin/datahub/pull/2516
    🎉 5
  • c
    Hi everyone, I just opened a PR to fix some issues we were having with the lineage/urns generated after ingesting metadata from dbt, if anyone here could test/and or review that would be great: https://github.com/linkedin/datahub/pull/2596
  • e
    Cool！How about the lineage of mysql and other source ?  Are there other components to handle them?
  • p
    Hi Folks, I tried ingesting metadata from Redshift, but I think the script gathers tables list at the beginning and if a temp table is dropped in between, the script reports table not found… (edited) This should be a warning and not an error in my opinion. I am using the REST api way, should I ingest using Kafka and try? will this issue  go away?
  • p
    Dont know the issue, I get this during ingestion, right at the end:
  • p
    However the catalog shows the tables
  • l
    @powerful-telephone-71997 can we setup a call to unblock you?
  • a
    Getting the following error while ingesting data from dbt to datahub GMS.
  • a
    Appreciate any help!
  • b
    i have the datahub running in docker containers in a VM, and I tried to ingest via REST API at the host container, using acryl-datahub package. I fed a json file to the rest api using
    Copy code
    datahub ingest -c <recipe name>
    I kept getting this message:
  • m
    Hey , i don't know if i'm on the right channel , i'm working to deploy datahub on k8s
  • m
    did i miss a componant in my deployement
    Copy code
    datahub-datahub-frontend-94d4cd6b5-nch6g                   1/1     Running   0          14h
datahub-datahub-gms-748884b4db-69cg2                       0/1     Running   1          6m53s
datahub-prerequisites-cp-schema-registry-dd6d4fb86-n55dz   2/2     Running   0          14h
datahub-prerequisites-kafka-0                              1/1     Running   0          9m15s
datahub-prerequisites-neo4j-community-0                    1/1     Running   0          14h
datahub-prerequisites-zookeeper-0                          1/1     Running   0          14h
elasticsearch-master-0                                     1/1     Running   0          14h
elasticsearch-master-1                                     1/1     Running   0          14h
elasticsearch-master-2                                     1/1     Running   0          14h
flux-68bdf85d98-gmblt                                      1/1     Running   0          14h
1...131132133...144Latest