https://datahubproject.io logo
Join Slack
Powered by
# ingestion
  • w
    I’m noticing some odd behavior. I ingested some data via the 
    hive
    ingestion recipe leveraging 
    acryl-datahub[hive, datahub-kafka]==0.8.1.1
    which created a 
    DatasetSnapshot
    through the MCE Consumer which I can retrieve from the GMS with a request to the 
    /datasets?action=getSnapshot
    endpoint using the 
    urn
    I see in the kafka message. However, when I look in the datahub frontend, I can’t find the dataset anywhere and it doesn’t come back when search for the dataSet’s name. Kind of confused as to what the problem would be, any ideas?
    e
    b
    • 3
    • 30
  • s
    Apologies for what might be a philosophical question. Let's say we use Airflow to perform ML operations, there are a number of operations that are a good fit for DataHub - for example any data engineering that creates tabular data as inputs to model training or any tabular data outputed as part of model inference. DataHub gives me a great way of visualizing the pipeline and data. There are aspects of the pipeline that DataHub does not allow me to vizualize however: 1. Lets say I download zipped data from an FTP site using airflow for example, we do not seem to have an emitter for FTP sites with raw files or raw files sitting in S3 2. Once my data ie engineered, I might use it to train an ML model. It would be good to be able to visualize the output ML model in my dependencies as well Thoughts?
    l
    g
    +2
    • 5
    • 9
  • c
    Hello team. I am trying to ingest a dataJob through GMS API, but getting validation error, complaining about dataJobInfo.type. I tried setting that field to values like "SQL" or "HIVE" as indicated by sample datajob data in below link, but still failing:
    Copy code
    <https://github.com/linkedin/datahub/blob/0b75b4a96a801a91fe434c87f6c737d24d63eb14/metadata-ingestion/examples/mce_files/bootstrap_mce.json#L996>
    Validation errors from GMS API:
    Copy code
    [HTTP Status:400]: Parameters of method 'ingest' failed validation with error 'ERROR :: /snapshot/aspects/1/com.linkedin.datajob.DataJobInfo/type :: union type is not backed by a DataMap or null\n'\at com.linkedin.restli.server.RestLiServiceException.fromThrowable(RestLiServiceException.java:315)\n\tat com.linkedin.restli.server.BaseRestLiServer.buildPreRoutingError(BaseRestLiServer.java:158)\n\tat com.linkedin.restli.server.BaseRestLiServer.handleResourceRequest(BaseRestLiServer.java:198)
    b
    • 2
    • 1
  • c
    Example json passed to GMS API:
    Copy code
    {
  "snapshot": {
    "urn": "urn:li:dataJob:(urn:li:dataFlow:(glue,logistics-load,PROD),logistics-load)",
    "aspects": [
      {
        "com.linkedin.common.Ownership": {
          "owners": [
            {
              "owner": "urn:li:corpuser:dataservices",
              "type": "DATAOWNER"
            }
          ],
          "lastModified": {
            "time": 1581407189000,
            "actor": "urn:li:corpuser:dataservices"
          }
        }
      },
      {
        "com.linkedin.datajob.DataJobInfo": {
          "name": "logistics-load",
          "description": "Tranform and load logistics data into Redshift",
          "type": "SQL"
        }
      },
      {
        "com.linkedin.datajob.DataJobInputOutput": {
          "inputDatasets": [
            "urn:li:dataset:(urn:li:dataPlatform:s3,logistics_raw.shipment,PROD)"
          ],
          "outputDatasets": [
            "urn:li:dataset:(urn:li:dataPlatform:redshift,redshift_edw_production.edw_logistics_box,PROD)"
          ]
        }
      }
    ]
  }
}
    b
    • 2
    • 10
  • b
    Is it possible to reverse an ingestion once it's been succesfully ingested? I've had a whole section of the application stop working after ingesting an improperly formated json and had to nuke everything before getting it to work again. Are rollbacks of any sort possible?
    🙌 1
    b
    • 2
    • 1
  • g
    Does Datahub allow ingestion of metadata from salesforce, powerBI and talend (etl)?
    h
    g
    e
    • 4
    • 12
  • h
    Hello! We are working on a way of backing up the data that we've manually entered via the datahub UI (particularly tags and column descriptions when not present in our data warehouse). To date we've been making backups of the mysql db and then restore a backup if ever we were to need to migrate datahub (ie from local to cloud/etc). However when restoring using this method elastic search needs to be re-indexed. The tags/descriptions show up, but can no-longer search for a particular tag. 1. Is there a way to re-index elastic search? 2. Is there a different way to backup this metadata that was not directly ingested but manually entered through the UI such that we don't lose if if out datahub instance ever goes down?
    b
    • 2
    • 2
  • w
    Hello. We are planning to evaluate datahub entities (mainly datasets) against company compliance rules which could be reported back to the dataset owners. When I think about it that seems very similar to the ingestion pipeline but reversed such that the source would be datahub, the transformers would be the compliance rules and the output a db, console, api etc. Has anyone else done something like this already? Would it make sense to make a feature request?
    h
    g
    l
    • 4
    • 16
  • a
    Hello! I have a question about data lineage: Besides visualizing data lineage on UI, is there anyway to use that lineage for detecting job failure and prevent data flowing from impaired upstream source ?
    g
    l
    • 3
    • 5
  • f
    Hi guys, any plans to ingest views with the Oracle connector. At this stage it's only picking up tables.
    b
    l
    +2
    • 5
    • 13
  • s
    Hello team, I would like to ask if DataHub currently support lineage for DockerOperator from airflow, I've been looking into some example but I couldnt find any.
    b
    g
    • 3
    • 8
  • s
    @gray-shoe-75895, yah I confirmed DockerOperator works, just need inlets, outlets as you said. So what happened to me is a bit weird.. I had to run 
    nuke.sh
    to clear all and run 
    quickstart.sh
    again. Does anyone see any weird behavior like this?
    g
    • 2
    • 1
  • m
    Hi hows it going? I was scouring through your documentation to find an example to ingest Airflow DAG metadata (not lineage), but was unsuccessful (high chance I overlooked something). Inspired by your demo (https://demo.datahubproject.io/browse/pipelines/airflow/prod) would love to know how to see Airflow DAG and task metadata in in DataHub. Any guidance will be appreciated. Thank you in advance.
    h
    m
    g
    • 4
    • 9
  • g
    in the table or schema_pattern variables in the ingest recipes is it possible to use regex expressions? There's a bunch of 'system' tables in my schema that I want to exclude from ingesting. Their naming convention is i<sequencenr> so would be easiest to exclude with a regex pattern...
    s
    • 2
    • 3
  • c
    Hello! i have test json /opt/datahub/metadata-ingestion/examples/mce_files/bootstrap_mce.json 1000 rows, how you can use huge file? i would like to add table
    g
    • 2
    • 10
  • m
    from the log i see
    Copy code
    2021-06-17 13:31:21.912:WARN:oejs.HttpChannel:qtp544724190-9: /health
datahub-datahub-gms-748884b4db-69cg2 datahub-gms 2021-06-17T13:31:21.918950727Z javax.servlet.ServletException: javax.servlet.UnavailableException: Servlet Not Initialized
    g
    • 2
    • 3
  • g
    Why did i get this error ? I had start DataHub successfully and could access and browse on ui. But i got this error when i tried to ingest metadata. I've never got this error before
    g
    • 2
    • 11
  • c
    Hello! delete dataset "aspects": [ { "com.linkedin.pegasus2avro.common.Status": { "removed": null } please help delete schema
    b
    e
    +2
    • 5
    • 30
  • i
    Hi folks, when do you trigger an ingestion from a data source? For example, if I want to get the metadata about a Kafka broker (https://datahubproject.io/docs/metadata-ingestion/#kafka-metadata-kafka), I can run the 'datahub' CLI command manually and get that data. Is it expected that this would be set up to run on a schedule? Or perhaps it could be triggered as part of a DAG?
    g
    s
    +2
    • 5
    • 5
  • b
    was wondering if anyone could shed some light on this. im trying to programmatically create datasets using the example in /datahub/metadata-ingestion/examples/library/lineage_emitter_rest.py instead of using the CLI method. i could create institutional memory, dataset properties of a new dataset by modifying the example of lineage, and the dataset could be seen in the browser, not differently compared to the quickstart datasets. But there seem to be an extra step that I missed when i try creating a SchemaMetadataClass. the REST API accepted my request, but i cant browse the newly created dataset. the error message in the browser read:
    Copy code
    Exception while fetching data (/browse) : java.lang.RuntimeException: Failed to execute browse: entity type DATASET, path [prod, goonrtpe], filters: null, start: 0, count: 10
    mae-consumer logs says:
    Copy code
    datahub-mae-consumer      | org.springframework.kafka.listener.ListenerExecutionFailedException: Listener method 'public void com.linkedin.metadata.kafka.DataHubUsageEventsProcessor.consume(org.apache.kafka.clients.consumer.ConsumerRecord<java.lang.String, java.lang.String>)' threw exception; nested exception is java.lang.ClassCastException: com.linkedin.metadata.key.CorpUserKey cannot be cast to com.linkedin.identity.CorpUserInfo; nested exception is java.lang.ClassCastException: com.linkedin.metadata.key.CorpUserKey cannot be cast to com.linkedin.identity.CorpUserInfo
    i've compared the data stored in MySQL for the datasets created using programmatically generated ones and the pipeline created ones and dont see a difference. I'm using a slightly older version of datahub v0.8.1
    m
    b
    • 3
    • 7
  • g
    Hi, Can i ask where is my metadata is stored on my local machine ? How can i backup my metadata just in case something bad happened ? If it's okay, can you give me the directory ? Once I start docker, everything runs perfectly and I have no clue where it is ?
    b
    g
    +3
    • 6
    • 8
  • g
    Hi, Can i ask what is the default mechanism of metadata ingestion , either over Kafka or Rest ?
    m
    a
    b
    • 4
    • 4
  • c
  • a
    Hello everyone. Is there any way to add authentication to the GMA rest API?
    b
    • 2
    • 2
  • b
    [Fixed] Hello. When ingesting data for Charts, I'm trying to define multiple Data Sources but only one gets ingested. Is there a way I can fix this? Picture shows the json I'm ingesting and the final result (it only ingests the source I have defined last)
    c
    l
    g
    • 4
    • 5
  • b
    Will UTF-8 character encoding system be supported at any point? Currently non english characters get warped to random signs.
    g
    • 2
    • 4
  • s
    Hi the airflow integration in datahub is very cool. I wonder if there is much interest/demand in similar integration with Lyft Flyte?
    👍 1
    l
    h
    • 3
    • 3
  • c
    <!here> A couple people have been asking for tips on ingesting metadata from AWS S3, so we've put together a guide on using AWS Glue to crawl S3 buckets, which can then be ingested into DataHub. This setup prevents us from having to crawl large S3 buckets directly and also leverages Glue's powerful built-in classifiers. We now also support ingesting jobs and pipelines from Glue by default, so you'll be able to view the complete flow of information in DataHub. Feel free to message me with any questions!
    🎉 3
    🙌 3
    b
    • 2
    • 6
  • f
    Copy code
    Traceback (most recent call last):
........
  File "/usr/local/lib/python3.8/site-packages/avrogen/avrojson.py", line 272, in <listcomp>
    return [self._generic_from_json(x, writers_schema.items, readers_schema.items)
  File "/usr/local/lib/python3.8/site-packages/avrogen/avrojson.py", line 248, in _generic_from_json
    result = self._union_from_json(json_obj, writers_schema, readers_schema)
  File "/usr/local/lib/python3.8/site-packages/avrogen/avrojson.py", line 304, in _union_from_json
    raise schema.AvroException('Datum union type not in schema: %s', value_type)
avro.schema.AvroException: ('Datum union type not in schema: %s', 'com.linkedin.pegasus2avro.common.BrowsePaths')
    g
    • 2
    • 4
  • c
    Integration Survey: What orchestrator are you running?
1...567...144Latest