A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hi all,
Has anyone used Ingestion with profiling on the S3?
As my folder (Table) has several files, this has generated several entities, and I would like to unify them into a single object.
source:
    type: s3
    config:
        path_specs:
            -
                include: '<s3://bucket/table_folder/*.*>'
                table_name: kinesis_folder
        aws_config:
            aws_region: eu-central-1
            aws_role: 'arn:aws:iam:acc:role/s3_data_catalog_read_only'
        profiling:
            enabled: true

<@UV14447EU>  might be able to help here! :)

<@U068C8YJLBY> please, can you change the include to:
``` include: '<s3://bucket/{table}/*.*>'```
This way it won’t ingest individual files.