A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hi Team, Using the 0.9.3 version of Datahub, trying to Publish S3 Datasets, The pipeline runs but unable to view the Dataset in UI. I dont see any logs in the GMS pod ( Datahub is deployed on EKS )

Hi <@U029VFMT1GV>, when looking from the ingestion view do you see that entities have been ingested? <https://datahubproject.io/docs/quick-ingestion-guides/bigquery/configuration#validate-ingestion-runs>

<@U042KCFV9GX> - this is not via the Ingestion runs from UI. This is using the Ingestion scripts

```"""Metadata Ingestion DAG"""
from datahub.ingestion.run.pipeline import Pipeline
import csv
import logging



if __name__ == '__main__':
    env = "development"
    PIPELINE_CONFIG = {
                "source": {
                    "type": "s3",
                    "config": {
                        "env": "PROD",
                        "profiling":{
                            "enabled":"false"
                        },
                        "path_specs": [
                            {"include":"s3a://&lt;path&gt;"}
                        ],
                        "aws_config": {
                            "aws_access_key_id": "&lt;awsseretkey&gt;",
                            "aws_region": "us-east-1",
                            "aws_secret_access_key": "&lt;awssecretaccesskey&gt;"
                        }
                    }
                },
                "transformers": [
                    {
                        "type": "set_dataset_browse_path",
                        "config": {
                            "path_templates": [
                                "&lt;browsepath&gt;"
                            ]
                        }
                    },
                "sink": {
                    "type": "datahub-rest",
                    "config": {
                        "server": GMS_SERVER_DICT[env]
                    }
                }
            }
    print(f" Pipeline config is {PIPELINE_CONFIG}")
    pipeline = Pipeline.create(PIPELINE_CONFIG)
    pipeline.run()
    logging.warning('Pipeline Execution!')
    pipeline.raise_from_status()
    print(pipeline.log_ingestion_stats())```

Are we able to give a regex pattern to include the files? or we are able to give only a specific file ?

Hi <@U029VFMT1GV>, you can use `profile_patterns.allow` and `profile_patterns.deny` for regex filtering <https://datahubproject.io/docs/generated/ingestion/sources/s3#config-details>

<@U042KCFV9GX> - Are we able to publish S3 Datasets at folder level ? instead of creating a dataset for each part file ? is there a configuration for this ?