Hi, I am trying to perform a very simple ingestion with delta lakes using the following recipe via UI

source:
    type: delta-lake
    config:
        base_path: '<s3://mybucketpath>'
        s3:
            aws_config:
                aws_access_key_id: XXXXXX
                aws_secret_access_key: XXXXXXX
sink:
    type: console

I get this error:

'"/tmp/datahub/ingest/venv-ffbf74a0-cc21-4052-b01e-9e37f43cf20d/lib/python3.9/site-packages/datahub/ingestion/source/delta_lake/config.py", '
           'line 79, in validate_config\n'
           '    75   @pydantic.root_validator()\n'
           '    76   def validate_config(cls, values: Dict) -> Dict[str, Any]:\n'
           '    77       values["_is_s3"] = is_s3_uri(values["base_path"])\n'
           '    78       if values["_is_s3"]:\n'
           '--> 79           if values["s3"] is None:\n'
           '    80               raise ValueError("s3 config must be set for s3 path")\n'
           '\n'
           '---- (full traceback above) ----\n'
           'File "/tmp/datahub/ingest/venv-ffbf74a0-cc21-4052-b01e-9e37f43cf20d/lib/python3.9/site-packages/datahub/cli/ingest_cli.py", line 106, in '
           'run\n'
           '    pipeline = Pipeline.create(pipeline_config, dry_run, preview, preview_workunits)\n'
           'File "/tmp/datahub/ingest/venv-ffbf74a0-cc21-4052-b01e-9e37f43cf20d/lib/python3.9/site-packages/datahub/ingestion/run/pipeline.py", line '
           '204, in create\n'
           '    return cls(\n'
           'File "/tmp/datahub/ingest/venv-ffbf74a0-cc21-4052-b01e-9e37f43cf20d/lib/python3.9/site-packages/datahub/ingestion/run/pipeline.py", line '
           '152, in __init__\n'
           '    self.source: Source = source_class.create(\n'
           'File '
           '"/tmp/datahub/ingest/venv-ffbf74a0-cc21-4052-b01e-9e37f43cf20d/lib/python3.9/site-packages/datahub/ingestion/source/delta_lake/source.py", '
           'line 99, in create\n'
           '    config = DeltaLakeSourceConfig.parse_obj(config_dict)\n'
           'File "pydantic/main.py", line 521, in pydantic.main.BaseModel.parse_obj\n'
           'File "pydantic/main.py", line 339, in pydantic.main.BaseModel.__init__\n'
           'File "pydantic/main.py", line 1064, in pydantic.main.validate_model\n'
           'File '
           '"/tmp/datahub/ingest/venv-ffbf74a0-cc21-4052-b01e-9e37f43cf20d/lib/python3.9/site-packages/datahub/ingestion/source/delta_lake/config.py", '
           'line 79, in validate_config\n'
           '    if values["s3"] is None:\n'
           '\n'
           "KeyError: 's3'\n"

Full log is attached

@plain-guitar-45103: looks like you might be missing a required field

aws_region

under the s3: aws_config: section

ok let me give that a try

@plain-guitar-45103 Somehow read table is failing here. I am suspecting that your s3 credentials are not working or path does not exists. Meanwhile i am working on getting elegant error messages.

Thank you for your response @careful-pilot-86309! I investigated a little further and shared my thoughts here