clever-air-4600
08/03/2022, 5:55 PM'warnings': {},
'failures': {},
'cli_version': '0.8.41.2',
'cli_entry_location': '***/POC_DataHub/venv/lib/python3.8/site-packages/datahub/__init__.py',
'py_version': '3.8.10 (default, Jun 22 2022, 20:18:18) \n[GCC 9.4.0]',
'py_exec_path': '***/POC_DataHub/venv/bin/python3',
'os_details': 'Linux-5.15.0-41-generic-x86_64-with-glibc2.29',
'filtered': []}
Sink (datahub-rest) report:
{'records_written': '23',
'warnings': [],
'failures': [],
'downstream_start_time': '2022-08-03 14:49:50.681803',
'downstream_end_time': '2022-08-03 14:50:20.485661',
'downstream_total_latency_in_seconds': '29.803858',
'gms_version': 'v0.8.41'}
Pipeline finished successfully producing 22 workunits
so everything seems to be working fine, i check the database and i see info of the bucket, but nothing shows up in datahub.
This is my recipe:
source:
type: s3
config:
path_specs:
-
include: "<s3://info_excluded/2021/11/01/*.*>"
aws_config:
aws_access_key_id: ***
aws_secret_access_key: ***
aws_region: ***
env: "PROD"
profiling:
enabled: True
transformers:
- type: "simple_add_dataset_tags"
config:
tag_urns:
- "urn:li:tag:probando_s3"
sink:
type: "datahub-rest"
config:
server: "<http://localhost:8080>"
do you know how to solve this? thanksmammoth-bear-12532
sink:
type: file
config:
filename: metadata_events.json
cc @careful-pilot-86309