hi guys im trying to ingest data from s3, i upgrad...
# troubleshoot
c
hi guys im trying to ingest data from s3, i upgraded to the last version of datahub and i get this message after the ingestion:
Copy code
'warnings': {},
 'failures': {},
 'cli_version': '0.8.41.2',
 'cli_entry_location': '***/POC_DataHub/venv/lib/python3.8/site-packages/datahub/__init__.py',
 'py_version': '3.8.10 (default, Jun 22 2022, 20:18:18) \n[GCC 9.4.0]',
 'py_exec_path': '***/POC_DataHub/venv/bin/python3',
 'os_details': 'Linux-5.15.0-41-generic-x86_64-with-glibc2.29',
 'filtered': []}
Sink (datahub-rest) report:
{'records_written': '23',
 'warnings': [],
 'failures': [],
 'downstream_start_time': '2022-08-03 14:49:50.681803',
 'downstream_end_time': '2022-08-03 14:50:20.485661',
 'downstream_total_latency_in_seconds': '29.803858',
 'gms_version': 'v0.8.41'}

Pipeline finished successfully producing 22 workunits
so everything seems to be working fine, i check the database and i see info of the bucket, but nothing shows up in datahub. This is my recipe:
Copy code
source:
  type: s3
  config:
    path_specs:
      -
        include: "<s3://info_excluded/2021/11/01/*.*>"

    aws_config:
      aws_access_key_id: ***
      aws_secret_access_key: ***
      aws_region: ***
    env: "PROD"
    profiling:
      enabled: True

transformers:
  - type: "simple_add_dataset_tags"
    config:
      tag_urns:
        - "urn:li:tag:probando_s3"

sink:
  type: "datahub-rest"
  config:
    server: "<http://localhost:8080>"
do you know how to solve this? thanks
m
Hi @clever-air-4600 can you send the events to a file sink and paste the output you get here?
Copy code
sink:
  type: file
  config:
    filename: metadata_events.json
cc @careful-pilot-86309