https://datahubproject.io logo
Join Slack
Powered by
hi, my ingestion works fine without any error. How...
# ingestion
g
hi, my ingestion works fine without any error. However I cannot see it in datahub web interface. Have you ever seen this? I just run "datahub ingest -c ingestion.yaml" and it shows pipeline finished successfully
b
Can you show your ingestion.yaml file?
g
Copy code
source:
 type: oracle
 config:
    host_port: 192.168.103.38:1521
    platform_instance: sos

    # Credentials
    username: ***********
    password: "************"

    # Options
    service_name: CSOS
    schema_pattern:
      allow:
        - "sos"

sink:
  type: "datahub-rest"
  config:
    server: "<http://localhost:6080>"

transformers:
  - type: "simple_add_dataset_tags"
    config:
      tag_urns:
        - "urn:li:tag:csos"
sure, here's the yml file used for the ingestion.
m
do you have the ingestion logs?
g
I found something interesting , when I see the ingestion log, it says scanned table=0, seems not table is scanned during the ingestion. That's why there's nothing to show in UI.
Copy code
Source (oracle) report:
{'workunits_produced': 7,
 'workunit_ids': ['container-info--urn:li:container:9ee09833c5582ad0567013897cf90b29',
                  'container-platforminstance--urn:li:container:9ee09833c5582ad0567013897cf90b29',
                  'container-subtypes--urn:li:container:9ee09833c5582ad0567013897cf90b29',
                  'container-info-cecusr-urn:li:container:617d4e72b9498153569dd0f6c8276632',
                  'container-platforminstance-cecusr-urn:li:container:617d4e72b9498153569dd0f6c8276632',
                  'container-subtypes-cecusr-urn:li:container:617d4e72b9498153569dd0f6c8276632',
                  'container-parent-container-cecusr-urn:li:container:617d4e72b9498153569dd0f6c8276632-urn:li:container:9ee09833c5582ad0567013897cf90b29'],
 'warnings': {},
 'failures': {},
 'cli_version': '0.8.34.2',
 'cli_entry_location': '/usr/local/lib/python3.8/dist-packages/datahub/__init__.py',
 'py_version': '3.8.10 (default, Mar 15 2022, 12:22:08) \n[GCC 9.4.0]',
 'py_exec_path': '/usr/bin/python3',
 'os_details': 'Linux-5.4.0-104-generic-x86_64-with-glibc2.29',
 'tables_scanned': 0,
 'views_scanned': 0,
 'entities_profiled': 0,
'filtered': ['activiti.*',
              'address.*',
.....
'sys_knowledge_base.*',
              'toad.*',
              'wmsys.*',
              'xdb.*',
              'xs$null.*'],
 'soft_deleted_stale_entities': [],
 'query_combiner': None}
Sink (datahub-rest) report:
{'records_written': 7,
 'warnings': [],
 'failures': [],
 'downstream_start_time': datetime.datetime(2022, 7, 1, 5, 27, 5, 610591),
 'downstream_end_time': datetime.datetime(2022, 7, 1, 5, 27, 5, 782290),
 'downstream_total_latency_in_seconds': 0.171699,
 'gms_version': 'v0.8.34'}

Pipeline finished successfully
with --debug option I can see more detailed log. Below is the event that is blocking ingestion to succeed.
Copy code
[2022-07-01 07:39:57,382] DEBUG    {datahub.telemetry.telemetry:200} - Sending init Telemetry
[2022-07-01 07:39:57,448] DEBUG    {datahub.telemetry.telemetry:211} - Error reporting telemetry: HTTPSConnectionPool(host='<http://api.mixpanel.com|api.mixpanel.com>', port=443): Max retries exceeded with url: /engage (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f809e242670>: Failed to establish a new connection: [Errno 111] Connection refused'))
[2022-07-01 07:39:57,448] DEBUG    {datahub.telemetry.telemetry:232} - Sending Telemetry
[2022-07-01 07:39:57,454] DEBUG    {datahub.telemetry.telemetry:236} - Error reporting telemetry: HTTPSConnectionPool(host='<http://api.mixpanel.com|api.mixpanel.com>', port=443): Max retries exceeded with url: /track (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f809e242cd0>: Failed to establish a new connection: [Errno 111] Connection refused'))
m
You can set 
export DATAHUB_TELEMETRY_ENABLED=false
That will disable telemetry tracking. But i dont think that caused the ingestion failure
g
yep it seems ingestion process is still completed event cannot access to telemetry URL.
2 Views
Open in Slack
PreviousNext
Powered by