Hi Super Devs: Have you got experience with this error ?

ERROR    {datahub.entrypoints:195} - Command failed: Failed to find a registered source for type redshift: 'str' object is not callable

Facing this error whilst ingesting a redshift source table. The recipe and creds are all fine

Hi Everyone, I have successfully imported my metadata from some s3 paths in datahub using

from datahub.ingestion.run.pipeline import Pipeline.

Now I want to ingest the data dictionaries of the various metada ingested. I see the option to do this via the datahub interface but I can't find any documentation for doing this programmatically. Has anyone done this in the past? Is there any suggestion?

Heya I believe I’ve found a bug with the Airflow plugin: https://github.com/datahub-project/datahub/issues/8058. In summary: when

datahub.capture_ownership_info = false

, the owners of Airflow pipelines are removed on each dag run.

Hi , I am trying to ingest mysql metadata to datahub using airflow code: from datetime import datetime, timedelta from airflow import DAG from airflow.operators.python import PythonVirtualenvOperator def ingest_from_mysql(): from datahub.ingestion.run.pipeline import Pipeline pipeline = Pipeline.create( # This configuration is analogous to a recipe configuration. { "source": { "type": "mysql", "config": { # If you want to use Airflow connections, take a look at the snowflake_sample_dag.py example. "username": "user", "password": "pass", "database": "db_name", "host_port": "localhost:3306", }, }, "sink": { "type": "datahub-rest", "config": {"server": "http://localhost:8080"}, }, } ) pipeline.run() pipeline.pretty_print_summary() pipeline.raise_from_status() with DAG( "datahub_mysql_ingest", default_args={ "owner": "airflow", }, description="An example DAG which ingests metadata from MySQL to DataHub", start_date=datetime(2022, 1, 1), schedule_interval=timedelta(days=1), catchup=False, ) as dag: task0 = PythonOperator( task_id="mysql_metadata", python_callable=ingest_from_mysql, dag=dag ) task0 when i run the dag i am getting

AttributeError: 'function' object has no attribute 'run'

Can anyone please help me with this

Hello. Not exactly an ingestion question per se, but posting here anyway. We are on version 0.10.1. Is it correctly understood that neither the OpenAPI or Rest.li APIs don't enforce authorisation, regardless of whether metadata service authentication is enabled or not?

Hi Team, when trying to ingest snowflake through UI getting below issue. it is failing to register source with below issue. ERROR: Could not find a version that satisfies the requirement acryl-datahub[datahub-kafka,datahub-rest,snowflake]==e06445c Full log: ---------------------------------------------------------------- 63-b9b5-3834b748f042 (snowflake) [exec_id=8b043d24-fc08-4263-b9b5-3834b748f042] 2023-05-12 101145.249616 INFO: Starting execution for task with name=RUN_INGEST [8b043d24-fc08-4263-b9b5-3834b748f042 logs] Obtaining venv creation lock... [8b043d24-fc08-4263-b9b5-3834b748f042 logs] Acquired venv creation lock [8b043d24-fc08-4263-b9b5-3834b748f042 logs] venv doesn't exist.. minting.. [8b043d24-fc08-4263-b9b5-3834b748f042 logs] Requirement already satisfied: pip in /tmp/datahub/ingest/venv-snowflake-e06445c/lib/python3.10/site-packages (22.3.1) [8b043d24-fc08-4263-b9b5-3834b748f042 logs] Collecting pip [8b043d24-fc08-4263-b9b5-3834b748f042 logs] Downloading pip-23.1.2-py3-none-any.whl (2.1 MB) [8b043d24-fc08-4263-b9b5-3834b748f042 logs] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 34.7 MB/s eta 00000 [8b043d24-fc08-4263-b9b5-3834b748f042 logs] Collecting wheel [8b043d24-fc08-4263-b9b5-3834b748f042 logs] Using cached wheel-0.40.0-py3-none-any.whl (64 kB) [8b043d24-fc08-4263-b9b5-3834b748f042 logs] Requirement already satisfied: setuptools in /tmp/datahub/ingest/venv-snowflake-e06445c/lib/python3.10/site-packages (65.5.0) [8b043d24-fc08-4263-b9b5-3834b748f042 logs] Collecting setuptools [8b043d24-fc08-4263-b9b5-3834b748f042 logs] Downloading setuptools-67.7.2-py3-none-any.whl (1.1 MB) [8b043d24-fc08-4263-b9b5-3834b748f042 logs] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 50.7 MB/s eta 00000 [8b043d24-fc08-4263-b9b5-3834b748f042 logs] Installing collected packages: wheel, setuptools, pip [8b043d24-fc08-4263-b9b5-3834b748f042 logs] Attempting uninstall: setuptools [8b043d24-fc08-4263-b9b5-3834b748f042 logs] Found existing installation: setuptools 65.5.0 [8b043d24-fc08-4263-b9b5-3834b748f042 logs] Uninstalling setuptools-65.5.0: [8b043d24-fc08-4263-b9b5-3834b748f042 logs] Successfully uninstalled setuptools-65.5.0 [8b043d24-fc08-4263-b9b5-3834b748f042 logs] Attempting uninstall: pip [8b043d24-fc08-4263-b9b5-3834b748f042 logs] Found existing installation: pip 22.3.1 [8b043d24-fc08-4263-b9b5-3834b748f042 logs] Uninstalling pip-22.3.1: [8b043d24-fc08-4263-b9b5-3834b748f042 logs] Successfully uninstalled pip-22.3.1 [8b043d24-fc08-4263-b9b5-3834b748f042 logs] Successfully installed pip-23.1.2 setuptools-67.7.2 wheel-0.40.0 [8b043d24-fc08-4263-b9b5-3834b748f042 logs] ERROR: Ignored the following versions that require a different python version: 0.8.24.1 Requires-Python >=3.6, <=3.9.9; 0.8.24.2 Requires-Python >=3.6, <=3.9.9; 0.8.24.3 Requires-Python >=3.6, <=3.9.9; 0.8.25 Requires-Python >=3.6, <=3.9.9; 0.8.25.0 Requires-Python >=3.6, <=3.9.9; 0.8.25.1 Requires-Python >=3.6, <=3.9.9; 0.8.25.2 Requires-Python >=3.6, <=3.9.9; 0.8.26.0 Requires-Python >=3.6, <=3.9.9; 0.8.26.1 Requires-Python >=3.6, <=3.9.9; 0.8.26.2 Requires-Python >=3.6, <=3.9.9; 0.8.26.3 Requires-Python >=3.6, <=3.9.9; 0.8.26.4 Requires-Python >=3.6, <=3.9.9; 0.8.26.5 Requires-Python >=3.6, <=3.9.9; 0.8.26.6 Requires-Python >=3.6, <=3.9.9; 0.8.26.7 Requires-Python >=3.6, <=3.9.9; 0.8.26.7rc1 Requires-Python >=3.6, <=3.9.9; 0.8.26.7rc2 Requires-Python >=3.6, <=3.9.9; 0.8.26.8 Requires-Python >=3.6, <=3.9.9; 0.8.26.8rc1 Requires-Python >=3.6, <=3.9.9; 0.8.27 Requires-Python >=3.6, <=3.9.9; 0.8.27.1 Requires-Python >=3.6, <=3.9.9; 0.8.27.1rc1 Requires-Python >=3.6, <=3.9.9; 0.8.27.2 Requires-Python >=3.6, <=3.9.9; 0.8.27.2rc1 Requires-Python >=3.6, <=3.9.9; 0.8.27.2rc2 Requires-Python >=3.6, <=3.9.9; 0.8.27.2rc3 Requires-Python >=3.6, <=3.9.9 [8b043d24-fc08-4263-b9b5-3834b748f042 logs] ERROR: Could not find a version that satisfies the requirement acryl-datahub[datahub-kafka,datahub-rest,snowflake]==e06445c (from versions: 0.0.2, 0.0.3, 0.0.4, 0.0.5, 0.1.0, 0.1.1, 0.1.2, 0.1.3, 0.2.0, 0.2.1, 0.2.2, 0.2.3, 0.3.0, 0.3.1, 0.3.2, 0.3.3, 0.3.4, 0.4.0, 0.8.1.0, 0.8.1.1, 0.8.1.2, 0.8.3.0, 0.8.3.1, 0.8.3.2, 0.8.3.3, 0.8.4.0, 0.8.5.0, 0.8.5.1, 0.8.5.2, 0.8.6.0, 0.8.6.1, 0.8.6.2, 0.8.6.3, 0.8.6.4, 0.8.6.5, 0.8.7.0, 0.8.8.0, 0.8.8.1, 0.8.8.2, 0.8.8.3, 0.8.8.4, 0.8.9.0, 0.8.10.0, 0.8.10.1, 0.8.10.2, 0.8.11.0, 0.8.11.1, 0.8.12.0, 0.8.13.0, 0.8.13.1, 0.8.14.0, 0.8.14.1, 0.8.14.2, 0.8.15.0, 0.8.15.1, 0.8.15.2, 0.8.15.3, 0.8.15.4, 0.8.15.5, 0.8.15.6, 0.8.15.7, 0.8.15.8, 0.8.15.9, 0.8.15.10, 0.8.16.0, 0.8.16.1, 0.8.16.2, 0.8.16.3, 0.8.16.4, 0.8.16.5, 0.8.16.6, 0.8.16.7, 0.8.16.8, 0.8.16.9, 0.8.16.11, 0.8.16.12, 0.8.17.0, 0.8.17.1, 0.8.17.2, 0.8.17.3, 0.8.17.4, 0.8.17.5, 0.8.17.6, 0.8.17.7, 0.8.18, 0.8.18.1, 0.8.19.0, 0.8.19.1, 0.8.20.0, 0.8.21.0, 0.8.22.1, 0.8.23.0, 0.8.23.1, 0.8.24.0, 0.8.28.0rc1, 0.8.28.0, 0.8.28.1, 0.8.29, 0.8.29.1, 0.8.29.2, 0.8.30.0, 0.8.31, 0.8.31.1rc1, 0.8.31.1, 0.8.31.2, 0.8.31.3rc1, 0.8.31.3, 0.8.31.4rc1, 0.8.31.4, 0.8.31.5rc1, 0.8.31.5, 0.8.31.6rc1, 0.8.31.6rc2, 0.8.31.6, 0.8.32rc1, 0.8.32rc2, 0.8.32rc3, 0.8.32rc4, 0.8.32, 0.8.32.1, 0.8.32.2rc1, 0.8.32.2, 0.8.32.3rc1, 0.8.32.3, 0.8.32.4rc1, 0.8.32.4rc2, 0.8.32.4, 0.8.32.5rc1, 0.8.32.5, 0.8.32.6rc2, 0.8.32.6rc3, 0.8.32.6, 0.8.32.7, 0.8.33rc1, 0.8.33, 0.8.33.1, 0.8.33.2rc1, 0.8.33.2rc2, 0.8.33.2, 0.8.33.3rc2, 0.8.33.3rc3, 0.8.33.3, 0.8.34, 0.8.34.1rc1, 0.8.34.1rc2, 0.8.34.1rc3, 0.8.34.1, 0.8.34.2rc1, 0.8.34.2rc2, 0.8.34.2rc3, 0.8.34.2rc4, 0.8.34.2, 0.8.34.3rc1, 0.8.35.0rc2, 0.8.35, 0.8.35.1rc1, 0.8.35.1, 0.8.35.2rc1, 0.8.35.2, 0.8.35.3rc1, 0.8.35.3, 0.8.35.4rc1, 0.8.35.4, 0.8.35.5rc1, 0.8.35.5, 0.8.35.6rc1, 0.8.35.6rc2, 0.8.35.6, 0.8.35.7rc1, 0.8.35.7, 0.8.35.8rc1, 0.8.35.8rc2, 0.8.35.8rc3, 0.8.36.0rc0, 0.8.36rc1, 0.8.36, 0.8.36.1rc1, 0.8.36.1rc2, 0.8.36.1rc6, 0.8.36.1rc7, 0.8.36.1rc8, 0.8.36.1rc9, 0.8.36.1rc10, 0.8.37rc0, 0.8.37, 0.8.38, 0.8.38.1rc0, 0.8.38.1rc1, 0.8.38.1, 0.8.38.2rc1, 0.8.38.2, 0.8.38.3rc1, 0.8.38.3, 0.8.38.4rc0, 0.8.38.4rc2, 0.8.38.4rc3, 0.8.38.4, 0.8.38.5rc0, 0.8.38.5, 0.8.39rc0, 0.8.39, 0.8.39.1rc1, 0.8.39.1rc2, 0.8.39.1rc3, 0.8.39.1rc4, 0.8.39.1rc5, 0.8.39.1rc6, 0.8.39.1rc7, 0.8.39.1rc8, 0.8.40rc1, 0.8.40, 0.8.40.1, 0.8.40.2rc0, 0.8.40.2, 0.8.40.3rc0, 0.8.40.3rc1, 0.8.40.3rc2, 0.8.40.3rc3, 0.8.40.3, 0.8.40.4rc1, 0.8.40.4rc2, 0.8.41rc2, 0.8.41, 0.8.41.1rc0, 0.8.41.1rc1, 0.8.41.1rc2, 0.8.41.1rc3, 0.8.41.1rc4, 0.8.41.1, 0.8.41.2rc0, 0.8.41.2rc1, 0.8.41.2, 0.8.41.3rc1, 0.8.41.3rc2, 0.8.41.3rc3, 0.8.42rc1, 0.8.42rc2, 0.8.42, 0.8.43rc2, 0.8.43rc3, 0.8.43rc4, 0.8.43, 0.8.43.1rc0, 0.8.43.1rc1, 0.8.43.1, 0.8.43.2rc0, 0.8.43.2rc1, 0.8.43.2, 0.8.43.3rc0, 0.8.43.3rc1, 0.8.43.3rc2, 0.8.43.3rc3, 0.8.43.3rc5, 0.8.43.3, 0.8.43.4rc1, 0.8.43.4rc2, 0.8.43.4, 0.8.43.5rc1, 0.8.43.5rc2, 0.8.43.5rc3, 0.8.43.5, 0.8.43.6rc0, 0.8.43.6rc1, 0.8.43.6, 0.8.44rc0, 0.8.44rc1, 0.8.44rc3, 0.8.44rc4, 0.8.44rc5, 0.8.44, 0.8.44.1rc0, 0.8.44.1rc1, 0.8.44.1rc2, 0.8.44.1rc3, 0.8.44.1rc4, 0.8.44.1, 0.8.44.2rc0, 0.8.44.2rc1, 0.8.44.2rc2, 0.8.44.2, 0.8.44.3rc0, 0.8.44.3rc1, 0.8.44.3rc2, 0.8.44.3rc3, 0.8.44.3rc4, 0.8.44.3, 0.8.44.4rc0, 0.8.44.4rc1, 0.8.44.4, 0.8.44.5rc0, 0.8.44.5rc1, 0.8.44.5rc2, 0.8.44.5rc3, 0.8.44.5, 0.8.44.6rc0, 0.8.45rc1, 0.8.45, 0.8.45.1rc0, 0.8.45.1rc2, 0.8.45.1rc3, 0.8.45.1rc4, 0.8.45.1rc5, 0.8.45.1, 0.8.45.2rc0, 0.8.45.2rc1, 0.8.45.2rc2, 0.8.45.2, 0.8.45.3rc0, 0.8.45.3rc1, 0.8.45.3rc2, 0.8.45.3rc3, 0.8.45.3rc4, 0.8.45.3rc5, 0.9.0rc4, 0.9.0rc5, 0.9.0rc6, 0.9.0, 0.9.0.1rc0, 0.9.0.1, 0.9.0.2rc0, 0.9.0.2rc1, 0.9.0.2rc2, 0.9.0.2rc3, 0.9.0.2rc4, 0.9.0.2, 0.9.0.3rc0, 0.9.0.3, 0.9.0.4rc0, 0.9.0.4, 0.9.0.5rc0, 0.9.0.5rc1, 0.9.0.5rc2, 0.9.0.5, 0.9.1rc0, 0.9.1, 0.9.1.1rc0, 0.9.1.1rc1, 0.9.1.1rc2, 0.9.2, 0.9.2.1rc0, 0.9.2.1rc1, 0.9.2.1rc2, 0.9.2.1, 0.9.2.2rc0, 0.9.2.2rc1, 0.9.2.2rc2, 0.9.2.2rc3, 0.9.2.2, 0.9.2.3rc1, 0.9.2.3rc2, 0.9.2.3rc3, 0.9.2.3rc4, 0.9.2.3, 0.9.2.4rc1, 0.9.2.4rc2, 0.9.2.4, 0.9.2.5rc1, 0.9.2.5rc3, 0.9.2.5rc4, 0.9.2.5rc5, 0.9.2.5rc8, [...truncated] [8b043d24-fc08-4263-b9b5-3834b748f042 logs] ERROR: No matching distribution found for acryl-datahub[datahub-kafka,datahub-rest,snowflake]==e06445c [2023-05-12 101153,435] INFO {acryl.executor.execution.sub_process_ingestion_task:119} - Got EOF from subprocess exec_id=8b043d24-fc08-4263-b9b5-3834b748f042 - stopping log monitor [2023-05-12 101153,436] INFO {acryl.executor.execution.sub_process_ingestion_task:184} - Detected subprocess exited exec_id=8b043d24-fc08-4263-b9b5-3834b748f042 [2023-05-12 101153,577] INFO {acryl.executor.execution.sub_process_ingestion_task:158} - Detected subprocess return code exec_id=8b043d24-fc08-4263-b9b5-3834b748f042 - stopping logs reporting [2023-05-12 101153,580] DEBUG {acryl.executor.execution.default_executor:136} - Cleaned up task for 8b043d24-fc08-4263-b9b5-3834b748f042 ------------------------------------------------------------------ please help me datahub action version == v0.0.10

This message was deleted.

Hi All, I tried to follow Spark Lineage using blog https://aws.amazon.com/blogs/big-data/part-2-deploy-datahub-using-aws-managed-services-and-ingest-metadata-from-aws-glue-and-amazon-redshift/ I was able to implement the steps mentioned under section:

Capture data lineage

AWS glue job ran successfully but on Datahub UI lineage was not created properly(as mentioned in the blog). My source and target are not visible in the lineage. Does anyone have any idea about this issue ? Datahub Version: v0.10.2

Hi Team, the database_alias for the data ingestion source is deprecated, so how can the databases with the same name for different data sources be placed in different containers

Hi, I am getting this error while running recipe file, It is saying pipeline_name required if stateful_ingestion enabled, what is pipeline_name, not mentioned in the documentation

Hey team, i am not able to ingest airflow metadata to datahub via datahub_kafka_default, its not throwing any error to me even if give wrong bootstrap url, My config file and log file:

Hi team, we are not able to ingest metadata from GX despite providing all required information like actions, libraries, and server details. In particular, even after pushing the checkpoint into datahub, we are not able to enable the validation tab and what we find out is the final metadata ingestion is not happening.

Someone help me out with this, Thanks in advance

Does anyone know what part of the Snowflake Ingestion is specifically responsible for creating the Table -> View Lineage?? I am running this ingestion with all the lineage flags set to True, it is succeeding, but I don't get the table-> view lineage?

pipeline_name: snowflake-lineage-ingestion
    source:
      type: snowflake
      config:
        # This option is recommended to be used to ingest all lineage
        ignore_start_time_lineage: false

        # Coordinates
        account_id: ${SNOWFLAKE_ACCOUNT_ID}
        warehouse: ${SNOWFLAKE_WAREHOUSE}

        # Credentials
        username: ${SNOWFLAKE_USERNAME}
        password: ${SNOWFLAKE_PASSWORD}
        role: ${SNOWFLAKE_ROLE}

        #this ingestion is just for lineage
        include_view_lineage: true
        include_table_lineage: true
        include_usage_stats: true
        include_column_lineage: true

        stateful_ingestion:
          enabled: true

        profiling:
          # Change to false to disable profiling
          enabled: false

    sink:
      type: "datahub-rest"
      config:
        server: ${DATAHUB_GMS_ENDPOINT}

Hi all, I create new bug: ingest/s3_data_lake_ingestor_skips_directories_with_similar_prefix https://github.com/datahub-project/datahub/issues/8077 and MR https://github.com/datahub-project/datahub/pull/8078 Please review && commit.

This message was deleted.

When I was trying to ingest sqlalchemy as type through UI I am getting this. Through CLI I know after installing missing module we can run datahub ingest command but is there any way to do it with UI ingestion? @better-orange-49102 @modern-artist-55754 Gentle reminder

Hi, having trouble in a lineage in mongodb ingestion. In documentation it is mentioned that mongodb has by default lineage enabled but when I ingest, I am not able to see any lineage. Can anyone suggest the solution?

Hey, do we have any plan of removing confluent-schema-registry , i can see its in progress in roadmap from many days, any ETA. Thanks in advance

Do anyone can share the document of data flow where the frontend pass the credentials and trigger the metadata ingestion?. I am able to crack the ingestion code in UI but not able to catch the flows :(

Hey team when are we expecting 0.10.3 version? Any ETA . Thanks

Has anyone successfully ingested tableau metadata? Ingestion ran successfully but 0 assests ingested.

'[2023-05-19 23:52:03,881] DEBUG    {datahub.ingestion.run.pipeline:199} - Source type:tableau,<class '
           "'datahub.ingestion.source.tableau.TableauSource'> configured\n"
           '[2023-05-19 23:52:03,881] INFO     {datahub.ingestion.run.pipeline:200} - Source configured successfully.\n'
           '[2023-05-19 23:52:03,882] INFO     {datahub.cli.ingest_cli:129} - Starting metadata ingestion\n'
           '[2023-05-19 23:52:03,885] DEBUG    {datahub.ingestion.source.tableau:364} - Query workbooksConnection to get 10 objects with offset 0\n'
           '[2023-05-19 23:52:03,885] INFO     {tableau.endpoint.metadata:61} - Querying Metadata API\n'
           '[2023-05-19 23:52:04,041] INFO     {tableau.endpoint.auth:66} - Signed out\n'
           '[2023-05-19 23:52:04,042] INFO     {datahub.ingestion.reporting.file_reporter:54} - Wrote SUCCESS report successfully to '
           "<_io.TextIOWrapper name='/tmp/datahub/ingest/9906dc8e-3870-45b0-8c86-d08b5360adf7/ingestion_report.json' mode='w' encoding='UTF-8'>\n"
           '[2023-05-19 23:52:04,042] INFO     {datahub.cli.ingest_cli:150} - Finished metadata ingestion\n'
           '[2023-05-19 23:52:04,042] DEBUG    {datahub.telemetry.telemetry:243} - Sending Telemetry\n'
           '\n'
           'Cli report:\n'
           "{'cli_version': '0.9.1',\n"
           " 'cli_entry_location': '/tmp/datahub/ingest/venv-tableau-0.9.1/lib/python3.10/site-packages/datahub/__init__.py',\n"
           " 'py_version': '3.10.7 (main, Oct  5 2022, 14:33:54) [GCC 10.2.1 20210110]',\n"
           " 'py_exec_path': '/tmp/datahub/ingest/venv-tableau-0.9.1/bin/python3',\n"
           " 'os_details': 'Linux-5.10.147-133.644.amzn2.x86_64-x86_64-with-glibc2.31',\n"
           " 'mem_info': '64.43 MB'}\n"
           'Source (tableau) report:\n'
           "{'events_produced': '0',\n"
           " 'events_produced_per_sec': '0',\n"
           " 'event_ids': [],\n"
           " 'warnings': {},\n"
           " 'failures': {},\n"
           " 'soft_deleted_stale_entities': [],\n"
           " 'start_time': '2023-05-19 23:52:03.500064 (now).',\n"
           " 'running_time': '0.65 seconds'}\n"
           'Sink (datahub-rest) report:\n'
           "{'total_records_written': '0',\n"
           " 'records_written_per_second': '0',\n"
           " 'warnings': [],\n"
           " 'failures': [],\n"
           " 'start_time': '2023-05-19 23:52:03.380872 (now).',\n"
           " 'current_time': '2023-05-19 23:52:04.149734 (now).',\n"
           " 'total_duration_in_seconds': '0.77',\n"
           " 'gms_version': 'v0.9.1',\n"
           " 'pending_requests': '0'}\n"
           '\n'
           ' Pipeline finished successfully; produced 0 events in 0.65 seconds.\n'

```

Hello - is there a way i can start clean with my ingestion? I ingested GBQ and DBT data but i would like to start over. This did not clean the references -

datahub delete --entity_type dataset --env prod --hard.

I also recreated the pods but that also did not help.

can i get the concept of create a source for my airflow ? I am using datahub quickstart and not see any airflow source in ingestion

Hello Team Is there a feature available to display transformations applied on the column lineage ?

Hi Team, I would like to reframe the same question in my previous message if it was not clear enough. So, we are trying to integrate Datahub with Great Expectations and in this process, we have set up the expectations suite, stored the validation results, added the checkpoint to the snowflake config context, and ran the same to see if the results were appearing in the validation tab of datahub for that particular table. Initially, we have done this on the datahub version which is running on Azure Kubernetes services but we had a doubt if this would succeed as this is a cloud environment. So, we started our work on a datahub hosted on a local server, but the ingestion is not successful either way. As I mentioned earlier below things are in place to get this job done, please let us know if we are something 1. Context has been created with Snowflake connection details for the table to validate. Expectation Suite is created, Rules are added to Suite and saved the same. 2. Checkpoint is also added to the context batch_request pointing to the Snowflake Table to be Validated and listing actions information with the "datahub.integrations.great_expectations.action" class and Datahub Local Host connection. 3. Finally, validator and run_checkpoint are called on top of the context object to complete the work. Having done all these, we weren't able to find the expected result in Datahub. We are running this through Python and it is returning "exit Code 0" Kindly let us know if anyone has an idea of where it was going wrong. I have mentioned a few code lines below in the sequence we have executed. In depth, if anything is required please let us know, and I’ll share in detail. Code: Creating Context : data_context_config = DataContextConfig( datasources={ "sqlalchemy_datasource": DatasourceConfig( class_name="Datasource", execution_engine={ "class_name": "SqlAlchemyExecutionEngine", "connection_string": f<snowflake connection> }, data_connectors={ "runtime_data_connector": { "class_name": "RuntimeDataConnector", "batch_identifiers": [ "customerid_key_0", "customerid_key_1"], } } ) }, expectations_store_name="expectations_store", validations_store_name="validations_store", evaluation_parameter_store_name="evaluation_parameter_store", checkpoint_store_name="checkpoint_store", store_backend_defaults=InMemoryStoreBackendDefaults(), ) context = gx.get_context(project_config=data_context_config) context.create_expectation_suite(expectation_suite_name=<suite name>) suite.add_expectation_configurations(rules) context.save_expectation_suite(expectation_suite=suite, expectation_suite_name=group_name) context.add_checkpoint context.get_validator(batch_request=, expectation_suite=suite) context.run_checkpoint

Hi Team, I'm running a MWAA instance and trying to reach out the Datahub but I'm getting the following error: ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997) Does anyone know how to disable this SSL verification?

Hi team . Iam using datahub and need to ingest spark linage into my datahub running in server at 9002 port .Iam using Jupyter and spark 3.1.3 . Its run ok but i dont see any pipeline in my datahub ui.Please help me.This is my SparkSession config :

image.png