:metadata-ingestion:lint

mypy

metadata-ingestion

mypy src/ tests/ examples/

df = spark.read.format("csv").option("header", "true").load("dbfs:/FileStore/shared_uploads/nishchay.agarwal@meesho.com/services_classification.csv")
df.write.mode("overwrite").saveAsTable("new_p")

While I am running this command via Databricks Cluster, pipeline is created successfully as per name given in cluster spark conf spark.datahub.databricks.cluster shell_dbx, but 
while I am running delta table command, I am getting error :
22/09/09 09:06:56 ERROR DatasetExtractor: class org.apache.spark.sql.catalyst.plans.logical.Project is not supported yet. Please contact datahub team for further support. 

Also, I am not able to see schema of dataset that I build using spark-lineage, also both upstream & downstream table is showing same as per screenshot (that's not expected)
Also, can you help me, how to enable Delta catalog support from databricks, because its not working on Databricks

pub struct DeltaTableLoadOptions {
	..............
    /// Indicates whether DeltaTable should track files.
    /// This defaults to `true`
    ///
    /// Some append-only applications might have no need of tracking any files.
    /// Hence, DeltaTable will be loaded with significant memory reduction.
    pub require_files: bool,
}

process_commit

pipeline.py

McpEmitter: REST Emitter Configuration

22/09/12 17:54:35 ERROR DatahubSparkListener: Application end event received, but start event missing for appId local-1662980072825

~~~~ Execution Summary ~~~~

RUN_INGEST - {'errors': [],
 'exec_id': '7f529d57-21f5-4d39-a8e8-2b92580692ab',
 'infos': ['2022-09-12 10:22:14.801662 [exec_id=7f529d57-21f5-4d39-a8e8-2b92580692ab] INFO: Starting execution for task with name=RUN_INGEST',
           '2022-09-12 10:22:14.855554 [exec_id=7f529d57-21f5-4d39-a8e8-2b92580692ab] INFO: Caught exception EXECUTING '
           'task_id=7f529d57-21f5-4d39-a8e8-2b92580692ab, name=RUN_INGEST, stacktrace=Traceback (most recent call last):\n'
           '  File "/usr/local/lib/python3.9/site-packages/acryl/executor/execution/default_executor.py", line 121, in execute_task\n'
           '    self.event_loop.run_until_complete(task_future)\n'
           '  File "/usr/local/lib/python3.9/site-packages/nest_asyncio.py", line 89, in run_until_complete\n'
           '    return f.result()\n'
           '  File "/usr/local/lib/python3.9/asyncio/futures.py", line 201, in result\n'
           '    raise self._exception\n'
           '  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 256, in __step\n'
           '    result = coro.send(None)\n'
           '  File "/usr/local/lib/python3.9/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 71, in execute\n'
           '    validated_args = SubProcessIngestionTaskArgs.parse_obj(args)\n'
           '  File "pydantic/main.py", line 521, in pydantic.main.BaseModel.parse_obj\n'
           '  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__\n'
           'pydantic.error_wrappers.ValidationError: 1 validation error for SubProcessIngestionTaskArgs\n'
           'debug_mode\n'
           '  extra fields not permitted (type=value_error.extra)\n']}
Execution finished with errors.

(pymysql.err.OperationalError) (3159, 'Connections using insecure transport are prohibited while --require_secure_transport=ON.')
(Background on this error at: <http://sqlalche.me/e/13/e3q8>) due to 
		'(3159, 'Connections using insecure transport are prohibited while --require_secure_transport=ON.')'.

source:
  type: "dbt"
  config:
    # Coordinates
    # To use this as-is, set the environment variable DBT_PROJECT_ROOT to the root folder of your dbt project
    manifest_path: "${DBT_PROJECT_ROOT}/target/manifest.json"
    catalog_path: "${DBT_PROJECT_ROOT}/target/catalog.json"
    test_results_path: "${DBT_PROJECT_ROOT}/target/run_results.json" # optional for recording dbt test results after running dbt test

    # Options
    target_platform: "redshift" # e.g. bigquery/postgres/etc.
sink:
  type: "datahub-rest"
  config:
    server: "<http://localhost:8080>"

"retries exceeded with url: /nifi-api/access/token (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] "

source:
    type: kafka
    config:
        platform_instance: <platform_instance>
        connection:
            consumer_config:
                security.protocol: SASL_PLAINTEXT
                sasl.username: <user>
                sasl.mechanism: PLAIN
                sasl.password: <password>
            bootstrap: 'broker1:9092'
            schema_registry_url: '<http://schema-registry:8081>'

File "/usr/local/lib/python3.9/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 98, in _read_output_lines\n'
           '    line_bytes = await ingest_process.stdout.readline()\n'
           '  File "/usr/local/lib/python3.9/asyncio/streams.py", line 549, in readline\n'
           '    raise ValueError(e.args[0])\n'
           'ValueError: Separator is not found, and chunk exceed the limit\n']}