Hi All - I have not been able to get any integrations to work, so I am not sure what I am doing wrong. I have the following integration yaml:

source:
  type: data-lake
  config:
    env: "PROD"
    platform: "local-data-lake"
    base_path: "~/.datahub/data_test2.json"
    profiling:
      enabled: true

sink:
  type: console

and am getting the following error:

---- (full traceback above) ----
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 82, in run
    pipeline = Pipeline.create(pipeline_config, dry_run, preview)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/datahub/ingestion/run/pipeline.py", line 175, in create
    return cls(config, dry_run=dry_run, preview_mode=preview_mode)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/datahub/ingestion/run/pipeline.py", line 127, in __init__
    self.source: Source = source_class.create(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/datahub/ingestion/source/data_lake/__init__.py", line 248, in create
    return cls(config, ctx)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/datahub/ingestion/source/data_lake/__init__.py", line 176, in __init__
    self.init_spark()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/datahub/ingestion/source/data_lake/__init__.py", line 242, in init_spark
    self.spark = SparkSession.builder.config(conf=conf).getOrCreate()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pyspark/sql/session.py", line 186, in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pyspark/context.py", line 378, in getOrCreate
    SparkContext(conf=conf or SparkConf())
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pyspark/context.py", line 133, in __init__
    SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pyspark/context.py", line 327, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pyspark/java_gateway.py", line 105, in launch_gateway
    raise Exception("Java gateway process exited before sending its port number")

Exception: Java gateway process exited before sending its port number
[2022-05-03 22:15:55,416] INFO     {datahub.entrypoints:161} - DataHub CLI version: 0.8.30.0 at /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/datahub/__init__.py
[2022-05-03 22:15:55,416] INFO     {datahub.entrypoints:164} - Python version: 3.10.0 (v3.10.0:b494f5935c, Oct  4 2021, 14:59:20) [Clang 12.0.5 (clang-1205.0.22.11)] at /Library/Frameworks/Python.framework/Versions/3.10/bin/python3 on macOS-11.6.5-x86_64-i386-64bit
[2022-05-03 22:15:55,416] INFO     {datahub.entrypoints:167} - GMS config {}

Hi @limited-agent-54038, Is the recipe running alright when profiling is not enabled ? I believe the error is due spark not setup correctly. Data lake source uses pyspark for profiling (see compatibility). Do you have spark installed and the necessary configurations set ?

I installed pyspark and verified that it is installed, what configurations do I need to set? - you were right, it does work when I run it with profiling not enabled

do you have java installed ? PySpark requires Java 8 or later with

JAVA_HOME

properly set.