Hello, I am facing some strange issue when running the ingestion script within the virtualBox Ubuntu guest system. After executing the ingest command I end up with exception thrown stating that the ENV variables for spark an java are undefined:

[2022-02-18 14:30:10,541] ERROR    {logger:26} - Please set env variable SPARK_VERSION
JAVA_HOME is not set
[2022-02-18 14:30:10,896] ERROR    {datahub.entrypoints:119} - File "/usr/local/lib/python3.8/site-packages/datahub/cli/ingest_cli.py", line 77, in run
    67   def run(config: str, dry_run: bool, preview: bool, strict_warnings: bool) -> None:
 (...)
    73       pipeline_config = load_config_file(config_file)
    74   
    75       try:
    76           logger.debug(f"Using config: {pipeline_config}")
--> 77           pipeline = Pipeline.create(pipeline_config, dry_run, preview)
    78       except ValidationError as e:
[...]

Because of the exception I am unable to use ingestion process properly for my own metadata setup. Could you help resovle this issue? 1. In order to reproduce please use following yml (example.yml) file:

source:
  type: data-lake
  config:
    env: "PROD"
    platform: "dataLake"
    base_path: "load"
    profiling:
      enabled: true
sink:
  type: "datahub-rest"
  config:
    server: "<http://localhost:8080>"

2. Then simply run command within the metadata-ingestion folder as:

./metadata-ingestion/scripts/datahub_docker.sh ingest -c example.yml

If it will be required I can also paste the whole error log from my faulting operation.

And do you have the SPARK_VERSION set?

I had installed spark following this instruction: https://www.tecmint.com/install-apache-spark-on-ubuntu/ after executing

spark-submit --version

I see that command ran with success. Where this SPARK_VERSION should be set and to what value?