Hi Everyone, We have installed pyspark version 3.0.3 and hadoop 3.2 and updated its content in bashrc file in the below path. Still we are getting the below error:- PATH: export SPARK_HOME=/home/ubuntu/spark-3.0.3-bin-hadoop3.2 export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin Also please help, where we have to update "SPARK_VERSION environment variable"

datahub.ingestion.run.pipeline.PipelineInitError: Failed to find a registered source for type s3: SPARK_VERSION environment variable is required. Supported values are: dict_keys(['3.3', '3.2', '3.1', '3.0', '2.4'])

Hi, have you tried this steps in the thread? 🙂 https://datahubspace.slack.com/archives/CV2KB471C/p1689869521641019

Hello @delightful-ram-75848, I have tried the mentioned steps , still getting same error. Can you please suggest any other alternative to fix this issue.

@dazzling-yak-93039 would love your help here!

Hi @able-ghost-38838, I don't see the

SPARK_VERSION

env variable being set in the code you sent, can you ensure that is set? Which of the approaches that Hyejin sent did you try?

Hello @dazzling-yak-93039, I have added the

SPARK_VERSION

env variable as well as added a service for Spark and then it worked. Thanks for your help!

So where we need to setup spark and add env variable? I have setup datahub as pod in my K8s cluster. Facing SPARK VERSION error when trying to connect s3. @dazzling-yak-93039 appreciating ur support here

@red-florist-94889 You don't necessarily need to set up spark if you're not using it, as long as you just set up that env variable then the library should be able to work for what it's being used for!

Where to setup the spark environment variable? I'm using 0.10.4 version in kubernetes cluster

Does this help you? https://datahubspace.slack.com/archives/C029A3M079U/p1689090270693809?thread_ts=1689071272.782439&cid=C029A3M079U

I also want to note that you can upgrade the datahub CLI connector to a newer version to avoid this bug! As Hyejin posted here: https://datahubspace.slack.com/archives/CV2KB471C/p1689896896538989?thread_ts=1689869521.641019&cid=CV2KB471C

But upgrading to 0.10.5 is causing Kafka error.

Can you add it in the env section of the actions component in the docker compose file as mentioned above? Also, I just wanted to clarify, you can update only that component to 0.10.5.2rc2 to avoid this bug. You do not need to upgrade the rest of the components.

Yup that for setup using docker. I'm installing using helm. So I don't have docker file to add environment variable. So you mean to upgrade only the datahub not the dependence S