Hello everyone, I'm currently evaluating DataHub ...
# troubleshoot
l
Hello everyone, I'm currently evaluating DataHub for adoption (particularly the delta-lake source), but running into issues with the pyspark versioning. I'm currently running pyspark v3.3.2, which appears to be incompatible with DataHub because the version is pinned to 3.0.3. This PR would solve our issue: #6852 Bump pyspark dependency to =3.1.3> but this was reverted later here, and I couldn't dig up logs or a thread here for a reason: https://github.com/datahub-project/datahub/pull/6954 Is there any way we can get this reopened/the version loosened? Thanks!
📖 1
🔍 1
✅ 1
l
Hey there 👋 I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request. Let's double check a few things first: ✅ There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? ✅ button ✅ It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues? ✅ button Did you find a solution to your issue? ❌ Sorry you weren't able to find a solution. I'm sending you some tips on info you can provide to help the community troubleshoot. Whenever you feel your issue is solved, please react ✅ to your original message to let us know!
l
For context, I'm running datahub v0.10.2.2, and specifically looking at using a programmatic pipeline (per the example here) to push events to datahub running in Docker. However, when I try to add acryl-datahub[delta-lake] to a poetry project, I get a "version solving failed" error
b
@dazzling-judge-80093 @gray-shoe-75895 We need to evaluate this PR to unblock these folks 🙂 https://github.com/datahub-project/datahub/issues/6852
a
l
Ah, that is frustrating - I'll keep an eye out on the pydeequ issue then! Thank you both!
d
@rich-restaurant-61261 thanks for reporting, the fix is on the way and will be available with the next release.
b
@rich-restaurant-61261 I am getting the same error in quickstart docker setup when ingesting from s3 datalake. It was working ok but I am getting this error recently.I had to add the mentioned SPARK_VERSION value in the actions service in compose file to resolve this.
r
Hi @brave-engine-32813, what is the action service in compose file? can you explain a little bit more?
b
I setup datahub in my local system via quickstart guide. Basically,a docker compose file is run to bring up all services. I added the environment variable in the action framework section of docker compose file. I am not sure how you deployed datahub but having that variable set in env worked as workaround.
h
My setup is the same, and I tried the same way. Once you have it, I assume the file is in .datahub/quickstart/docker-compose.yml. After this change, how do you start the datahub? Do you use datahub docker quickstart or docker-compose up?
b
@happy-gigabyte-50024 docker compose up