Hi everyone i m trying to setup a elementary alerts send rep Elementary Community #community-support

Hi everyone, i'm trying to setup a elementary ale...

Ondra Paul

02/20/2025, 7:45 AM

Hi everyone, i'm trying to setup a elementary alerts/send-reports from inside Databricks Workflows. As per others here, I created a custom python script, which is stored inside Github repo with my dbt project. When I try to run this python file from databrick (using

spark_python_task: python_file: elementary_runner.py source: GIT

) - with this command

Copy code

ommand '['edr', 'monitor', '--slack-token', '[REDACTED]', '--slack-channel-name', 'temp-testing', '--profiles-dir', '.', '--target-path', '/tmp/elementary', '--config-dir', '/tmp/elementary']

I get an error that Elementary is trying to create a folder inside the cloned repo (error below), which it probably can't. What is the correct set up of Elementary commands / settings to successfully run inside the Databricks Workflows?🙏 I tried running the script locally and everything executes as expected, so the issue is the implementation with Databricks Workflows.

elementary version - 0.16.1 or 0.16.2 tried both

dbt verion - 1.9.1

databricks - using a ephemeral job cluster

➕ 1

Ondra Paul

02/20/2025, 7:46 AM

Copy code

025-02-20 07:36:50,567 - INFO - Executing command with timeout 600s
2025-02-20 07:36:53,649 - ERROR - Command failed with error:
Traceback (most recent call last):
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/edr", line 5, in <module>
    from elementary.cli.cli import cli
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/elementary/cli/cli.py", line 17, in <module>
    elementary.cli.upgrade.recommend_version_upgrade()
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/elementary/cli/upgrade.py", line 11, in recommend_version_upgrade
    config = Config()
             ^^^^^^^^
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/elementary/config/config.py", line 96, in __init__
    os.makedirs(os.path.abspath(self.target_dir), exist_ok=True)
  File "<frozen os>", line 225, in makedirs
OSError: [Errno 22] Invalid argument: '/Workspace/Repos/.internal/6ea504e3ca_commits/cae639aa8cd160b3fe7097af7ea9ba0dbf143097/edr_target'

Noya Offer

02/20/2025, 3:12 PM

Hey! See this thread, I believe this solution can help 🙂

Ondra Paul

02/21/2025, 11:12 AM

Yeah, I saw it, but even if I tried to set the

—target-path /tmp

it failed on the

Config()

class as shown in the logs and it agained tried to create a edr_target folder in a wrong location…

Ondra Paul

02/21/2025, 8:40 PM

@Noya Offer ok found the issue and how to fix it. The class

Config

from

elementary-data

is called even before running edr monitor/report etc. commands, so setting the

--target-path

in those command is too late. Elementary already will try to create the

edr_target

folder, because it is checking the potential available upgrades. And since no path is provided, it defaults to `.edr/edr_target .`So it is a bug or not?🤔. So basically the workaround is to switch the current working directory to one which is writeable, before running any edr command. Also, it is useful to keep the original directory (in my case I have

profiles.yml

stored there). With this fix, the py script is executing nicely.

37 Views

Open in Slack

Previous Next