https://linen.dev logo
Join Slack
Powered by
Hi :slightly_smiling_face: We’re using elementary ...
# community-support
a
Hi 🙂 We’re using elementary in Databricks. We’re aiming to share the observability report and send slack alerts by running 
edr
commands using os.system in a Databricks notebook. I’ll paste the logs in the thread. We are using 
elementary-data
and 
elementary-data[databricks] 0.13.0
and 
dbt-databricks 1.7.2
(have also tried 0.12.1 with dbt 1.6.0). We have installed the libraries using 
%pip
and are using 
os.system
. We think elementary is running it’s internal dbt commands, which are trying and failing to make directories on Databricks. Is this right? Are there any requirements we’re missing in databricks? Is there anything else we can try? Thanks
Copy code
________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

Any feedback and suggestions are welcomed! join our community here - <https://bit.ly/slack-elementary>

2023-12-05 11:44:56 — INFO — Running with edr=0.13.0
2023-12-05 11:44:56 — INFO — Downloading edr internal dbt package
2023-12-05 11:44:56 — INFO — Running dbt --log-format json deps --project-dir /local_disk0/.ephemeral_nfs/envs/pythonEnv-fb6c7159-2a86-45b1-9030-1efe49064ec9/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /Workspace/Repos/.internal/c1227e94c2_commits/bc29762f7c8f5f029a8ccaadec54a0d90a35ffe9/scripts/observability/../..
2023-12-05 11:45:01 — INFO — Running with dbt=1.7.3
2023-12-05 11:45:01 — INFO — Installing dbt-labs/dbt_utils
2023-12-05 11:45:01 — INFO — Installed from version 0.8.6
2023-12-05 11:45:01 — INFO — Updated version available: 1.1.1
2023-12-05 11:45:01 — INFO — Installing elementary-data/elementary
2023-12-05 11:45:01 — INFO — Installed from version 0.13.0
2023-12-05 11:45:01 — INFO — Up to date!
2023-12-05 11:45:01 — INFO — 
2023-12-05 11:45:01 — INFO — Updates available for packages: ['dbt-labs/dbt_utils']                 
Update your versions in packages.yml, then run dbt deps
2023-12-05 11:45:08 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
Encountered an error:
[Errno 22] Invalid argument: 'dbt_packages'
Traceback (most recent call last):
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-fb6c7159-2a86-45b1-9030-1efe49064ec9/lib/python3.10/site-packages/dbt/cli/requires.py", line 90, in wrapper
    result, success = func(*args, **kwargs)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-fb6c7159-2a86-45b1-9030-1efe49064ec9/lib/python3.10/site-packages/dbt/cli/requires.py", line 75, in wrapper
    return func(*args, **kwargs)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-fb6c7159-2a86-45b1-9030-1efe49064ec9/lib/python3.10/site-packages/dbt/cli/requires.py", line 168, in wrapper
    return func(*args, **kwargs)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-fb6c7159-2a86-45b1-9030-1efe49064ec9/lib/python3.10/site-packages/dbt/cli/requires.py", line 197, in wrapper
    return func(*args, **kwargs)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-fb6c7159-2a86-45b1-9030-1efe49064ec9/lib/python3.10/site-packages/dbt/cli/requires.py", line 244, in wrapper
    return func(*args, **kwargs)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-fb6c7159-2a86-45b1-9030-1efe49064ec9/lib/python3.10/site-packages/dbt/cli/requires.py", line 284, in wrapper
    return func(*args, **kwargs)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-fb6c7159-2a86-45b1-9030-1efe49064ec9/lib/python3.10/site-packages/dbt/cli/main.py", line 724, in run_operation
    results = task.run()
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-fb6c7159-2a86-45b1-9030-1efe49064ec9/lib/python3.10/site-packages/dbt/task/run_operation.py", line 51, in run
    self.compile_manifest()
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-fb6c7159-2a86-45b1-9030-1efe49064ec9/lib/python3.10/site-packages/dbt/task/base.py", line 173, in compile_manifest
    self.graph = compiler.compile(self.manifest)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-fb6c7159-2a86-45b1-9030-1efe49064ec9/lib/python3.10/site-packages/dbt/compilation.py", line 453, in compile
    self.initialize()
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-fb6c7159-2a86-45b1-9030-1efe49064ec9/lib/python3.10/site-packages/dbt/compilation.py", line 284, in initialize
    make_directory(self.config.packages_install_path)
  File "/usr/lib/python3.10/functools.py", line 889, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-fb6c7159-2a86-45b1-9030-1efe49064ec9/lib/python3.10/site-packages/dbt/clients/system.py", line 128, in _
    raise e
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-fb6c7159-2a86-45b1-9030-1efe49064ec9/lib/python3.10/site-packages/dbt/clients/system.py", line 122, in _
    os.makedirs(path)
  File "/usr/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 22] Invalid argument: 'dbt_packages'
n
Hey @Alex Smith! You are defiantly right 🙂 Elementary CLI has an internal dbt project (unrelated to your project). This means that the installation of 
edr
has 2 steps: a. 
pip install
of the python package and python dependencies. b. 
dbt deps
executed by 
edr
to install dbt packages (elementary and dbt-utils). You don’t need to actually do the scecond step, 
edr
does it automatically if the packages are missing. So the edr need to have permissions to the directory where it’s running, is that something possible for you? There might be a workaround if not
a
Thanks Noa! I don't think this is possible. What workaround are you suggesting?
n
Can you maybe try adding a step for running 
cd /you_python_location/site-packages/elementary/monitor/dbt_project/ && dbt deps
before the installation?
a
we've tried adding this step and we still have the same error
n
Can you tell me which versions you’re using for the dbt there? Maybe also try upgrading dbt-utils just to make sure everything is on the same version
The reason I’m asking is because it kind of seems like a dbt issue and less because of elementary cli (not sure though)
a
dbt-utiles is 1.1.1 and dbt-databricks is 1.7.2
n
So you just upgraded the dbt-utils? since in the log you send me there’s an upgrade request If so does it still happen? Can you send the new log?
a
Yes, we upgraded the dbt-utils. Here you go 🙂
Copy code
________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

You are using Elementary 0.13.0, however version 0.13.1 is available.
Consider upgrading by running: "/local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/python -m pip install --upgrade elementary-data"

Any feedback and suggestions are welcomed! join our community here - <https://bit.ly/slack-elementary>

2023-12-06 13:41:01 — INFO — Running with edr=0.13.0
2023-12-06 13:41:01 — INFO — Downloading edr internal dbt package
2023-12-06 13:41:01 — INFO — Running dbt --log-format json deps --project-dir /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /Workspace/Repos/.internal/c1227e94c2_commits/3a9cbcc2a1ee2ac2743c5dad927faa516fd374b5
2023-12-06 13:41:08 — INFO — Running with dbt=1.7.3
2023-12-06 13:41:08 — INFO — Installing dbt-labs/dbt_utils
2023-12-06 13:41:08 — INFO — Installed from version 1.1.1
2023-12-06 13:41:08 — INFO — Up to date!
2023-12-06 13:41:08 — INFO — Installing elementary-data/elementary
2023-12-06 13:41:08 — INFO — Installed from version 0.13.0
2023-12-06 13:41:08 — INFO — Up to date!
2023-12-06 13:41:16 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
Encountered an error:
[Errno 22] Invalid argument: 'dbt_packages'
Traceback (most recent call last):
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/cli/requires.py", line 90, in wrapper
    result, success = func(*args, **kwargs)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/cli/requires.py", line 75, in wrapper
    return func(*args, **kwargs)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/cli/requires.py", line 168, in wrapper
    return func(*args, **kwargs)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/cli/requires.py", line 197, in wrapper
    return func(*args, **kwargs)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/cli/requires.py", line 244, in wrapper
    return func(*args, **kwargs)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/cli/requires.py", line 284, in wrapper
    return func(*args, **kwargs)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/cli/main.py", line 724, in run_operation
    results = task.run()
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/task/run_operation.py", line 51, in run
    self.compile_manifest()
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/task/base.py", line 173, in compile_manifest
    self.graph = compiler.compile(self.manifest)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/compilation.py", line 453, in compile
    self.initialize()
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/compilation.py", line 284, in initialize
    make_directory(self.config.packages_install_path)
  File "/usr/lib/python3.10/functools.py", line 889, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/clients/system.py", line 128, in _
    raise e
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/clients/system.py", line 122, in _
    os.makedirs(path)
  File "/usr/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 22] Invalid argument: 'dbt_packages'
Do we need to enable any packages in Databricks in order to run elementary reports successfully? we’ve narrowed the problem down, having tried 
%sh python -c 'import os; os.mkdir("test")'
which fails with the same error. So we agree, this isn’t an elementary or dbt issue, rather something in our databricks. Do you know if this is a unity requirement, or there are some permissions we need to change to allow this?
o
Hi @Alex Smith, definitely looks like an issue with where you run your 
edr
command, 
os.mkdir
is a basic python command that creates a directory, it might be some issue with databricks notebook
a
Thank you, we were never able to come up with a solution to this but managed to get it working with github actions
o
Great! 👍
30 Views
Open in Slack
PreviousNext
Powered by