Hi everyone, I'm trying to integrate dbt elementary with Snowflake in my project. Currently i'm using

dbt-snowflake==1.6.2

The

python --version

is 3.9.0 After setting up the profile.yml and dbt_project.yml, running

dbt run -s elementary

, and running

dbt test

everything works fine in the database. But when I run the

edr report

it fails. In the packges.yml I used the following:

packages:
  - package: dbt-labs/codegen
    version: 0.10.0
  - package: dbt-labs/dbt_utils
    version: 1.1.1
  - package: elementary-data/elementary
    version: 0.15.1
    ## Docs: <https://docs.elementary-data.com>

And when I install the CLI, I used

elementary-data==0.15.1

Do I need to install another package of elementary or a different CLI? Does someone know which could be the root cause of failing the edr report command, when the elementary tables are correctly integrated in Snowflake database. Here I attach the failure:

edr report
    ________                          __
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ /
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /
                                                 /____/

Any feedback and suggestions are welcomed! join our community here - <https://bit.ly/slack-elementary>

2024-05-13 16:33:32 — INFO — Running with edr=0.15.1
2024-05-13 16:33:32 — INFO — Installing packages for edr internal dbt package...
2024-05-13 16:33:32 — INFO — Running dbt --log-format json deps --project-dir C:\Users\xxxxxxxxxxxxx\Documentos\ELEMENTARY_6\xxxxxx\dbt_scripts\venv_elementary\Lib\site-packages\elementary\monitor\dbt_project
2024-05-13 16:33:36 — INFO — Running with dbt=1.6.9
2024-05-13 16:33:36 — INFO — Installing dbt-labs/dbt_utils
2024-05-13 16:33:36 — INFO — Encountered an error:
[Errno 2] No such file or directory: 'dbt_packages\\dbt-utils-0.8.6\\integration_tests\\data\\schema_tests\\data_test_mutually_exclusive_ranges_no_gaps.csv'
2024-05-13 16:33:36 — INFO — Traceback (most recent call last):
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\dbt\cli\requires.py", line 87, in wrapper
    result, success = func(*args, **kwargs)
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\dbt\cli\requires.py", line 72, in wrapper
    return func(*args, **kwargs)
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\dbt\cli\requires.py", line 126, in wrapper
    return func(*args, **kwargs)
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\dbt\cli\requires.py", line 172, in wrapper
    return func(*args, **kwargs)
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\dbt\cli\main.py", line 461, in deps
    results = task.run()
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\dbt\task\deps.py", line 81, in run
    package.install(self.project, renderer)
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\dbt\deps\registry.py", line 60, in install
    self._install(project, renderer)
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\dbt\deps\base.py", line 117, in _install
    connection_exception_retry(download_untar_fn, 5)
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\dbt\utils.py", line 601, in _connection_exception_retry
    return fn()
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\dbt\deps\base.py", line 129, in download_and_untar
    system.untar_package(tar_path, deps_path, package_name)
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\dbt\clients\system.py", line 500, in untar_package
    tarball.extractall(dest_dir)
  File "C:\Users\xxxxxx\AppData\Local\Programs\Python\Python39\lib\tarfile.py", line 2034, in extractall
    self.extract(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "C:\Users\xxxxxx\AppData\Local\Programs\Python\Python39\lib\tarfile.py", line 2075, in extract
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "C:\Users\xxxxxx\AppData\Local\Programs\Python\Python39\lib\tarfile.py", line 2147, in _extract_member
    self.makefile(tarinfo, targetpath)
  File "C:\Users\xxxxxx\AppData\Local\Programs\Python\Python39\lib\tarfile.py", line 2188, in makefile
    with bltn_open(targetpath, "wb") as target:
FileNotFoundError: [Errno 2] No such file or directory: 'dbt_packages\\dbt-utils-0.8.6\\integration_tests\\data\\schema_tests\\data_test_mutually_exclusive_ranges_no_gaps.csv'

Traceback (most recent call last):
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\elementary\clients\dbt\dbt_runner.py", line 88, in _run_command
    result = subprocess.run(
  File "C:\Users\xxxxxx\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['dbt', '--log-format', 'json', 'deps', '--project-dir', 'C:\\Users\\xxxxxx\\xxxxxx\\Documentos\\ELEMENTARY_6\\xxxxxx\\dbt_scripts\\venv_elementary\\Lib\\site-packages\\elementary\\monitor\\dbt_project']' returned non-zero exit status 2.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\xxxxxx\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\xxxxxx\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\xxxxxxxxxxxxx\Documentos\ELEMENTARY_6\xxxxxx\dbt_scripts\venv_elementary\Scripts\edr.exe\__main__.py", line 7, in <module>
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\elementary\cli\cli.py", line 67, in invoke
    return super().invoke(ctx)
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\elementary\monitor\cli.py", line 442, in report
    data_monitoring = DataMonitoringReport(
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\elementary\monitor\data_monitoring\report\data_monitoring_report.py", line 42, in __init__
    super().__init__(
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\elementary\monitor\data_monitoring\data_monitoring.py", line 35, in __init__
    self.internal_dbt_runner = self._init_internal_dbt_runner()
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\elementary\monitor\data_monitoring\data_monitoring.py", line 61, in _init_internal_dbt_runner
    internal_dbt_runner = DbtRunner(
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\elementary\clients\dbt\dbt_runner.py", line 48, in __init__
    self._run_deps_if_needed()
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\elementary\clients\dbt\dbt_runner.py", line 318, in _run_deps_if_needed
    self.deps()
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\elementary\clients\dbt\dbt_runner.py", line 116, in deps
    success, _ = self._run_command(
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\elementary\clients\dbt\dbt_runner.py", line 99, in _run_command
    raise DbtCommandError(err, command_args, logs=logs)
elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.
Encountered an error:
[Errno 2] No such file or directory: 'dbt_packages\\dbt-utils-0.8.6\\integration_tests\\data\\schema_tests\\data_test_mutually_exclusive_ranges_no_gaps.csv'
Traceback (most recent call last):
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\dbt\cli\requires.py", line 87, in wrapper
    result, success = func(*args, **kwargs)
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\dbt\cli\requires.py", line 72, in wrapper
    return func(*args, **kwargs)
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\dbt\cli\requires.py", line 126, in wrapper
    return func(*args, **kwargs)
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\dbt\cli\requires.py", line 172, in wrapper
    return func(*args, **kwargs)
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\dbt\cli\main.py", line 461, in deps
    results = task.run()
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\dbt\task\deps.py", line 81, in run
    package.install(self.project, renderer)
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\dbt\deps\registry.py", line 60, in install
    self._install(project, renderer)
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\dbt\deps\base.py", line 117, in _install
    connection_exception_retry(download_untar_fn, 5)
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\dbt\utils.py", line 601, in _connection_exception_retry
    return fn()
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\dbt\deps\base.py", line 129, in download_and_untar
    system.untar_package(tar_path, deps_path, package_name)
  File "c:\users\xxxxxxxxxxxxx\documentos\elementary_6\xxxxxx\dbt_scripts\venv_elementary\lib\site-packages\dbt\clients\system.py", line 500, in untar_package
    tarball.extractall(dest_dir)
  File "C:\Users\xxxxxx\AppData\Local\Programs\Python\Python39\lib\tarfile.py", line 2034, in extractall
    self.extract(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "C:\Users\xxxxxx\AppData\Local\Programs\Python\Python39\lib\tarfile.py", line 2075, in extract
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "C:\Users\xxxxxx\AppData\Local\Programs\Python\Python39\lib\tarfile.py", line 2147, in _extract_member
    self.makefile(tarinfo, targetpath)
  File "C:\Users\xxxxxx\AppData\Local\Programs\Python\Python39\lib\tarfile.py", line 2188, in makefile
    with bltn_open(targetpath, "wb") as target:
FileNotFoundError: [Errno 2] No such file or directory: 'dbt_packages\\dbt-utils-0.8.6\\integration_tests\\data\\schema_tests\\data_test_mutually_exclusive_ranges_no_gaps.csv'

Thank you so much for your help.

No other pypi dependencies are required except dbt, dbt adapter (dbt-snowflake in your case) and elementary-data. First of all make sure you run

edr

command in project's directory. Then make sure you run

dbt deps

before running elementary. When we run

edr report

we provide the following options

--profile-target

and

--target-path

try adding them too. If that doesn't help try another elementary version 0.14.0 or 0.15.0 for example.

Hi Denis, Thanks for your reply. I'm running

edr

command in my project directory, but I set up a python environment inside the dbt project folder, where I downloaded all the packages. But I run the command

edr report --project-dir

(because I think that the root cause of failing is not finding the right directory) and I get the following result:

$ edr report --project-dir 'C:\Users\xxx\xxx\Documentos\ELEMENTARY_6\xxx\dbt_scripts'
    ________                          __
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ /
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /
                                                 /____/

Any feedback and suggestions are welcomed! join our community here - <https://bit.ly/slack-elementary>

2024-05-14 15:31:56 — INFO — Running with edr=0.15.1
2024-05-14 15:31:56 — INFO — Installing packages for edr internal dbt package...
2024-05-14 15:31:56 — INFO — Running dbt --log-format json deps --project-dir C:\Users\xxx\xxx\Documentos\ELEMENTARY_6\xxx\dbt_scripts\venv_elementary\Lib\site-packages\elementary\monitor\dbt_project
2024-05-14 15:32:00 — INFO — Running with dbt=1.6.9
2024-05-14 15:32:00 — INFO — Installing dbt-labs/dbt_utils
2024-05-14 15:32:00 — INFO — Encountered an error:
[Errno 2] No such file or directory: 'dbt_packages\\dbt-utils-0.8.6\\integration_tests\\data\\schema_tests\\data_test_mutually_exclusive_ranges_no_gaps.csv'

But instead of this command while running

edr report

dbt --log-format json deps --project-dir C:\Users\xxx\xxx\Documentos\ELEMENTARY_6\xxx\dbt_scripts\venv_elementary\Lib\site-packages\elementary\monitor\dbt_project

From my point of view I would expect:

dbt --log-format json deps --project-dir C:\Users\xxx\xxx\Documentos\ELEMENTARY_6\xxx\dbt_scripts\

Because inside the folder "dbt_scripts" there is my dbt project. Does that make sense?

Hello Artur

I've edited the above message to put more context!

EDR is very tricky, it does things right in

venv_elementary\Lib\site-packages\elementary\monitor\dbt_project

because EDR CLI itself is implemented as DBT project, very specific DBT project. Now I noticed that you run EDR on Windows, so the error may be related to different PATH issues. Try copying the project to a shorter path location, like

C:\\project_name

and try to run in that new folder.

Hi, Finally the solution was to execute all the elementary steps in a linux environment (settting it up via Docker). And define the profiles.yml in an appropiate way:

elementary:
  outputs:
    elementary:
      account: yoursnowflakeaccount
      database: yourdatabase
      password: yourpwd
      role: yourrole
      schema: elementary
      threads: 10
      type: snowflake
      user: youruser
      warehouse: yourwh

And execute the command:

edr report --project-dir yourdbtdirectory/ --profiles-dir directorywhereprofile.ymlis/ --profile-target elementary

We are having the similar problem on our side.