Hi team, hope everyone is safe and sound. I was hoping to seek some guidance on an issue I'm facing generating the elementary dashboard. I decided to keep up with the versions and upgraded the dbt package and CLI from

0.8.2

to

0.11.0

. Since then the process I had built to generate the dashboard isn't working anymore. The error message I receive is in the stack trace below. Unfortunately I don't have a way to get the

elementary.log

file from the cluster. Searching through the previous posts there were mentions of possible issue with

dbt

profile or dbt package and CLI versions not matching. I have tested this in a separate pipeline with new schema tables for elementary data and it still throws the same error. It looks like the internal dbt package that the CLI downloads fails to run the

dbt deps

command. Any thoughts or guidance is greatly appreciated.

Traceback (most recent call last):
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-f08aad5e-23ab-4bfb-97d3-0fbacdd0bbab/lib/python3.10/site-packages/elementary/clients/dbt/dbt_runner.py", line 82, in _run_command
    result = subprocess.run(
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['dbt', 'deps', '--project-dir', '/local_disk0/.ephemeral_nfs/envs/pythonEnv-f08aad5e-23ab-4bfb-97d3-0fbacdd0bbab/lib/python3.10/site-packages/elementary/monitor/dbt_project', '--profiles-dir', '..', '--target', 'au_prod_dbt']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-f08aad5e-23ab-4bfb-97d3-0fbacdd0bbab/bin/edr", line 8, in <module>
    sys.exit(cli())
  File "/databricks/python/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/databricks/python/lib/python3.10/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-f08aad5e-23ab-4bfb-97d3-0fbacdd0bbab/lib/python3.10/site-packages/elementary/cli/cli.py", line 67, in invoke
    return super().invoke(ctx)
  File "/databricks/python/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/databricks/python/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/databricks/python/lib/python3.10/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/databricks/python/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-f08aad5e-23ab-4bfb-97d3-0fbacdd0bbab/lib/python3.10/site-packages/elementary/monitor/cli.py", line 406, in report
    data_monitoring = DataMonitoringReport(
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-f08aad5e-23ab-4bfb-97d3-0fbacdd0bbab/lib/python3.10/site-packages/elementary/monitor/data_monitoring/report/data_monitoring_report.py", line 40, in __init__
    super().__init__(
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-f08aad5e-23ab-4bfb-97d3-0fbacdd0bbab/lib/python3.10/site-packages/elementary/monitor/data_monitoring/data_monitoring.py", line 36, in __init__
    self.internal_dbt_runner = self._init_internal_dbt_runner()
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-f08aad5e-23ab-4bfb-97d3-0fbacdd0bbab/lib/python3.10/site-packages/elementary/monitor/data_monitoring/data_monitoring.py", line 73, in _init_internal_dbt_runner
    internal_dbt_runner = DbtRunner(
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-f08aad5e-23ab-4bfb-97d3-0fbacdd0bbab/lib/python3.10/site-packages/elementary/clients/dbt/dbt_runner.py", line 42, in __init__
    self._run_deps_if_needed()
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-f08aad5e-23ab-4bfb-97d3-0fbacdd0bbab/lib/python3.10/site-packages/elementary/clients/dbt/dbt_runner.py", line 291, in _run_deps_if_needed
    self.deps(quiet=True)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-f08aad5e-23ab-4bfb-97d3-0fbacdd0bbab/lib/python3.10/site-packages/elementary/clients/dbt/dbt_runner.py", line 110, in deps
    success, _ = self._run_command(command_args=["deps"], quiet=quiet)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-f08aad5e-23ab-4bfb-97d3-0fbacdd0bbab/lib/python3.10/site-packages/elementary/clients/dbt/dbt_runner.py", line 93, in _run_command
    raise DbtCommandError(err, command_args, logs=logs)
elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.

Hi @Sham Sreedharan, Thank you for the kind words 🙏 Can you try to change the package number to 0.11.1 and run

dbt deps

and

dbt run -s elementary

?

Thank you for the suggestion @ella katz. I upgraded the dbt package to

0.11.1

but it seems the latest version of the CLI is still

0.11.0

. I'm installing via PIP. I can see the latest version in the elementary schema in the data warehouse as

0.11.1

in the

dbt_invocations

table. But I continue to face the same issue. I'll to extract the edr log file from the cluster to find further information. I appreciate your help.

Hi @Sham Sreedharan, Thanks for updating! Glad it works now 🙂