hey hey, just wondering if anyone's managed to get...
# community-support
g
hey hey, just wondering if anyone's managed to get an edr command running in Databricks dbt tasks? specifically
upload-source-freshness
, even with
--project-dir
it still throws a
Could not find profile named 'elementary'
error which seems unrelated (and definitely in my
profiles.yml
+ working locally with the same setup) suppling the target to the command also makes no odds it seems
Copy code
+ edr run-operation upload-source-freshness --project-dir .
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

Any feedback and suggestions are welcomed! join our community here - <https://bit.ly/slack-elementary>

2024-07-02 10:16:00 — INFO — Running with edr=0.15.1
2024-07-02 10:16:00 — INFO — Installing packages for edr internal dbt package...
2024-07-02 10:16:00 — INFO — Running dbt --log-format json deps --project-dir /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/elementary/monitor/dbt_project
2024-07-02 10:16:04 — INFO — Running with dbt=1.8.3
2024-07-02 10:16:04 — INFO — Updating lock file in file path: /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/elementary/monitor/dbt_project/package-lock.yml
2024-07-02 10:16:04 — INFO — Installing dbt-labs/dbt_utils
2024-07-02 10:16:04 — INFO — Installed from version 0.8.6
2024-07-02 10:16:04 — INFO — Updated version available: 1.2.0
2024-07-02 10:16:04 — INFO — Installing elementary-data/elementary
2024-07-02 10:16:04 — INFO — Installed from version 0.15.1
2024-07-02 10:16:04 — INFO — Updated version available: 0.15.2
2024-07-02 10:16:04 — INFO — 
2024-07-02 10:16:04 — INFO — Updates available for packages: ['dbt-labs/dbt_utils', 'elementary-data/elementary']                 
Update your versions in packages.yml, then run dbt deps
Traceback (most recent call last):
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/elementary/clients/dbt/dbt_runner.py", line 88, in _run_command
    result = subprocess.run(
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['dbt', '--log-format', 'json', 'run-operation', 'elementary.log_macro_results', '--args', '{"macro_name": "elementary_cli.can_upload_source_freshness", "macro_args": {"invocation_id": "bf1b3e98-3cf6-4f3e-9c13-6089ada010db"}}', '--project-dir', '/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/elementary/monitor/dbt_project']' returned non-zero exit status 2.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/edr", line 8, in <module>
    sys.exit(cli())
  File "/databricks/python/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/databricks/python/lib/python3.10/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/elementary/cli/cli.py", line 67, in invoke
    return super().invoke(ctx)
  File "/databricks/python/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/databricks/python/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/databricks/python/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/databricks/python/lib/python3.10/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/databricks/python/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/elementary/operations/cli.py", line 63, in upload_source_freshness
    UploadSourceFreshnessOperation(config).run(rows_per_insert)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/elementary/operations/upload_source_freshness.py", line 26, in run
    self.upload_results(results, metadata, rows_per_insert)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/elementary/operations/upload_source_freshness.py", line 50, in upload_results
    response = dbt_runner.run_operation(
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/elementary/clients/dbt/dbt_runner.py", line 160, in run_operation
    success, command_output = self._run_command(
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/elementary/clients/dbt/dbt_runner.py", line 99, in _run_command
    raise DbtCommandError(err, command_args, logs=logs)
elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.
Encountered an error:
Runtime Error
  Could not find profile named 'elementary'