Gary James
07/02/2024, 10:25 AMupload-source-freshness , even with --project-dir it still throws a Could not find profile named 'elementary' error which seems unrelated (and definitely in my profiles.yml + working locally with the same setup)
suppling the target to the command also makes no odds it seemsGary James
07/02/2024, 10:25 AM+ edr run-operation upload-source-freshness --project-dir .
________ __
/ ____/ /__ ____ ___ ___ ____ / /_____ ________ __
/ __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
/ /___/ / __/ / / / / / __/ / / / /_/ /_/ / / / /_/ /
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/ \__, /
/____/
Any feedback and suggestions are welcomed! join our community here - <https://bit.ly/slack-elementary>
2024-07-02 10:16:00 — INFO — Running with edr=0.15.1
2024-07-02 10:16:00 — INFO — Installing packages for edr internal dbt package...
2024-07-02 10:16:00 — INFO — Running dbt --log-format json deps --project-dir /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/elementary/monitor/dbt_project
2024-07-02 10:16:04 — INFO — Running with dbt=1.8.3
2024-07-02 10:16:04 — INFO — Updating lock file in file path: /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/elementary/monitor/dbt_project/package-lock.yml
2024-07-02 10:16:04 — INFO — Installing dbt-labs/dbt_utils
2024-07-02 10:16:04 — INFO — Installed from version 0.8.6
2024-07-02 10:16:04 — INFO — Updated version available: 1.2.0
2024-07-02 10:16:04 — INFO — Installing elementary-data/elementary
2024-07-02 10:16:04 — INFO — Installed from version 0.15.1
2024-07-02 10:16:04 — INFO — Updated version available: 0.15.2
2024-07-02 10:16:04 — INFO —
2024-07-02 10:16:04 — INFO — Updates available for packages: ['dbt-labs/dbt_utils', 'elementary-data/elementary']
Update your versions in packages.yml, then run dbt deps
Traceback (most recent call last):
File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/elementary/clients/dbt/dbt_runner.py", line 88, in _run_command
result = subprocess.run(
File "/usr/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['dbt', '--log-format', 'json', 'run-operation', 'elementary.log_macro_results', '--args', '{"macro_name": "elementary_cli.can_upload_source_freshness", "macro_args": {"invocation_id": "bf1b3e98-3cf6-4f3e-9c13-6089ada010db"}}', '--project-dir', '/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/elementary/monitor/dbt_project']' returned non-zero exit status 2.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/edr", line 8, in <module>
sys.exit(cli())
File "/databricks/python/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/databricks/python/lib/python3.10/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/elementary/cli/cli.py", line 67, in invoke
return super().invoke(ctx)
File "/databricks/python/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/databricks/python/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/databricks/python/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/databricks/python/lib/python3.10/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/databricks/python/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/elementary/operations/cli.py", line 63, in upload_source_freshness
UploadSourceFreshnessOperation(config).run(rows_per_insert)
File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/elementary/operations/upload_source_freshness.py", line 26, in run
self.upload_results(results, metadata, rows_per_insert)
File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/elementary/operations/upload_source_freshness.py", line 50, in upload_results
response = dbt_runner.run_operation(
File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/elementary/clients/dbt/dbt_runner.py", line 160, in run_operation
success, command_output = self._run_command(
File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/elementary/clients/dbt/dbt_runner.py", line 99, in _run_command
raise DbtCommandError(err, command_args, logs=logs)
elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.
Encountered an error:
Runtime Error
Could not find profile named 'elementary'