Hi everyone, I am trying to troubleshoot this issue for past few weeks and could not figure out what is causing the issue. We have

airflow

+

dbt core

+

elementary.

We use airflow dbt operators from this package :`airflow-dbt-python` to run our dbt -elementary jobs in airflow. Package versions Airflow==2.7.3 airflow-dbt-python==2.0.1 dbt-core==1.7.4 dbt-snowflake==1.7.1 elementary-data==0.13.2 I am facing issue while running dbt docs operator (

dbt docs generate

) and getting this error which is throwing from

macro on_run_end (macros/edr/system/hooks/on_run_end.sql)

. This failure is not happening for all the run, randomly it is throwing error for some runs. Has anyone faced similar issue? Thanks for looking !

[2024-02-09, 16:04:14 PST] {dbt.py:195} ERROR - An error has occurred while executing dbt generate
Traceback (most recent call last):
  File "/usr/local/airflow/.local/lib/python3.8/site-packages/airflow_dbt_python/operators/dbt.py", line 189, in execute
    result = self.dbt_hook.run_dbt_task(
  File "/usr/local/airflow/.local/lib/python3.8/site-packages/airflow_dbt_python/hooks/dbt.py", line 287, in run_dbt_task
    results = task.run()
  File "/usr/local/airflow/.local/lib/python3.8/site-packages/dbt/task/generate.py", line 211, in run
    compile_results = CompileTask.run(self)
  File "/usr/local/airflow/.local/lib/python3.8/site-packages/dbt/task/runnable.py", line 474, in run
    result = self.execute_with_hooks(selected_uids)
  File "/usr/local/airflow/.local/lib/python3.8/site-packages/dbt/task/runnable.py", line 435, in execute_with_hooks
    res = self.execute_nodes()
  File "/usr/local/airflow/.local/lib/python3.8/site-packages/dbt/task/runnable.py", line 359, in execute_nodes
    self.run_queue(pool)
  File "/usr/local/airflow/.local/lib/python3.8/site-packages/dbt/task/runnable.py", line 273, in run_queue
    self._raise_set_error()
  File "/usr/local/airflow/.local/lib/python3.8/site-packages/dbt/task/runnable.py", line 254, in _raise_set_error
    raise self._raise_next_tick
dbt.exceptions.DbtRuntimeError: Runtime Error
  Compilation Error in operation elementary-on-run-end-0 (./dbt_project.yml)
    'None' has no attribute 'get'
    
    > in macro on_run_end (macros/edr/system/hooks/on_run_end.sql)
    > called by operation elementary-on-run-end-0 (./dbt_project.yml)

Hey @Fency Clement thanks for reaching out 🙂 Any chance you can share with me the

edr.log

and

dbt.log

from this run?

@Noa Kurman, Thank you for looking into this issue. We are facing this issue only when running in Airflow deployed in EKS cluster using

DbtDocsGenerateOperator

operator. It is lost when task is completed. I am not sure if I can capture that. Let me see if I can save those using some custom functions.

hmm interesting question Basically you can disable elementary using this:

models:
  elementary:
    +schema: "elementary"
    +enabled: "{{ target.name in ['prod'] }}"

For example here we’re enable elementary only for prod Maybe you can use a similar way to do that on your env, or set a specific var when running dbt docs and then check this var in the condition You could also try setting these vars:

vars:
  disable_run_results: true
  disable_tests_results: true
  disable_dbt_invocation_autoupload: true

before the run which will skip the failing part, if this is the direction your going to

We are looking for disabling elementary only for dbt docs generate commands, but all other commands we want to enable elementary. Looks like approaches you have mentioned might disable elementary entirely

Sorry I might not have been clean enough, my suggestion (which is a bit creative so might not fit you 🙂) is to set these based on an env var, for example create an env var called

should_run_elementary

- then set this var to false just before running your

dbt docs

command and set it back to true afterwards Then you can use this var in the elementary enabled What do you think? In general I would still be interested in why it’s failing but I guess we’ll need the logs for that which you mentioned are unavailable