I could use some help troubleshooting the following Airflow task error:

elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.
Runtime Error
  Could not find profile named 'elementary'

I’m running dbt commands within a Docker container that executes as an ECS task on AWS Fargate (triggered from Airflow). Inside my Dockerfile, I’ve set the environment variable:

ENV DBT_PROFILES_DIR=/app/.dbt

My

profiles.yml

is valid and includes a proper

elementary

profile:

elementary:
  outputs:
    default:
      type: redshift
      host: "{{ env_var('DB_HOST') }}"
      user: "{{ env_var('DB_USER') }}"
      password: "{{ env_var('DB_PASSWORD') }}"
      port: "{{ env_var('DB_PORT') | as_number }}"
      dbname: my_db
      schema: elementary
      threads: 4
      sslmode: verify-full
      ra3_node: true

These environment variables are confirmed to be set and available in the container before

edr monitor

runs.

A message from kapa.ai

Given that

DBT_PROFILES_DIR

is already set in the container, do I still need to explicitly pass a

--profiles-dir

flag (for example,

--profiles-dir /app

) to

edr monitor

for it to locate the

elementary

profile inside ECS?

Your error message:

elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command. Runtime Error   Could not find profile named 'elementary'

is a common issue when the Elementary CLI (edr) cannot locate the

elementary

profile in your

profiles.yml

. Even if your

profiles.yml

is valid and the environment variable

DBT_PROFILES_DIR

is set, there are a few common pitfalls to check: 1. Location of `profiles.yml`: The path set in

DBT_PROFILES_DIR

should be the directory containing

profiles.yml

, not the file itself. For example, if your file is at

/app/.dbt/profiles.yml

, then

DBT_PROFILES_DIR

should be

/app/.dbt

(which you have set correctly). Double-check that the file is present at runtime and readable by the user running the command inside the container. This is a frequent source of issues, especially in Docker/ECS environments where file mounting or build context can be tricky [see here](https://slack.com/archives/C02CTC89LAX/p1688066687146099). 2. Profile Name: The profile name in your

profiles.yml

must be exactly

elementary

(case-sensitive), and the CLI will look for this key. Your example matches this requirement. 3. Permissions: Ensure that the user running the Airflow/ECS task has permission to read

/app/.dbt/profiles.yml

. Permission issues can sometimes manifest as "profile not found" errors. 4. Profile Target: If you have multiple outputs under

elementary

, make sure you specify the correct target with

--target

if it's not

default

. Otherwise, dbt will use

default

as the target [see here](https://slack.com/archives/C02CTC89LAX/p1680035534817339). 5. Environment Variables in Docker/ECS: Sometimes, environment variables (like

DB_HOST

,

DB_USER

, etc.) are not available to the process running dbt/edr if they're not set correctly in the ECS task definition or Dockerfile. Double-check that these are available at runtime. 6. Command Context: If you are running

edr

from a directory different from where your

profiles.yml

is, try explicitly passing the profiles directory:

edr monitor --profiles-dir /app/.dbt

This has resolved similar issues for other users [see here](https://slack.com/archives/C02CTC89LAX/p1688066687146099). 7. File Content Validation: Make sure there are no YAML formatting issues or hidden characters in your

profiles.yml

. Even a small typo can cause dbt to not recognize the profile. Summary of what to try: - Confirm

/app/.dbt/profiles.yml

exists and is readable inside the running container. - Run

ls -l /app/.dbt

and

cat /app/.dbt/profiles.yml

inside the container to verify. - Try running

edr monitor --profiles-dir /app/.dbt

explicitly. - Double-check that the

elementary

key is at the top level of your YAML and is not indented or nested incorrectly. - Ensure all required environment variables are available to the process. If all of the above are correct and the error persists, there may be an issue with how the container is built or how the ECS task is mounting or copying files. This is a common scenario in containerized and orchestrated environments [see here](https://slack.com/archives/C02CTC89LAX/p1688066687146099), [and here](https://slack.com/archives/C02CTC89LAX/p1686013979045139). If you need further help, sharing the output of

ls -l /app/.dbt

and the exact command you are running inside the container can help pinpoint the issue.

@kapa.ai Within dbt, the parent directory for

profiles.yml

is determined using the following precedence: 1.

--profiles-dir

option 2.

DBT_PROFILES_DIR

environment variable 3. current working directory 4.

~/.dbt/

directory As I set the environmental variable

DBT_PROFILES_DIR

(precedence item #2), shouldn't EDR commands resolve the location as

profiles.yml

as

/app/.dbt

correctly? Or would I still need to explicitly pass this?

edr monitor --profiles-dir /app/.dbt

A message from kapa.ai

@kapa.ai looking at Elementary documentation for the

--profiles-dir

option: • Profile directory path: The path of the directory in which your

profiles.yml

file is located. Default is

HOME_DIR/.dbt

. • This option must be configured together with profile name. • -p, --profiles-dir <PATH> Given the env var I have set above, how could I pass in that path using

--profiles-dir

with the

edr monitor

command?

A message from kapa.ai

@kapa.ai Could I specify a path using Elementary's env var setup?

You can define options either via the CLI or using environment variables in the following format: EDR_<COMMAND>_<OPTION>=<VALUE>.

The following example for

edr report

is given; I would like to apply something similar to `edr monitor`:

export EDR_REPORT_FILE_PATH="my_report.html"
edr report
# Equivalent to: edr report --file-path "my_report.html"

A message from kapa.ai