I'm trying to set up elementary OSS on Airflow wit...
# ask-elementary-ai
c
I'm trying to set up elementary OSS on Airflow with dbt cloud. Having trouble creating the profile with
dbt run-operation elementary.generate_elementary_cli_profile
- Runtime Error Could not find profile named 'default' AFAIK dbt cloud does not use the typical profile.yml. Is there any way to authenticate with not the service account JSON file? In prod airflow, the Airflow container does not have access to the dbt cloud service account's key. Whatsmore, I've tried to debug it and it seems it always tries to run dbt cmd when doing
edr report
so it's doesn't even matter if I already got my artifacts from a previous run EDR is strictly tighted to dbt, right? Meaning, in order to work properly any dbt cmd should also work in that orchestrator where is to be run @kapa.ai
k
A message from kapa.ai
You're correct that dbt Cloud does not use a local
profiles.yml
file in the same way as dbt Core, and this can cause confusion when setting up Elementary OSS in orchestrators like Airflow. Key points from the knowledge sources: - Elementary CLI (edr) requires a
profiles.yml
file
to connect to your data warehouse. This is true even if your dbt project runs in dbt Cloud. The CLI does not use dbt Cloud's internal authentication; it expects a standard dbt profile on the machine/container where it runs, with credentials to the DWH where the elementary models are written. This is a hard requirement for the CLI to function, as it needs to run dbt commands under the hood to read from the warehouse and generate reports or alerts [[CLI install & configure](https://docs.elementary-data.com/oss/cli-install)]. - Generating the profile: The recommended way is to run
dbt run-operation elementary.generate_elementary_cli_profile
in dbt Cloud, then copy the output from the job logs and paste it into a
profiles.yml
file on the machine where you run
edr
. You must fill in any missing fields, especially credentials. If you don't have access to a service account key file (e.g., for BigQuery), you must use another supported authentication method that works non-interactively (e.g., user/password for Snowflake, or environment variables for Redshift) [[dbt integration](https://docs.elementary-data.com/oss/integrations/dbt)]. - Elementary is tightly coupled to dbt: Yes,
edr
runs dbt commands internally. This means that the orchestrator (e.g., Airflow) must be able to run dbt commands successfully using the provided profile. If dbt cannot connect to your warehouse from Airflow, neither can Elementary. The CLI does not work by simply reading artifacts from a previous run; it needs to connect to the warehouse to read the tables created by the elementary dbt package [[Slack thread](https://slack.com/archives/C02CTC89LAX/p1686334697024729)]. - No alternative to service account key for BigQuery: The knowledge sources do not mention a way to authenticate to BigQuery for Elementary without a service account key file, except for using
keyfile_json
in the profile (which still requires the key content, just as a JSON string rather than a file) [[Slack thread](https://slack.com/archives/C02CTC89LAX/p1684317516894799)]. If you cannot provide credentials in the profile, Elementary CLI will not be able to connect. - Best practice for Airflow: Many users inject credentials into the Airflow container using environment variables and reference them in the
profiles.yml
file. The
profiles.yml
can be mounted or generated at runtime, but it must be present and valid for the CLI to work [[Slack thread](https://slack.com/archives/C02CTC89LAX/p1665489623025739)]. Summary: - You must provide a valid
profiles.yml
with credentials on the Airflow machine/container. - Elementary CLI cannot use dbt Cloud's internal authentication; it needs direct DWH credentials. -
edr
is tightly coupled to dbt and requires dbt commands to work in the orchestrator environment. - There is no supported way (per the knowledge sources) to run Elementary CLI without providing DWH credentials in the profile. If you cannot provide a service account key or equivalent credentials in your Airflow environment, you will not be able to run Elementary OSS CLI there. If this is a blocker, you might want to consider Elementary Cloud, which is designed to integrate directly with dbt Cloud and may better fit your constraints [[Slack thread](https://slack.com/archives/C02CTC89LAX/p1686334697024729)]. Let me know if you need more details on how to structure the
profiles.yml
or inject credentials securely!