Hey all - I'm looking at deploying Elementary at the moment and running into some issues with our multi GCP project architecture. I want to store our Elementary dataset in

project_a

, but we sometimes run DBT from

project_b

. I've added this to

dbt_project.yml

, which I thought should cause Elementary to always look in

project_a.elementary

when writing data from a `dbt run`:

models:
  elementary:
    +schema: "elementary"
    +project: "{{ env_var('PROJECT_A') }}"

However, this only works when the target in

profiles.yml

is also set to

PROJECT_A

. This is a problem for our implementation of DBT because we run queries from different GCP projects. Our

profiles.yml

looks like this:

my_project:
  target: prd
  outputs:
    prd:
      type: bigquery
      method: oauth
      project: "{{ env_var('GCP_RUNTIME_PROJECT') }}"
      dataset: dbt

If

GCP_RUNTIME_PROJECT

is set to

PROJECT_A

, elementary runs fine. When it's set to

PROJECT_B

I get the error:

Couldn't find Elementary's models in `PROJECT_B.dbt`. Please run `dbt run -s elementary --target dev`.

The service account used in

PROJECT_B

has permissions to write to

PROJECT_A.elementary

, so I was expecting it to work. But it seems to ignore the configuration set in

dbt_project

to always write to

PROJECT_A.elementary

. The dataset is being set to our default as well (

dbt

), rather than the configured dataset (

elementary

). Is this by design or a bug? I was hoping it would be possible to use Elementary when the target is changed in

profiles.yml

? If it's by design, any suggestions on how to work around it? I was thinking potentially having an elementary schema in each project, but that means having multiple reports/monitors setup, which will be trickier to manage. Thanks in advance

Hey @Will! Elementary currently does not support multi project, there’s also an open feature request for it Do you want elementary to ignore project_b or run project_b with the schema of elementary that is located in project_a? I wonder if the solution here helps your case as well or not 🙂 Can you let me know?

Thanks Noa. We don't have multiple dbt projects, but our single dbt project does write to multiple GCP projects (aka database). I think the example you linked to covers that scenario and means it's supported? I suspect I'm running into permissions issues writing between the projects - as we have a few restrictions in place around that

Hey @Will So you’re saying you implemented the solution as in the example I sent you and it still doesn’t work? I would expect to see errors if it was a permissions issue, do you get any errors?

It doesn't error, but I do get the warning:

Couldn't find Elementary's models in `PROJECT_B.dbt`. Please run `dbt run -s elementary --target stg`.

where DBT is our default output schema. I've tracked the error down to this line returning

None

. It seems to fallback to a default value of

target_database.default_schema

- the problem isn't in elementary though, figured out our permissions setup is preventing Elementary from seeing the

PROJECT_A.elementary

dataset

Hi @Will, so this is a permissions setup issue on your end? did you manage to resolve it?

Hey Ofek, yes it turned out to be the service account permissions on our end, I think if DBT cannot see the table at runtime it prints the above error after falling back to the default dataset

Awesome! great to hear it drop stoked