Im trying to collect source freshness results from dbt and display it in the Elementary Report / Dashboard. As I understand we can do this by following this guide: https://docs.elementary-data.com/cloud/guides/collect-source-freshness#collect-source-freshness-failures We use GCP and Google Cloud Workflows together with cloud build. My workflow for hosting docs and elementary report looks like this today: steps: - name: 'ghcr.io/dbt-labs/dbt-bigquery:1.7.0' id: source-freshness allowExitCodes: [0,1] allowFailure: true args: [ 'source', 'freshness', '--target', 'dev' ] - name: 'ghcr.io/elementary-data/elementary:v0.13.0' id: upload-source-freshness waitFor: - source-freshness args: [ 'run-operation', 'upload-source-freshness', '--project-dir', '/workspace', '--profiles-dir', '.dbt-profiles', '--profile-target', 'dev' ] - name: 'ghcr.io/elementary-data/elementary:v0.13.0' id: generate-elementary-report waitFor: - upload-source-freshness args: [ 'monitor', 'report', '--profiles-dir', '.dbt-profiles', '--profile-target', 'dev', '--file-path', 'index.html', '--env', 'dev' ] The --project dir path isn't working. I've tried: . pwd $PWD /workspace But none seems to work, I get this error in my cloud build logs: Step #2 - "upload-source-freshness": 2024-01-04 143117 — INFO — Running with edr=0.13.0 Step #2 - "upload-source-freshness": Error: Could not find sources.json at /usr/app/target/sources.json. Please run

dbt source freshness

before running this command.

Hi, elementary needs output generated by dbt in a shared file system. That means, that elementary and dbt need to run inside the same docker container. Could this be your issue? To me it looks as if you're workflow is executing multiple different containers which most likely produce output into their isolated file systems.

It still uses the same filesystem. I can see that the target/ folder is populated with the sources.json file. To me it seems that the elementary image is always looking into /usr/app/ which in this workspace isn't correct.. Is there a way of setting a path differently? If going by your approach @Willi, can the elementary image run dbt or do i need to make my own Docker image?

interesting, I understand that in the workflow you're running all docker containers have access to a shared file system and that you expect the container running

edr

to have access to that filesystem. However, when you point

edr

to it by using

--project-dir

, it does not pick it up. Could you share the code for your

check-files

? Is the edr container executing that? I checked how I run

edr

and I haven't used the

--project-dir

yet so I can neither deny or confirm that it works. I've been running

edr

either: • a) inside a custom docker image which also contains dbt • b) in github actions without docker so that both dbt and edr have access to the github workflow environment.

the check-files is simply: - name: 'ubuntu' id: check-files waitFor: - source-freshness args: [ 'ls', 'target' ]

I see, so this proves that the dbt image writes artefacts into the file system accessible to the workflow. However, my thought is that either: a) the elementary image cannot pick up files from outside the image via the

--project-dir

flag b) or, the

--project-dir

flag has a defect Would you agree with that analysis?

Yes! I resolved it , but I think this --project-dir args needs some looking over. My current solution is:

- name: '${var.elementary_image}'
                      id: source-freshness
                      allowExitCodes: [0,1]
                      allowFailure: true
                      entrypoint: '/bin/bash'
                      args:
                        - '-c'
                        - 'dbt source freshness --target dev && edr run-operation upload-source-freshness --project-dir . --profiles-dir .dbt-profiles --profile-target dev'

I see. You could separate the commands with

;

instead of

&&

to have them executed even if the preceding command failed. Does this solve everything now?

It did, thanks! But it's not optimal for error handling. I tried separating them again as two jobs and used: https://cloud.google.com/build/docs/build-config-file-schema#volumes But I got the same errors as in the beginning. @Maayan Salom Do you know who worked on this feature?

Hi @Christopher Ekfeldt, i may have found the cause of your issue:

edr run-operation upload-source-freshness

prefers the env variable

DBT_TARGET_PATH

over the cli argument

--project-dir

, so you can try overwriting it in your step

Sweet! That worked well, Thanks @Ofek Weiss. Although I hope you find the functionality of --project-dir to be lacking.

@Christopher Ekfeldt definitely lacking, I'd expect the flag to overwrite the env var, not the other way around. I'll check with the team to see if there is any reason to keep it that way, but either way it should definitely be documented and explained