Hi all, After upgrading from 0.15 to 0.19, I've ran into this issue: when trying to upload freshness results to elementary, it fails with the following error:

________                          __
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ /
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /
                                                 /____/


Any feedback and suggestions are welcomed! join our community here - <https://bit.ly/slack-elementary>

2025-07-04 04:26:40 — INFO — Running with edr=0.19.1
INFO:elementary.cli.cli:Running with edr=0.19.1
2025-07-04 04:26:40 — INFO — Installing packages for edr internal dbt package...
INFO:elementary.clients.dbt.command_line_dbt_runner:Installing packages for edr internal dbt package...
INFO:elementary.clients.dbt.command_line_dbt_runner:Running dbt command --log-format json deps --project-dir /home/runner/work/bi/bi/dbt/.venv/lib/python3.12/site-packages/elementary/monitor/dbt_project --profiles-dir /home/runner/work/bi/bi/dbt/ferryhopper --target prod
2025-07-04 04:26:40 — INFO — Running dbt command --log-format json deps --project-dir /home/runner/work/bi/bi/dbt/.venv/lib/python3.12/site-packages/elementary/monitor/dbt_project --profiles-dir /home/runner/work/bi/bi/dbt/ferryhopper --target prod
DEBUG:elementary.clients.dbt.command_line_dbt_runner:Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.can_upload_source_freshness", "macro_args": {"invocation_id": "16ef9295-1086-4466-a83a-7a06c4710721"}} --project-dir /home/runner/work/bi/bi/dbt/.venv/lib/python3.12/site-packages/elementary/monitor/dbt_project --profiles-dir /home/runner/work/bi/bi/dbt/ferryhopper --target prod
Error: Source freshness for invocation id 16ef9295-1086-4466-a83a-7a06c4710721 were already uploaded.
Error: Process completed with exit code 1.

The way that we run elementary is via a github action as follows. Note that I've cleared all caches and there were no retries!

name: dbt pipeline
  
on:
  ...

env:
  ...

jobs:
  scheduled_dbt_run:
    name: scheduled_dbt_run
    runs-on: ubuntu-latest
    concurrency:
      # Avoid multiple instances of dbt running at the same time on the same project
      group: dbt_ferryhopper_${{ github.ref == 'refs/heads/main' && 'prod' || 'qa' }}
      cancel-in-progress: true # Give priority to this run (see BI-1411)

    steps:

      # Preliminaries

      - name: Check out
        uses: actions/checkout@v4

      - name: Setup DBT
        uses: ./.github/actions/dbt_setup
        env:
          GOOGLE_SERVICE_ACCOUNT_DBT: ${{ env.GOOGLE_SERVICE_ACCOUNT_DBT }}
        with:
          should_run_dbt_seed: true
          should_create_bigquery_udfs: true
          dbt_dir: ${{ env.DBT_DIR }}
          dbt_target: ${{ env.DBT_TARGET }}
          venv_path: ${{ env.VENV_PATH }}


      # DBT run

      - name: Run dbt
        uses: nick-fields/retry@v3  # Retries to address sync issues
        with:
          timeout_minutes: 45
          max_attempts: 1
          retry_wait_seconds: 300
          retry_on: error
          command: |
            source $VENV_PATH/bin/activate
            dbt build --project-dir $DBT_DIR \
                      --profiles-dir $DBT_DIR \
                      --target $DBT_TARGET \
                      --full-refresh \
                      --exclude models/marts/anomalies_monitoring \
                      --exclude models/marts/reporting_google_analytics_4

      # Run freshness tests

      - name: Run freshness tests
        if: ${{ github.ref == 'refs/heads/main' }}
        run: |
          source $VENV_PATH/bin/activate
          dbt source freshness --project-dir $DBT_DIR \
                               --profiles-dir $DBT_DIR \
                               --target $DBT_TARGET \
                               --exclude source:airbyte_intercom

      [...]

      - name: Upload source freshness results to elementary
        # <https://docs.elementary-data.com/oss/guides/collect-dbt-source-freshness#collect-dbt-source-freshness-results>
        if: (github.ref == 'refs/heads/main') && (success() || failure()) # Only for main branch and except cancellation
        run: |
          source $VENV_PATH/bin/activate
          edr run-operation upload-source-freshness \
                      --project-dir $DBT_DIR \
                      --profiles-dir $DBT_DIR \
                      --profile-target $DBT_ELEMENTARY_TARGET

There is another discussion going on about this in the #C08HW57FGN9 channel, but it was not resolved.

Hi @George Tzanakis, Do you have the

source_freshness_run_project_hooks: true

flag set in your

dbt_project.yml

? If so, the source freshness results are already uploaded by the Elementary dbt package, which means you won’t be able to upload them again using

edr run-operation

. If you still prefer to upload them manually with the

edr

command, you can remove the flag from your

dbt_project.yml

. However, I recommend keeping the flag in place. Note that in dbt 1.10, this flag was changed to be true by default, so if you updated dbt, that could be the source of the problem.

Got it, thank you @Michael Myaskovsky!