The error message "Source freshness for invocation id ... were already uploaded" indicates that Elementary has detected an attempt to upload source freshness results for a dbt invocation that has already been processed. This is a safeguard to prevent duplicate uploads of the same source freshness results. Based on the knowledge sources, this situation can occur if: - The same

dbt source freshness

results are being uploaded multiple times, possibly due to a retry or a misconfiguration in your CI/CD pipeline. - There is a step in your workflow that triggers the upload more than once for the same invocation, for example, if the upload command is run after both

dbt run

and

dbt source freshness

, or if the pipeline is re-running failed steps without cleaning up previous artifacts. To resolve this: 1. Check your pipeline steps: Ensure that the upload command (e.g.,

edr run-operation upload-source-freshness

) is only executed once per unique

dbt source freshness

run. Avoid running it multiple times for the same invocation. 2. Clean up artifacts: If your pipeline reuses the

target/sources.json

file from a previous run, make sure to clean or overwrite it before each new run to avoid re-uploading old results. 3. Review concurrency: If you have concurrent jobs or retries, ensure they are not causing duplicate uploads for the same invocation ID. Concurrency issues have been noted as a source of similar problems in other contexts with Elementary and dbt integrations [Concurrency issue discussion](https://slack.com/archives/C02CTC89LAX/p1672768342230909). If you continue to see this error and your pipeline is correctly configured to avoid duplicate uploads, it may be helpful to review your logs (

edr.log

and

dbt.log

) for more context or share them with the Elementary support team for further troubleshooting. If you need more specific guidance, please provide details about your pipeline steps and any relevant logs. The knowledge sources do not mention a direct fix for this exact error message, but the above steps are consistent with known causes and best practices for similar issues.

@kapa.ai These are my git actions - name: Run dbt source freshness continue-on-error: true run: | dbt source freshness \ --profiles-dir $GITHUB_WORKSPACE/profile/ \ --project-dir $GITHUB_WORKSPACE - name: Upload source freshness results to Elementary continue-on-error: true uses: elementary-data/run-elementary-action@v1.11 with: warehouse-type: snowflake edr-command: | edr run-operation upload-source-freshness \ --profiles-dir $GITHUB_WORKSPACE/profile/ \ --project-dir $GITHUB_WORKSPACE

Your GitHub Actions workflow for uploading dbt source freshness results to Elementary looks mostly correct, but the error "Source freshness for invocation id ... were already uploaded" suggests that the same results are being uploaded more than once for a single dbt invocation. Key points to check and recommendations: 1. Single Upload per Invocation: Make sure that the

edr run-operation upload-source-freshness

command is only executed once for each unique run of

dbt source freshness

. If your workflow or job is retried, or if the same

sources.json

file is reused, this error can occur. 2. Job Structure: The error can happen if your workflow is split into separate jobs (not just steps), because the

target/sources.json

file generated by

dbt source freshness

may persist and be reused, causing Elementary to see it as a duplicate upload. It's recommended to keep both the

dbt source freshness

and the

edr run-operation upload-source-freshness

commands in the same job, so the context and files are fresh and unique for each run. This is supported by community advice: > "My hunch is that the problem is that you split the flow into 2 different jobs (dbt and elementary). Can you try and merge them into one job with multiple steps?" [Source](https://slack.com/archives/C02CTC89LAX/p1691621340641759) 3. Clean Artifacts: Ensure that the

target/sources.json

file is cleaned or overwritten before each run, so you are not re-uploading results from a previous invocation. 4. Continue-on-error: While you have

continue-on-error: true

, this will allow the workflow to proceed even if the upload fails, but it does not resolve the root cause. It's better to address the duplication issue directly. Summary of what to do: - Keep both commands in the same job (not separate jobs). - Ensure the

target/sources.json

is not reused across runs. - Only run the upload command once per

dbt source freshness

execution. If you follow these steps and still encounter the error, please provide more details about your workflow structure or any additional logs for further troubleshooting. Let me know if you need help restructuring your workflow YAML!