Based on the error message:

Error: Source freshness for invocation id 4f5676e9-d7d8-4f67-b1ad-71e4d155acf5 were already uploaded.

This indicates that Elementary is detecting the source freshness results for this specific invocation ID have already been uploaded. This is a new behavior in recent versions of Elementary, where duplicate uploads for the same invocation are prevented. What could be happening: - The

edr run-operation upload-source-freshness

command is being run more than once with the same invocation ID (which is typically tied to the dbt run or the freshness run). - The

sources.json

file generated by

dbt source freshness

is not being regenerated between runs, so the same invocation ID is reused. - There may be a change in how Elementary tracks or checks for previously uploaded freshness results in version 0.19.0 compared to 0.15.0. What to check and try: 1. Ensure

dbt source freshness

is run immediately before

edr run-operation upload-source-freshness

in the same Airflow task, and that it generates a new

sources.json

file each time. 2. Check for task retries or duplicate executions in Airflow that might cause the upload to be attempted multiple times with the same artifacts. 3. Clear or regenerate the

target/

directory (where

sources.json

is stored) before running

dbt source freshness

to ensure no stale files are present. 4. Review Airflow logs to confirm that the sequence of commands is as expected and that no previous upload for the same invocation is happening. Unfortunately, the knowledge sources do not contain explicit documentation about this specific error message or changes in invocation ID handling between versions 0.15.0 and 0.19.0. However, similar issues with source freshness uploads and artifact handling have been discussed, especially around ensuring the correct sequence of commands and artifact freshness[Slack thread on source freshness issues](https://slack.com/archives/C02CTC89LAX/p1694700068197439). If after these checks the issue persists, it may be worth reaching out to the Elementary community or support, as this could be a new or edge-case behavior introduced in the latest version. Be sure to provide your Airflow DAG/task sequence and any relevant logs for further troubleshooting.

The knowledge sources do not provide detailed documentation on exactly how the invocation ID is generated by Elementary or dbt. However, from user discussions and support responses, we know the following: - The invocation ID in the context of Elementary is associated with a specific dbt command execution (such as a run, test, or source freshness command). - For most dbt operations, the invocation ID is unique to each run and is used to track and log the results in Elementary tables. - In the case of source freshness uploads, the invocation ID stored in the dbt_source_freshness_results table is the one from the upload operation itself (i.e., the invocation of edr upload-source-freshness), not the original dbt source freshness run. This has been confirmed by Elementary support: > "the invocation_id is of the invocation who wrote this line to the DWH, meaning the invocation of upload_source_freshness. At the moment we don't save the source freshness invocation id..." [Slack thread on invocation_id in source freshness](https://slack.com/archives/C02CTC89LAX/p1689702183543509) - There is no explicit documentation in the knowledge sources about the algorithm or method used to generate the invocation ID itself (e.g., whether it is a UUID, hash, or other mechanism). If you need to track or debug invocation IDs, you may want to inspect the logs generated by dbt and Elementary, as they often print the invocation ID at the start of each run. If you need more technical details about the generation process, you may need to consult the dbt or Elementary source code, as the knowledge sources do not provide this information.

The knowledge sources do not mention any risk of invocation_id collisions due to the size or retention of the

elementary.dbt_source_freshness_results

table, nor do they recommend periodic cleanup of this table to prevent such collisions. Invocation IDs are intended to be unique per dbt/Elementary run, and collisions should not occur under normal operation. However, there is user interest in retention and pruning of these results for storage management reasons. One user asked about configuring retention or pruning for the *_results tables (including

dbt_source_freshness_results

), but there was no official answer or recommendation provided in the sources about periodic cleanup or its impact on invocation_id uniqueness or collisions. The only mention was a user speculating about the

--days-back

argument, but this was not confirmed as a solution for retention or collision prevention [Slack thread on retention/pruning](https://slack.com/archives/C02CTC89LAX/p1728558831092529). In summary: There is no official guidance in the knowledge sources recommending periodic cleanup of the

dbt_source_freshness_results

table to minimize invocation_id collisions. If you are concerned about storage or retention, you may consider your own data retention policies, but this is not documented as a method to prevent invocation_id issues. If invocation_id collisions are occurring, it may indicate a different underlying issue with how invocation IDs are generated or reused.

@kapa.ai does anything stand out about how we're executing the upload source freshness in our dag that might look off?

dbt_upload_source_freshness_check = BashOperator(
            task_id="dbt_source_freshness",
            bash_command=(
                f"{DBT_INITIAL_CMD} && "
                f"dbt --no-use-colors source freshness --exclude tag:disabled "  # noqa: E231
                f"--profiles-dir {DBT_PROFILE_DIR} "
                f"--profile {DBT_PROFILE} "
                f"--target {DBT_PROD_TARGET} && "
                f"ret=$? && "
                f"edr run-operation upload-source-freshness "
                f"--profiles-dir {DBT_PROFILE_DIR} "
                f"--project-dir {DBT_PROFILE_DIR}{DBT_E2_CODE_BASE} "
                f"--profile-target {DBT_PROD_TARGET} && "
                f"ret=$? && exit $ret"
            ),
            retries=1,
            on_failure_callback=generic_task_error_callback,
            executor_config=k8s_dbt_executor_config,
        )

Based on the knowledge sources, your DAG task for uploading source freshness appears to follow the generally recommended sequence: 1. Run

dbt source freshness

2. Immediately run

edr run-operation upload-source-freshness

in the same BashOperator/task This matches guidance from Elementary support, who have emphasized that both commands should be run in the same task or operator to ensure the

sources.json

file is fresh and available for upload, and to avoid issues with missing or stale artifacts [Slack thread on Airflow and source freshness](https://slack.com/archives/C02CTC89LAX/p1691513076109829) [Another thread on Airflow task sequence](https://slack.com/archives/C02CTC89LAX/p1697013303000359). A few things to double-check based on common issues: - Ensure that the

--project-dir

and

--profiles-dir

arguments are pointing to the correct locations for both dbt and edr commands. Mismatches here have caused issues for other users. - Confirm that the

sources.json

file is being generated in the expected location and is accessible to the

edr

command. - Make sure there are no task retries or duplicate executions that could cause the same invocation ID to be uploaded more than once, as this is a new check in recent Elementary versions. Otherwise, nothing in your command sequence stands out as incorrect based on the available knowledge sources. If you are still encountering the "already uploaded" error, it may be related to how invocation IDs are handled or reused, which is not fully documented in the sources. If the issue persists, consider sharing more detailed logs or reaching out to the Elementary community for further troubleshooting.