@kapa.ai We're integrating the Elementary dbt package into our Airflow-orchestrated dbt project running on AWS Athena with the dbt-athena-community adapter. Our Elementary tables are in Iceberg format. Our setup: - Many domain-specific Airflow DAGs, each running dozens of dbt models - Each model runs as an individual KubernetesPodOperator task (one pod per model) - Multiple DAGs run in parallel — at peak, 50–100 pods simultaneously, each firing Elementary's on-run-end hook at the end of their dbt invocation - Each hook triggers upload_run_results(), all writing concurrently to the same Elementary Athena tables What we know about Iceberg concurrency: We've read the AWS blog on concurrent write conflicts in Iceberg (https://aws.amazon.com/blogs/big-data/manage-concurrent-write-conflicts-in-apache-iceberg-on-the-aws-glue-data-catalog/). It states that pure INSERTs (AppendFiles) have no risk of data conflicts — only catalog commit conflicts, which Iceberg retries automatically. Since upload_run_results() is append-only, we believe concurrent podwrites should be safe in practice. Our questions: 1. Can you confirm that Elementary's upload_run_results() and upload_dbt_artifacts() are purely append/INSERT operations with no UPDATEs or DELETEs that could causenon-retryable data conflicts? 2. Have you seen issues in production with high-concurrency Iceberg INSERT patterns on Athena specifically? Are there any known limitations with the AWS Glue catalog'scommit retry behaviour under this kind of load? 3. We currently have disable_run_results: true set. Assuming the above is safe, is simply removing this flag the recommended way to re-enable run-level observability,or is there additional configuration needed? 4. Is there a supported way to upload run results outside of the on-run-end hook — for example, by reading a saved run_results.json file — similar to how edr run-operation upload-source-freshness reads sources.json? We'd prefer a decoupled approach if one exists.

Great set of questions. Here's what the knowledge sources can and cannot confirm for your specific setup: 1. Are upload_run_results() and upload_dbt_artifacts() purely append/INSERT operations? Partially confirmed, with an important distinction: •

upload_run_results()

/

dbt_run_results

and

elementary_test_results

— these are described as append-only tables (INSERT only, no deletes or updates). [Cosmos/Athena discussion] •

upload_dbt_artifacts()

/ artifact metadata tables (

dbt_models

,

dbt_sources

,

dbt_tests

, etc.) — these are NOT append-only. They use a truncate-and-insert (DELETE + INSERT) pattern when the models run. [artifacts write pattern] This is precisely why concurrent writes to these tables cause conflicts across all warehouses. So your Iceberg analysis about AppendFiles being safe applies to

upload_run_results()

, but not to

upload_dbt_artifacts()

, which involves non-retryable destructive operations. 2. Known issues with high-concurrency Iceberg INSERT on Athena? The knowledge sources confirm that Athena + Iceberg is a known problematic combination with Elementary: > "if I use iceberg tables format, it does not support parallel inserts" [GitHub issue] Even for append-only operations, Iceberg on Athena has been reported to have issues with parallel inserts at scale. The knowledge sources also note that Elementary has made fixes to properly sequence and clean temp tables/files for Athena — upgrading to the latest Elementary package is recommended. [Athena temp tables] The knowledge sources do not contain specific data on Glue catalog commit retry behaviour under 50–100 concurrent pod load. I cannot confirm or deny your specific concurrency assumptions at that scale. 3. Re-enabling run results by removing disable_run_results: true Yes, removing

disable_run_results: true

is the way to re-enable run-level observability. However, given your scale (50–100 concurrent pods), the knowledge sources strongly suggest also setting:

vars:
  disable_dbt_artifacts_autoupload: true

and running artifact uploads separately in a single controlled job:

dbt run --select elementary.edr.dbt_artifacts

This decouples the dangerous truncate-and-insert artifact writes from your parallel runs, while allowing append-only run results to flow through the hooks. [recommended pattern] You may also want to consider chunking inserts to handle large volumes:

vars:
  insert_rows_method: chunk
  dbt_artifacts_chunk_size: 5000

[chunking fix] 4. Uploading run results outside the on-run-end hook (decoupled approach) The knowledge sources confirm this is not natively supported as of the latest available information: > "there is no out-of-the-box mechanism to collect artifacts from S3 (or other object storage) and insert them into Elementary tables... This was recognized as a need, and the Elementary team expressed openness to further discussion, but no ready-made solution was provided." [batching discussion] The idea of reading

run_results.json

from S3 and ingesting it into Elementary tables has been discussed by community members and the Elementary team, but no supported command or API equivalent to

upload-source-freshness

exists for run results. [GitHub feature request] Summary Given your scale, I'd recommend reaching out to the Elementary team directly to discuss the batching/S3 ingestion approach, as they've expressed openness to collaborating on this. [community discussion] 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements

@kapa.ai Given the issue related to parallel inserts in Iceberg table is dated to before the mentioned Article that says inserts should cause no data conflicts. Do you think that means the problem has been solved on iceberg side?

This is a reasonable inference, but I need to be transparent about the limits of what the knowledge sources can confirm here. What the knowledge sources say The GitHub issue reporting Iceberg parallel insert problems on Athena is from the Elementary repository and reflects a limitation observed at that time. [GitHub issue] The knowledge sources do not contain any follow-up confirmation from the Elementary team or community members that this specific Iceberg limitation has since been resolved — the issue was simply closed due to inactivity, not because a fix was confirmed. What I cannot confirm The knowledge sources do not reference the AWS blog article you cited, nor do they contain any discussion of whether improvements to Iceberg's AppendFiles commit retry behaviour have resolved the parallel insert problem for Elementary's specific write patterns on Athena. Your reasoning is technically plausible — if Iceberg now handles concurrent AppendFiles safely via automatic catalog commit retries, then append-only Elementary writes could be safe. However: • I cannot confirm this from the knowledge sources alone. • The knowledge sources do not distinguish between catalog commit conflicts (retryable) and data conflicts in the context of Elementary's Athena integration specifically. • The issue closure was due to staleness, not resolution. [GitHub issue] Recommendation Given this uncertainty, I'd strongly suggest: 1. Testing empirically at moderate concurrency before scaling to 50–100 pods. 2. Reaching out directly to the Elementary team — they explicitly expressed openness to discussing Athena integration improvements. [community discussion] The knowledge sources simply don't contain enough information to confirm whether the Iceberg-side improvements you're referencing have made Elementary's Athena/Iceberg integration safe at your scale. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.