@Elementary Support Hello! We use dbt + elementary in BigQuery with Airflow orchestrator. We render single dbt model as single airflow task using Cosmos. So on every dbt model run (single Airflow task) we have elementary on run-end hook. The problem is that we have many models run in parallel and we ran into BigQuery limitation on dml operation per table, BigQuery allows no more than 25 DML statements every 10 seconds https://cloud.google.com/bigquery/quotas so elementary on run-end hooks fails with error

429 Exceeded rate limits: too many table dml insert operations for this table. For more information, see <https://cloud.google.com/bigquery/docs/troubleshoot-quotas>; reason: rateLimitExceeded, location: dml_job_insert_rate, message: Exceeded rate limits: too many table dml insert operations for this table. For more information, see <https://cloud.google.com/bigquery/docs/troubleshoot-quotas>

As we understand every on run-end hook write data to

dbt_run_results

elementary table and we have to many operation for this table. Maybe you have some idea how to handle with this problem?

Hey @Anton Ketska! Please check out the answer in #C08HW57FGN9. Let me know if you have any further questions 🙏

@Noya Offer Thanks! We already implemented step 1. Disable Artifacts Autoupload in Hooks and step 3. Reduce Parallelism or Use Airflow Pools to handle this errors. But step 3 significantly reduces performance as we can't run more than 25 models in parallel. Looks like step 2. Batch or Aggregate Results Outside Hooks is the best option, do you have some plans to implement something like mentioned in this thread https://elementary-community.slack.com/archives/C02CTC89LAX/p1740748388054459?