Mariano Re
04/23/2025, 5:56 PMdbt test execution inside an ECS container:kapa.ai
04/23/2025, 5:56 PMkapa.ai
04/23/2025, 5:56 PMdbt test in an ECS container, this appears to be related to resource constraints. Several users in the Elementary Data community have encountered similar issues. The most common cause for this type of error is insufficient memory allocation to your container. When Elementary Data runs operations, particularly when generating reports or running tests, it can require significant memory resources, especially with larger dbt projects. From the knowledge sources, there are multiple instances where users encountered Failed to run dbt command errors in containerized environments (Docker, ECS, Kubernetes) that were resolved by increasing memory allocation: 1. One user reported that their Elementary Data Report (EDR) generation via Kubernetes started failing with a similar error, and the solution was to lock dependency versions [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1719519223140569). 2. Another user experienced the error subprocess.CalledProcessError: Command '[dbt...]' died with <Signals.SIGKILL: 9> which indicates the process was forcibly killed due to exceeding available resources [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1729498047922669). 3. A user running Elementary in ECS had to increase memory from 600MB to 1024MB for a project with around 2000 models and 1000 tests [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1678963756105439). To resolve this issue, I recommend: 1. Increase the memory allocation for your ECS container 2. Check if there are any dependency conflicts that might be causing issues 3. Verify that all required permissions are set correctly (some users encountered permission errors) If increasing resources doesn't solve the problem, you might want to check the dbt logs for more specific error messages, as the Elementary error message sometimes doesn't show the root cause. Let me know if you need more specific guidance on any of these steps! I'm here to help with any Elementary Data related questions you might have. Note: If you're running a large dbt project, you might also consider optimizing your test selection to reduce resource requirements. I'm here to help with any Elementary Data related questions you might have. Note: If you're running a large dbt project, you might also consider optimizing your test selection to reduce resource requirements.Mariano Re
04/23/2025, 6:02 PMMissing Elementary models in 'AwsDataCatalog.elementary'. Please run 'dbt run -s elementary --target athena'.
[2025-04-23, 16:24:21 UTC] {ecs.py:227} INFO - [2025-04-23 16:24:16,958] 16:24:16 9 of 9 PASS not_null_gd_occ_interaction_attention_level_vcc_uuid ............... [PASS in 2.84s]
[2025-04-23, 16:24:21 UTC] {ecs.py:227} INFO - [2025-04-23 16:24:16,963] 16:24:16
[2025-04-23, 16:24:21 UTC] {ecs.py:227} INFO - [2025-04-23 16:24:17,066] 16:24:17 Missing Elementary models in 'AwsDataCatalog.elementary'. Please run 'dbt run -s elementary --target athena'.
[2025-04-23, 16:24:21 UTC] {ecs.py:227} INFO - [2025-04-23 16:24:17,112] 16:24:17 1 of 1 START hook: elementary.on-run-end.0 ..................................... [RUN]
[2025-04-23, 16:24:21 UTC] {ecs.py:227} INFO - [2025-04-23 16:24:17,112] 16:24:17 1 of 1 OK hook: elementary.on-run-end.0 ........................................ [OK in 0.15s]
[2025-04-23, 16:24:21 UTC] {ecs.py:227} INFO - [2025-04-23 16:24:17,115] 16:24:17
[2025-04-23, 16:24:21 UTC] {ecs.py:227} INFO - [2025-04-23 16:24:17,115] 16:24:17 Finished running 2 project hooks, 9 data tests in 0 hours 0 minutes and 14.24 seconds (14.24s).
[2025-04-23, 16:24:21 UTC] {ecs.py:227} INFO - [2025-04-23 16:24:17,688] 16:24:17
[2025-04-23, 16:24:21 UTC] {ecs.py:227} INFO - [2025-04-23 16:24:17,689] 16:24:17 Completed successfully
Before that I attached to the execution role the policy to allow glue:GetTables to the elementary Glue database.kapa.ai
04/23/2025, 6:02 PMkapa.ai
04/23/2025, 6:02 PMMissing Elementary models in 'AwsDataCatalog.elementary'. Please run 'dbt run -s elementary --target athena'. This message appears because Elementary Data can't find its required tables in your Athena database. This is a known issue that several users have encountered, particularly with Athena as the data warehouse. From the knowledge sources, there are a few key points to understand: 1. Elementary Data requires its models to be built before it can store test results and other artifacts. When you run dbt test, Elementary tries to write to these tables via its on-run-end hook. 2. Several users have reported this exact issue with different data warehouses, including Athena. The solution is typically to run dbt run -s elementary with the correct target before running tests. 3. For Athena specifically, there are some known compatibility challenges. As mentioned in one discussion, "Athena is not officially supported yet" by Elementary, though community members have been working on it. To resolve this issue: 1. First, try running the suggested command: dbt run -s elementary --target athena 2. If that doesn't work, try adding --full-refresh flag: dbt run -s elementary --full-refresh --target athena - This helped one user who reported: "this actually solved it for me!" [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1702420274487159) 3. Make sure your dbt_project.yml has the correct schema configuration: yaml models: elementary: +schema: elementary 4. For Athena specifically, you might need to add an Athena-specific macro called complete_buckets_cte as mentioned by one user trying to use Elementary with Athena. Note that while you've added the glue:GetTables permission, which is good, Athena has some specific challenges with Elementary's incremental update logic. One user noted: "Some incremental update logic doesn't work..." with Athena. If you continue to face issues, you might want to check if the Elementary tables are being created in a different schema than expected, which was the case for some users. Let me know if these steps help or if you need further assistance! [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1681248141754269) indicates that Athena users face specific challenges with Elementary's table update mechanism. kapa.ai: I've provided information based on similar issues in the Elementary Data community. Would you like more specific guidance on any of these steps? kapa.ai: I've provided information based on similar issues in the Elementary Data community. Would you like more specific guidance on any of these steps?Mariano Re
04/24/2025, 3:23 PMkapa.ai
04/24/2025, 3:23 PMkapa.ai
04/24/2025, 3:23 PMMissing Elementary models in 'AwsDataCatalog.elementary' even after trying the suggested solutions. This is a persistent issue that several Elementary Data users have encountered, particularly with Athena. Let me suggest some additional troubleshooting steps based on similar cases in the knowledge sources: 1. Check schema configuration: Make sure your Elementary tables are being created in the correct schema. One user mentioned that their tables were being created in jomccr_elementary schema while the error was looking for them in DATABASE.jomccr [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1702420274487159). 2. Try a more aggressive refresh: Some users found success with: dbt run -s elementary --vars '{elementary_full_refresh: true}' --target athena 3. Check for schema mismatches: One user reported that changing +schema: "elementary" to +schema: 'elementary' (single quotes instead of double quotes) fixed their issue [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1741684375762739). 4. Verify target consistency: Make sure you're running Elementary commands and your dbt tests on the same target. One support staff noted: "elementary and dbt test were not running on the same target" [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1684095583307509). 5. Check for Athena-specific issues: For Athena users specifically, there are known issues with incremental tables. One user mentioned: "the branch will not work if default table_type will be set to `iceberg`" [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1697812888632319). If you're still facing issues after trying these steps, it might be worth checking if there are any Athena-specific configurations needed for Elementary, as Athena is not officially supported yet by Elementary Data. Would you be able to share more details about your setup, such as your dbt_project.yml configuration for Elementary and the output of dbt run -s elementary --target athena? This would help narrow down the specific issue you're encountering. I'm here to help you troubleshoot this further!