<@U088WCP3B4P> I got the following error in a `dbt...
# ask-elementary-ai
m
@kapa.ai I got the following error in a
dbt test
execution inside an ECS container:
k
A message from kapa.ai
Based on the error you're experiencing with
dbt test
in an ECS container, this appears to be related to resource constraints. Several users in the Elementary Data community have encountered similar issues. The most common cause for this type of error is insufficient memory allocation to your container. When Elementary Data runs operations, particularly when generating reports or running tests, it can require significant memory resources, especially with larger dbt projects. From the knowledge sources, there are multiple instances where users encountered
Failed to run dbt command
errors in containerized environments (Docker, ECS, Kubernetes) that were resolved by increasing memory allocation: 1. One user reported that their Elementary Data Report (EDR) generation via Kubernetes started failing with a similar error, and the solution was to lock dependency versions [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1719519223140569). 2. Another user experienced the error
subprocess.CalledProcessError: Command '[dbt...]' died with <Signals.SIGKILL: 9>
which indicates the process was forcibly killed due to exceeding available resources [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1729498047922669). 3. A user running Elementary in ECS had to increase memory from 600MB to 1024MB for a project with around 2000 models and 1000 tests [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1678963756105439). To resolve this issue, I recommend: 1. Increase the memory allocation for your ECS container 2. Check if there are any dependency conflicts that might be causing issues 3. Verify that all required permissions are set correctly (some users encountered permission errors) If increasing resources doesn't solve the problem, you might want to check the dbt logs for more specific error messages, as the Elementary error message sometimes doesn't show the root cause. Let me know if you need more specific guidance on any of these steps! I'm here to help with any Elementary Data related questions you might have. Note: If you're running a large dbt project, you might also consider optimizing your test selection to reduce resource requirements. I'm here to help with any Elementary Data related questions you might have. Note: If you're running a large dbt project, you might also consider optimizing your test selection to reduce resource requirements.
m
@kapa.ai The "error" that I'm seeing is the following:
Missing Elementary models in 'AwsDataCatalog.elementary'. Please run 'dbt run -s elementary --target athena'.
Copy code
[2025-04-23, 16:24:21 UTC] {ecs.py:227} INFO - [2025-04-23 16:24:16,958] 16:24:16  9 of 9 PASS not_null_gd_occ_interaction_attention_level_vcc_uuid ............... [PASS in 2.84s]
[2025-04-23, 16:24:21 UTC] {ecs.py:227} INFO - [2025-04-23 16:24:16,963] 16:24:16  
[2025-04-23, 16:24:21 UTC] {ecs.py:227} INFO - [2025-04-23 16:24:17,066] 16:24:17  Missing Elementary models in 'AwsDataCatalog.elementary'. Please run 'dbt run -s elementary --target athena'.
[2025-04-23, 16:24:21 UTC] {ecs.py:227} INFO - [2025-04-23 16:24:17,112] 16:24:17  1 of 1 START hook: elementary.on-run-end.0 ..................................... [RUN]
[2025-04-23, 16:24:21 UTC] {ecs.py:227} INFO - [2025-04-23 16:24:17,112] 16:24:17  1 of 1 OK hook: elementary.on-run-end.0 ........................................ [OK in 0.15s]
[2025-04-23, 16:24:21 UTC] {ecs.py:227} INFO - [2025-04-23 16:24:17,115] 16:24:17  
[2025-04-23, 16:24:21 UTC] {ecs.py:227} INFO - [2025-04-23 16:24:17,115] 16:24:17  Finished running 2 project hooks, 9 data tests in 0 hours 0 minutes and 14.24 seconds (14.24s).
[2025-04-23, 16:24:21 UTC] {ecs.py:227} INFO - [2025-04-23 16:24:17,688] 16:24:17  
[2025-04-23, 16:24:21 UTC] {ecs.py:227} INFO - [2025-04-23 16:24:17,689] 16:24:17  Completed successfully
Before that I attached to the execution role the policy to allow glue:GetTables to the elementary Glue database.
k
A message from kapa.ai
Based on your logs, you're encountering a common issue with Elementary Data:
Missing Elementary models in 'AwsDataCatalog.elementary'. Please run 'dbt run -s elementary --target athena'.
This message appears because Elementary Data can't find its required tables in your Athena database. This is a known issue that several users have encountered, particularly with Athena as the data warehouse. From the knowledge sources, there are a few key points to understand: 1. Elementary Data requires its models to be built before it can store test results and other artifacts. When you run
dbt test
, Elementary tries to write to these tables via its on-run-end hook. 2. Several users have reported this exact issue with different data warehouses, including Athena. The solution is typically to run
dbt run -s elementary
with the correct target before running tests. 3. For Athena specifically, there are some known compatibility challenges. As mentioned in one discussion, "Athena is not officially supported yet" by Elementary, though community members have been working on it. To resolve this issue: 1. First, try running the suggested command:
dbt run -s elementary --target athena
2. If that doesn't work, try adding
--full-refresh
flag:
dbt run -s elementary --full-refresh --target athena
- This helped one user who reported: "this actually solved it for me!" [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1702420274487159) 3. Make sure your
dbt_project.yml
has the correct schema configuration:
Copy code
yaml    models:      elementary:        +schema: elementary
4. For Athena specifically, you might need to add an Athena-specific macro called
complete_buckets_cte
as mentioned by one user trying to use Elementary with Athena. Note that while you've added the
glue:GetTables
permission, which is good, Athena has some specific challenges with Elementary's incremental update logic. One user noted: "Some incremental update logic doesn't work..." with Athena. If you continue to face issues, you might want to check if the Elementary tables are being created in a different schema than expected, which was the case for some users. Let me know if these steps help or if you need further assistance! [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1681248141754269) indicates that Athena users face specific challenges with Elementary's table update mechanism. kapa.ai: I've provided information based on similar issues in the Elementary Data community. Would you like more specific guidance on any of these steps? kapa.ai: I've provided information based on similar issues in the Elementary Data community. Would you like more specific guidance on any of these steps?
m
@kapa.ai It's still happening
k
A message from kapa.ai
I understand that you're still experiencing the issue with the message
Missing Elementary models in 'AwsDataCatalog.elementary'
even after trying the suggested solutions. This is a persistent issue that several Elementary Data users have encountered, particularly with Athena. Let me suggest some additional troubleshooting steps based on similar cases in the knowledge sources: 1. Check schema configuration: Make sure your Elementary tables are being created in the correct schema. One user mentioned that their tables were being created in
jomccr_elementary
schema while the error was looking for them in
DATABASE.jomccr
[Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1702420274487159). 2. Try a more aggressive refresh: Some users found success with:
Copy code
dbt run -s elementary --vars '{elementary_full_refresh: true}' --target athena
3. Check for schema mismatches: One user reported that changing
+schema: "elementary"
to
+schema: 'elementary'
(single quotes instead of double quotes) fixed their issue [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1741684375762739). 4. Verify target consistency: Make sure you're running Elementary commands and your dbt tests on the same target. One support staff noted: "elementary and dbt test were not running on the same target" [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1684095583307509). 5. Check for Athena-specific issues: For Athena users specifically, there are known issues with incremental tables. One user mentioned: "the branch will not work if default
table_type
will be set to `iceberg`" [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1697812888632319). If you're still facing issues after trying these steps, it might be worth checking if there are any Athena-specific configurations needed for Elementary, as Athena is not officially supported yet by Elementary Data. Would you be able to share more details about your setup, such as your dbt_project.yml configuration for Elementary and the output of
dbt run -s elementary --target athena
? This would help narrow down the specific issue you're encountering. I'm here to help you troubleshoot this further!