Vamsi
07/15/2025, 8:39 PMkapa.ai
07/15/2025, 8:39 PMkapa.ai
07/15/2025, 8:39 PMedr against. - The profiles.yml file is not correctly configured or not found in the expected location. - There may be a mismatch between the dbt and Elementary package versions. Troubleshooting steps: - Ensure you have run dbt run --select elementary and that the Elementary schema exists and is populated in your Snowflake database. - Double-check that your profiles.yml is in the correct location (typically ~/.dbt/) and that the schema/database names match exactly (case-sensitive). - Make sure your elementary profile has a target specified, or that you are passing the correct target with --target when running edr commands. - If you are using dbt version 1.5.0, note that there is a known bug that causes this error. Upgrading to dbt 1.5.1 or later resolves it [see here](https://slack.com/archives/C02CTC89LAX/p1695830563447949). ### 2. "Invalid Identifier UNIQUE_KEY" (Snowflake 42000 error) This error is related to a SQL compilation issue, where the identifier UNIQUE_KEY is not found or is incorrectly referenced. This can happen if the dbt package or Elementary CLI is not up to date, or if the models were not rebuilt after a version upgrade. Troubleshooting steps: - After upgrading the Elementary dbt package, you must explicitly run dbt run --select elementary to rebuild the Elementary models and tables. Running dbt run without the --select elementary option may not be sufficient [see here](https://slack.com/archives/C02CTC89LAX/p1740038661814829). - Ensure that your dbt and Elementary package versions are compatible. - If you recently updated the Elementary dbt package, always rebuild the models as described above. ### Summary of Actions 1. Upgrade dbt to at least 1.5.1 if you are on 1.5.0. 2. Ensure your profiles.yml is correct and in the right location. 3. Run: dbt run --select elementary after any Elementary dbt package upgrade. 4. Confirm that the schema/database names in your profile match exactly what exists in Snowflake. If you follow these steps and still encounter issues, please provide your profiles.yml and the full error logs for further troubleshooting. References: - [dbt 1.5.0 bug and fix](https://slack.com/archives/C02CTC89LAX/p1695830563447949) - [Explicit dbt run --select elementary required after upgrade](https://slack.com/archives/C02CTC89LAX/p1740038661814829) - [Schema/profile troubleshooting](https://slack.com/archives/C02CTC89LAX/p1701295678298669)Vamsi
07/15/2025, 8:51 PMVamsi
07/15/2025, 8:51 PMkapa.ai
07/15/2025, 8:51 PMkapa.ai
07/15/2025, 8:52 PMprofiles.yml file for use with Snowflake, based on the official documentation and community examples: --- ### Sample elementary Profile for Snowflake yaml elementary: outputs: default: type: snowflake account: [account id] user: [username] password: [password] role: [user role] database: [database name] warehouse: [warehouse name] schema: [schema name]_elementary threads: 4 target: default - Replace the bracketed values with your actual Snowflake credentials and details. - The schema is typically your main schema suffixed with _elementary (e.g., MYSCHEMA_elementary). [Reference: Elementary CLI Quickstart](https://docs.elementary-data.com/oss/quickstart/quickstart-cli) --- ### Example profiles.yml with Multiple Profiles yaml deployment: outputs: staging: type: snowflake account: [account id] user: [username] password: [password] role: [user role] database: [database name] warehouse: [warehouse name] schema: [main schema] threads: 8 target: staging elementary: outputs: default: type: snowflake account: [account id] user: [username] password: [password] role: [user role] database: [database name] warehouse: [warehouse name] schema: [main schema]_elementary threads: 8 target: default - This example shows both a deployment profile (for your main dbt runs) and an elementary profile (for Elementary CLI). - Make sure the elementary profile points to the schema where the Elementary tables are created. [Reference: Community Example](https://slack.com/archives/C02CTC89LAX/p1741869083525939) --- If you need a sample for a different warehouse (e.g., BigQuery, Redshift), let me know!Vamsi
07/15/2025, 9:07 PMkapa.ai
07/15/2025, 9:07 PMkapa.ai
07/15/2025, 9:07 PMedr is installed and linked If running where edr (on Windows) or which edr (on Mac/Linux) returns nothing, it means the edr command is not available in your system's PATH. This usually means the Elementary CLI is not installed, or your Python/Scripts directory is not in your PATH. How to fix: - Ensure you have installed the CLI: pip install elementary-data pip install 'elementary-data[snowflake]' - Make sure your Python Scripts directory is in your PATH. On Windows, you may need to add something like C:\Users\<your_user>\AppData\Local\Programs\Python\Python39\Scripts to your PATH variable. [See this guide for adding Python to PATH](https://geek-university.com/add-python-to-the-windows-path/) - After updating your PATH, restart your terminal and run edr --help to confirm installation. [More troubleshooting steps here](https://docs.elementary-data.com/oss/general/troubleshooting) - If you still have issues, you can always run the CLI by calling Python directly: python -m elementary.cli --help --- ### 2. Errors: "Unable to get the latest invocation: list index out of range" and "Failed to parse elementary database and schema" These errors are commonly caused by: - The Elementary schema or tables not existing in your target database. - The profiles.yml file not being found or misconfigured. - A mismatch between dbt and Elementary package versions. - Running edr in the wrong directory or with the wrong environment. Troubleshooting steps: - Make sure you have run: dbt run --select elementary This creates the necessary Elementary tables in your database. [Reference](https://slack.com/archives/C02CTC89LAX/p1701295678298669) - Ensure your profiles.yml is in the default location (~/.dbt/profiles.yml) or specify the path with --profiles-dir. - The profile name in profiles.yml should be elementary, and the schema/database should match exactly (case-sensitive). - If you are using a virtual environment, ensure you are running edr from within that environment, and that the environment's Scripts directory is in your PATH. - If you are on dbt 1.5.0, upgrade to at least 1.5.1 due to a known bug. - If you recently upgraded the Elementary dbt package, always run dbt run --select elementary to rebuild the models. --- ### 3. SQL Compilation Error: Invalid Identifier ‘UNIQUE_KEY’ This error suggests that the dbt package or Elementary CLI is referencing a column or identifier that does not exist in your schema, possibly due to version mismatches or incomplete model builds. How to fix: - After upgrading the Elementary dbt package, always run: dbt run --select elementary This ensures all models and tables are rebuilt with the correct structure. [Reference](https://slack.com/archives/C02CTC89LAX/p1740038661814829) - Double-check that your dbt and Elementary package versions are compatible. --- ### Summary - Ensure edr is installed and in your PATH. - Confirm your profiles.yml is correct and in the right location. - Run dbt run --select elementary after any package upgrade. - Check for version compatibility between dbt and Elementary. - If you still have issues, provide your profiles.yml and error logs for further help. If you need a sample profiles.yml, let me know!Vamsi
07/15/2025, 9:17 PMkapa.ai
07/15/2025, 9:17 PMkapa.ai
07/15/2025, 9:18 PMprofiles.yml file and reads from the tables created by the Elementary dbt package to generate these reports. It is essential for visualizing and exploring your data quality trends and pipeline health [[Elementary OSS Introduction](https://docs.elementary-data.com/oss/oss-introduction)]. ### How to Install the CLI To install the Elementary CLI, you typically run: pip install elementary-data For Snowflake (and other specific warehouses), you should also run: pip install 'elementary-data[snowflake]' After installation, you can check if the CLI is available by running: edr --help If you are behind a corporate proxy, you may need to configure your environment to allow pip to access the internet through the proxy. This is usually done by setting the HTTP_PROXY and HTTPS_PROXY environment variables before running the install commands. For example: set HTTP_PROXY=<http://your.proxy.server:port> set HTTPS_PROXY=<http://your.proxy.server:port> pip install elementary-data If you encounter SSL certificate issues, you may need to update your certificates or use a tool like certifi as discussed in community support threads [[Slack: SSL certificate troubleshooting](https://slack.com/archives/C02CTC89LAX/p1700825084444359)]. ### Why Are Reports Blank? Blank reports are usually caused by one of the following: - The Elementary dbt package tables (like elementary_test_results) are empty or not created. - The CLI cannot connect to the correct schema or database (often due to a misconfigured profiles.yml). - Permissions issues or missing data in the warehouse. To resolve this: 1. Ensure you have run dbt run --select elementary and dbt test to populate the tables. 2. Check that your profiles.yml is correctly configured and points to the schema where the Elementary tables are located. 3. Make sure the CLI is using the correct profile and has the necessary permissions [[Troubleshooting Guide](https://docs.elementary-data.com/oss/general/troubleshooting)]. If you continue to see blank reports, follow the troubleshooting steps in the documentation or reach out to the Elementary community for support. --- If you need a sample profiles.yml or further help with proxy configuration, let me know!Vamsi
07/16/2025, 3:56 AMkapa.ai
07/16/2025, 3:56 AMkapa.ai
07/16/2025, 3:57 AMprofiles.yml file. The profile name is case-sensitive, so if your profile is named ELEMENTARY (all uppercase), it will not be found. You need to rename your profile in profiles.yml to elementary (all lowercase) to match what the CLI expects. The schema name (ELEMENTARY) can remain uppercase if that's how it exists in your database, but the profile name in profiles.yml must be lowercase elementary. Regarding your dbt command: - The dbt run --select elementary command refers to the dbt model or package named elementary, not the profile. This is correct if your dbt package or model is named elementary. - The error is unrelated to the case of the model/package in the --select argument; it is about the profile name in profiles.yml. Summary: - Change your profile name in profiles.yml to elementary (lowercase). - Keep your schema name as ELEMENTARY if that's what you use in your warehouse. - Continue running dbt run --select elementary if your dbt package/model is named elementary. This matches the requirements and troubleshooting steps discussed in the knowledge sources, where the CLI expects a profile named elementary and is case-sensitive about it [see here](https://slack.com/archives/C02CTC89LAX/p1686013979045139).Vamsi
07/16/2025, 4:40 AMkapa.ai
07/16/2025, 4:40 AMkapa.ai
07/16/2025, 4:40 AMedr report require urllib3? The knowledge sources show that urllib3 is present in the environment when running Elementary CLI (edr), and warnings related to urllib3 versions have appeared in logs. However, there is no explicit statement that edr report directly requires urllib3. It is likely used indirectly via dependencies such as requests, but the knowledge sources do not confirm a direct requirement. If you encounter urllib3-related errors, ensuring it is installed and compatible with your environment is recommended, but this is not explicitly documented as a requirement for edr report in the sources provided. ### 2. How to manually install dbt packages (when dbt deps is blocked by a proxy)? If your corporate proxy blocks access to dbt Hub or GitHub, dbt deps will fail because it cannot download the required packages. This is a common issue in restricted environments. Workarounds from the knowledge sources: - Pre-install dbt packages during Docker build: Run dbt deps in your Dockerfile or build environment where internet access is available, so the dependencies are already present at runtime and edr does not attempt to download them again. For example: dockerfile RUN dbt deps --project-dir /path/to/your/dbt_project For the Elementary internal dbt project, you may need to run: dockerfile RUN dbt deps --project-dir /usr/local/lib/python3.11/site-packages/elementary/monitor/dbt_project (Adjust the path to match your environment.) [See this approach discussed here](https://slack.com/archives/C02CTC89LAX/p1697103899679929), [and here](https://slack.com/archives/C02CTC89LAX/p1665377430565369). - Install from GitHub directly: If you cannot use dbt Hub, you can specify the Elementary dbt package directly from GitHub in your `packages.yml`: yaml - git: "<https://github.com/elementary-data/dbt-data-reliability.git>" revision: <version> [See details here](https://slack.com/archives/C02CTC89LAX/p1676595374183909). - Proxy and SSL issues: If you are behind a proxy, you may need to configure your environment variables (HTTP_PROXY, HTTPS_PROXY) and update your SSL certificates. Some users have resolved SSL issues by upgrading certifi or installing additional certificate packages, but the sources do not provide a universal solution for all proxy environments. [See related discussion here](https://slack.com/archives/C02CTC89LAX/p1700825084444359). Summary: - edr report may use urllib3 indirectly, but this is not explicitly documented as a requirement. - To manually install dbt packages in a restricted environment, pre-run dbt deps in a location with internet access, or use a direct GitHub reference in packages.yml. - For proxy/SSL issues, set proxy environment variables and update certificates as needed. If you need more specific steps for your environment, please provide details about your setup.Vamsi
07/16/2025, 4:49 AMkapa.ai
07/16/2025, 4:49 AMkapa.ai
07/16/2025, 4:50 AMdbt deps command, specifically for the Elementary internal dbt project, would look like this: dbt deps --project-dir /usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project You should adjust the path according to your Python and environment setup. For example, in an Airflow Docker container, it might be: dbt deps --project-dir /home/airflow/.local/lib/python3.8/site-packages/elementary/monitor/dbt_project The key is to point --project-dir to the location of the dbt_project.yml file for the project you want to install dependencies for. For your own dbt project, it would typically be the root directory of your dbt project, such as: dbt deps --project-dir /path/to/your/dbt_project For the Elementary internal dbt project, use the path where the Elementary package is installed in your environment, as shown above. [Reference: Slack discussion on Docker and internal dbt project paths](https://slack.com/archives/C02CTC89LAX/p1686278697336859)Vamsi
07/17/2025, 1:16 PMkapa.ai
07/17/2025, 1:16 PMkapa.ai
07/17/2025, 1:16 PMdbt deps to download packages. This is a common issue in environments with strict corporate proxies. Solution: You need to ensure that the required dbt packages (such as elementary and dbt_utils) are already present in the internal dbt project's dbt_packages directory before running edr report. If these packages are present, the CLI will not attempt to connect to dbt Hub, and report generation can proceed offline. How to do this: 1. Pre-populate dbt packages: - On a machine with internet access, run dbt deps for the Elementary internal dbt project directory (e.g., /usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project). - Copy the resulting dbt_packages directory (which will contain the elementary and dbt_utils packages) to your offline or proxy-restricted environment, placing it in the same path. 2. Verify package presence: - The CLI checks if the required packages are present in the dbt_packages directory. If they are, it will not attempt to run dbt deps or connect to dbt Hub. - This approach has been confirmed by multiple users and support staff as a working solution for offline or proxy-restricted environments [see here](https://slack.com/archives/C02CTC89LAX/p1679910207387459), [and here](https://slack.com/archives/C02CTC89LAX/p1697103899679929), [and here](https://slack.com/archives/C02CTC89LAX/p1665377430565369). 3. Run `edr report`: - Once the packages are in place, you can run edr report without the CLI attempting to connect to the internet. Note: If you still see blank reports, ensure that: - The elementary_test_results table in your database contains data (i.e., you have run dbt test after setting up Elementary). - Your profiles.yml is correctly configured and accessible to the CLI. Summary: You do not need a live connection to hub.getdbt.com if you pre-populate the required dbt packages in the correct directory. This allows you to generate reports in a proxy-restricted or offline environment. If you need more detailed steps for copying the packages or setting up the environment, let me know!Vamsi
07/17/2025, 1:19 PMkapa.ai
07/17/2025, 1:19 PMkapa.ai
07/17/2025, 1:19 PM/usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project (The exact path may vary depending on your Python version and environment.) b. Install the Required Packages - On a machine with internet access, run: bash dbt deps --project-dir /usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project This will download the required dbt packages (such as elementary and dbt_utils) into the dbt_packages directory inside the internal dbt project. --- ## 2. Copy the dbt_packages Directory to Your Target (Offline/Proxy) Environment a. Locate the dbt_packages Directory - After running the above command, you should see a dbt_packages directory inside the internal dbt project path. b. Copy the Directory - Copy the entire dbt_packages directory (and its contents) from the source machine to the same path in your target environment. - You can use scp, rsync, or any file transfer method allowed in your organization. c. Ensure Correct Permissions - Make sure the user running the Elementary CLI in the target environment has read access to the copied dbt_packages directory. --- ## 3. Verify the Setup - In your target environment, check that the dbt_packages directory exists at: /usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project/dbt_packages (Adjust the path as needed for your Python version.) --- ## 4. Run the Elementary CLI - Now, when you run: bash edr report --project-dir <your_dbt_project> --profiles-dir <your_profiles_dir> the CLI should not attempt to connect to hub.getdbt.com, because the required packages are already present. This should resolve the proxy/SSL errors and allow report generation to proceed. --- ## 5. Troubleshooting - If you still see blank reports: - Ensure your elementary_test_results table in your data warehouse contains data (run dbt test after setting up Elementary). - Double-check your profiles.yml configuration and that the CLI is using the correct profile and schema. - Make sure the copied dbt_packages directory is not empty and contains the expected packages. --- This process is confirmed by multiple users and support staff as the recommended way to work around proxy/internet restrictions for Elementary report generation [see here](https://slack.com/archives/C02CTC89LAX/p1679910207387459), [and here](https://slack.com/archives/C02CTC89LAX/p1697103899679929), [and here](https://slack.com/archives/C02CTC89LAX/p1665377430565369). If you need help with a specific step (such as finding the correct path or copying files), please provide more details about your environment (OS, Python version, etc.).Vamsi
07/17/2025, 2:48 PMkapa.ai
07/17/2025, 2:48 PMkapa.ai
07/17/2025, 2:48 PM/usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project/dbt_packages. The sources confirm that this directory should contain the dbt packages required by Elementary, such as elementary and dbt_utils, after running dbt deps in the internal dbt project directory. However, they do not include a file-by-file or folder-by-folder breakdown of the contents. If you need to verify the structure, the best approach is to run dbt deps on a machine with internet access and inspect the resulting dbt_packages directory, then copy it as described in the previous instructions. If you encounter specific errors or missing files, please share the error message for more targeted help. [Reference: Slack discussion on copying dbt_packages for offline/proxy environments](https://slack.com/archives/C02CTC89LAX/p1679910207387459)Vamsi
07/17/2025, 3:19 PMkapa.ai
07/17/2025, 3:19 PMkapa.ai
07/17/2025, 3:19 PMedr report is still trying to connect to hub.getdbt.com and running dbt deps, it means that the required dbt packages (such as elementary and dbt_utils) are either missing or not correctly installed in the dbt_packages directory of the internal Elementary dbt project. The CLI checks for these packages, and if they are not present, it will attempt to download them from dbt Hub, which requires internet access. Key points from the knowledge sources: - The CLI will only skip the dbt deps step if the required packages are already present in the internal dbt project's dbt_packages directory. If they are missing, it will always try to connect to dbt Hub to fetch them, regardless of your proxy settings or offline status. This is confirmed in multiple community discussions and support responses [see here](https://slack.com/archives/C02CTC89LAX/p1679910207387459), [and here](https://slack.com/archives/C02CTC89LAX/p1697103899679929), [and here](https://slack.com/archives/C02CTC89LAX/p1665377430565369). - To prevent this, you must: 1. On a machine with internet access, run dbt deps in the internal dbt project directory (e.g., /usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project). 2. Copy the resulting dbt_packages directory (with all its contents) to the same location in your offline or proxy-restricted environment. 3. Ensure the permissions are correct so that the user running edr can read the files. - If the packages are present, edr report will not attempt to connect to dbt Hub. If you have already copied the dbt_packages directory and it is still trying to connect, double-check: - The path is correct and matches the internal dbt project used by your installed Elementary CLI. - The dbt_packages directory is not empty and contains the required packages. - You are not accidentally running edr report with a different Python environment or Elementary installation. If these steps are followed and the issue persists, the knowledge sources do not provide further troubleshooting steps for this specific scenario. In that case, you may need to review your environment setup or seek additional support from the Elementary community. [Reference: Slack discussion on offline/proxy environments and dbt_packages requirements](https://slack.com/archives/C02CTC89LAX/p1679910207387459)Vamsi
07/17/2025, 6:09 PMkapa.ai
07/17/2025, 6:09 PMkapa.ai
07/17/2025, 6:10 PMc:\users\kid\appdata\roaming\python\python312\site-packages\elementary\monitor\dbt_project - Run: dbt deps --project-dir c:\users\kid\appdata\roaming\python\python312\site-packages\elementary\monitor\dbt_project - This will create a dbt_packages directory with the required packages (including the elementary package and its dbt_project.yml). 2. Copy the dbt_packages directory: - Copy the entire dbt_packages directory from the online machine to the same path in your offline environment. 3. Verify the structure: - After copying, ensure that the following file exists: c:\users\kid\appdata\roaming\python\python312\site-packages\elementary\monitor\dbt_project\dbt_packages\elementary\dbt_project.yml - The dbt_packages directory should contain the elementary and dbt_utils packages, each with their own files and a dbt_project.yml in the elementary folder. 4. Run your edr report command again in the offline environment. - The CLI should now find the required files and not attempt to connect to dbt Hub. This process is confirmed as the recommended way to work around proxy/internet restrictions for Elementary report generation in the knowledge sources [see here](https://slack.com/archives/C02CTC89LAX/p1679910207387459), [and here](https://slack.com/archives/C02CTC89LAX/p1665377430565369). If you still get a file not found error, double-check that the path and directory structure exactly match what the CLI expects.Vamsi
07/21/2025, 2:19 PM