To implement Elementary on your VM using an offline method (i.e., downloading the GitHub package, unzipping, and running locally to monitor BigQuery test results), here’s a step-by-step guide based on the available knowledge sources: --- ### 1. Download and Install the Elementary dbt Package - Offline Download: Download the Elementary dbt package from the [GitHub repository](https://github.com/elementary-data/elementary) on a machine with internet access. - Transfer & Unzip: Transfer the package to your VM and unzip it into your dbt project’s

packages

directory. - Install Dependencies: On your VM, ensure you have Python, dbt, and the required dbt adapter for BigQuery installed. Install the Elementary Python package (if you can’t use pip online, download the wheel or source distribution and install it manually). ### 2. Configure dbt and Elementary - dbt Project Setup: Make sure your dbt project is configured to connect to BigQuery. Your

profiles.yml

should point to your BigQuery project, dataset, and use a service account key file. - Elementary dbt Package Setup: Add the Elementary package to your

packages.yml

(if not already present from the manual download). Run:

dbt deps   dbt run --select elementary

This will create the necessary Elementary tables in your BigQuery environment. ### 3. Run dbt Tests - Execute your dbt tests as usual:

dbt test

This will populate the

elementary_test_results

table in your BigQuery dataset. ### 4. Generate and Monitor Reports with Elementary CLI - Use the Elementary CLI to generate reports and monitor test results. Example commands:

edr report --profiles-dir . --profile-target <your_target> --env <your_env>   edr monitor --profiles-dir . --profile-target <your_target> --env <your_env>

Make sure the CLI is installed and available on your VM. If you need to install it offline, download the package and install it manually. ### 5. Troubleshooting - If you don’t see test results, check that: - The

elementary_test_results

table exists and contains data. - You have run

dbt test

after setting up the package. - The table is materialized as incremental, not as a view or regular table. - Your service account has the necessary permissions for BigQuery and the information schema ([see permissions guide](https://docs.elementary-data.com/cloud/integrations/dwh/bigquery#permissions-and-security)). - If you encounter issues, refer to the [Elementary troubleshooting guide](https://docs.elementary-data.com/cloud/guides/troubleshoot). --- Note: - The process above assumes you are comfortable with manual package management and have a way to transfer files to your VM. - If you need to run the CLI completely offline, ensure all Python dependencies are also downloaded and installed manually. If you need more specific details (e.g., exact CLI flags or advanced configuration), please clarify your requirements. The knowledge sources do not provide a full offline installation guide, but the steps above are based on the standard setup process adapted for offline use [Cloud > Guides > Troubleshooting](https://docs.elementary-data.com/cloud/guides/troubleshoot) [Connect to BigQuery](https://docs.elementary-data.com/cloud/integrations/dwh/bigquery).

@kapa.ai I have Python 3.9.12 and pip packages are downloaded from artifactory using proxy. But the github is unable to connect as we have box restrictions to connect external from network side We need to manually download and settup Edr report failed when dbt run happened Saying edr needs to connect to some network

Based on the knowledge sources, your issue is common when running

edr report

in a restricted network environment. Elementary’s CLI (

edr

) uses an internal dbt project and, by default, tries to run

dbt deps

to download required dbt packages (like

elementary

and

dbt_utils

) from the internet. If your environment blocks external connections (e.g., to GitHub or dbt Hub), this step will fail and prevent report generation. Key points and solutions from the knowledge sources: - Why does this happen? The CLI checks if the required dbt packages are present in its internal dbt project. If not, it tries to download them using

dbt deps

, which requires internet access. If your server cannot reach external sites, this step fails and so does

edr report

[Slack: Red Hat, network restrictions](https://slack.com/archives/C02CTC89LAX/p1706540191539319), [Slack: Airflow, no internet](https://slack.com/archives/C02CTC89LAX/p1679910207387459), [Slack: Docker, offline](https://slack.com/archives/C02CTC89LAX/p1665377430565369). - How to work around it? You need to pre-populate the internal dbt project’s

dbt_packages

directory with the required packages before running

edr report

. This can be done by running

dbt deps

in the correct directory on a machine with internet access, then copying the resulting

dbt_packages

folder to your restricted environment. For example: 1. On a machine with internet access, install the same version of Elementary and run:

dbt deps --project-dir /path/to/site-packages/elementary/monitor/dbt_project

2. Copy the entire

dbt_packages

directory from that location to the same path on your restricted VM. 3. Ensure permissions are correct so the CLI can read these files. 4. Now, when you run

edr report

, it should not attempt to download packages and should proceed without network access [Slack: Docker, offline](https://slack.com/archives/C02CTC89LAX/p1665377430565369), [Slack: Airflow, no internet](https://slack.com/archives/C02CTC89LAX/p1679910207387459). - Flags like

--update-dbt-package

Even with

--update-dbt-package=False

, if the packages are missing, Elementary will still try to download them. The only way to avoid this is to ensure the packages are already present [Slack: Airflow, no internet](https://slack.com/archives/C02CTC89LAX/p1679910207387459). Summary: You must manually populate the internal dbt project's

dbt_packages

directory with the required packages before running

edr report

in a network-restricted environment. This is the only way to run Elementary fully offline, as confirmed by multiple community threads and support responses. If you need more detailed steps for your specific setup, let me know!

For the offline setup you described, your

packages.yml

should include the required dbt packages for Elementary to function. Based on the knowledge sources, you need at least the following:

yaml packages:   - package: dbt-labs/dbt_utils     version: 1.1.1   - package: elementary-data/elementary     version: <your_elementary_version>

Replace

<your_elementary_version>

with the version you are using (for example,

0.15.1

). This matches the examples provided in the knowledge sources and is sufficient for Elementary’s internal dbt project to work, as long as the corresponding

dbt_packages

directory is pre-populated with these packages before running

edr report

in your offline environment [Slack: Airflow, no internet](https://slack.com/archives/C02CTC89LAX/p1679910207387459), [GitHub Issue: Concurrent DBT jobs](https://github.com/elementary-data/elementary/issues/1161). If you use additional dbt packages in your project, you can add them as needed, but the two above are the minimum required for Elementary. Note: - The

packages.yml

file should be present in both your main dbt project and in the internal dbt project used by Elementary CLI (i.e., in the

elementary/monitor/dbt_project

directory). - The versions in

packages.yml

must match the versions of the packages you manually downloaded and placed in

dbt_packages

. If you have further restrictions or custom packages, adjust accordingly, but the above is the standard setup for Elementary in an offline environment.

Certainly! Here are the step-by-step instructions for setting up Elementary in an offline environment, specifically for BigQuery, based strictly on the knowledge sources: --- ## 1. Prepare the Required dbt Packages - On a machine with internet access, install the same version of Elementary as you will use on your VM. - Locate the internal dbt project directory used by Elementary CLI. This is typically at:

<python_site_packages>/elementary/monitor/dbt_project

- In that directory, create a

packages.yml

file with the following content (replace the version as needed):

yaml     packages:       - package: dbt-labs/dbt_utils         version: 1.1.1       - package: elementary-data/elementary         version: 0.16.4

(Adjust the

elementary

version to match your CLI version) [See example in the docs](https://docs.elementary-data.com/oss/quickstart/quickstart-cli-package#step-by-step:install-elementary-dbt-package) - Run the following command to download the required dbt packages:

dbt deps --project-dir <python_site_packages>/elementary/monitor/dbt_project

- After this, the

dbt_packages

directory will be populated with the necessary packages. --- ## 2. Transfer the Internal dbt Project - Copy the entire

dbt_project

directory (including

packages.yml

and

dbt_packages

) from the online machine to the same location on your offline VM. - Ensure the directory structure and permissions are preserved. --- ## 3. Install Python and pip Packages on the VM - Install Python 3.9 (as you mentioned you have 3.9.12). - Install all required pip packages (dbt, dbt-bigquery, elementary-data, etc.) using your internal artifactory or proxy. --- ## 4. Set Up Your dbt Project and Profiles - Make sure your dbt project is configured for BigQuery and that your

profiles.yml

is set up correctly. - The

elementary

profile should be present in your

~/.dbt/profiles.yml

and point to your BigQuery dataset, as shown in the docs:

yaml     elementary:       outputs:         default:           type: bigquery           method: service-account           project: <your_project>           dataset: <your_dataset>_elementary           threads: 4           keyfile: <path_to_your_service_account_json>       target: default

[See profile example](https://docs.elementary-data.com/oss/quickstart/quickstart-cli#configuring-the-elementary-profile) --- ## 5. Run dbt and Elementary - Run your dbt models and tests as usual:

dbt run     dbt test

- Now, run the Elementary CLI to generate the report:

edr report

The CLI should now use the pre-populated internal dbt project and not attempt to download anything from the internet. If the

dbt_packages

directory is present and correct, it will not fail due to network restrictions [Slack: Airflow, no internet](https://slack.com/archives/C02CTC89LAX/p1679910207387459), [Slack: Docker, offline](https://slack.com/archives/C02CTC89LAX/p1665377430565369). --- ## 6. Troubleshooting - If you see errors about missing packages, double-check that the

dbt_packages

directory is present and contains both

elementary

and

dbt_utils

. - If you see errors about permissions, ensure the user running the CLI has read access to the internal dbt project directory. --- Summary: - Prepare the internal dbt project with all dependencies on a machine with internet. - Copy the entire internal dbt project (with

dbt_packages

) to your offline VM. - Install all Python and pip dependencies via your internal proxy/artifactory. - Configure your dbt project and profiles. - Run

dbt

and

edr report

as usual—no internet required for the report step. If you follow these steps, you should be able to run Elementary fully offline as required by your environment.

I'm sorry to hear that the steps did not work and that

edr report

is still trying to download packages and removing everything from the

dbt_packages

folder. Based on the knowledge sources, this is a known issue: the Elementary CLI (

edr

) uses its own internal dbt project and, during execution, it may forcibly run

dbt deps

and overwrite the

dbt_packages

directory—even if you have pre-populated it. This behavior is confirmed in several community threads and is a limitation of the current implementation: - The CLI checks for required packages and, if it determines they are missing or not matching the expected versions, it will attempt to download them, regardless of your manual setup. Even flags like

--update-dbt-package=false

do not prevent this if the CLI thinks the packages are not present or not correct. This is why your manual population of

dbt_packages

may be overwritten or ignored [Slack: Red Hat, network restrictions](https://slack.com/archives/C02CTC89LAX/p1706540191539319), [Slack: Airflow, no internet](https://slack.com/archives/C02CTC89LAX/p1679910207387459), [Slack: Docker, offline](https://slack.com/archives/C02CTC89LAX/p1665377430565369). - There is a recent update in the Elementary repository mentioning a new

dbt-init

command to help set up the internal dbt project for CI/CD pipelines that do not have internet access. This allows you to inject dbt dependencies ahead of time, but the knowledge sources do not provide detailed documentation or step-by-step usage for this feature yet. You can see the mention here: > "Added dbt-init command to setup internal elementary dbt project so that CI/CD pipelines can inject dbt dependencies when the target environment does not have internet access to download dbt packages." [GitHub PR: Fixed setup of internal dbt project](https://github.com/elementary-data/elementary/pull/1867) In summary: The knowledge sources confirm that, as of now, the CLI may still try to download and overwrite packages even if you pre-populate

dbt_packages

. The new

dbt-init

command may address this, but there is not enough information in the sources to provide a reliable, step-by-step offline solution using this feature. Recommendation: - Monitor the Elementary documentation and GitHub for updates on the

dbt-init

command and improved offline support. - If you need a workaround now, you may need to reach out to the Elementary team directly for guidance on using the new

dbt-init

feature or for a supported offline workflow. If you have more details about your version or setup, or if you see any specific error messages, please share them and I can help troubleshoot further based on the available knowledge.

Yes, the behavior described for `edr monitor`—where the CLI attempts to download and manage dbt packages in its internal dbt project—applies to

edr report

as well. Both commands use the internal dbt project and will try to ensure the required dbt packages (like

elementary

and

dbt_utils

) are present. If they are missing or not matching the expected versions, the CLI will attempt to download them, which can cause issues in offline or restricted environments. This is confirmed in multiple community threads and support responses, where users encountered the same package download and overwrite behavior with both

edr monitor

and

edr report

commands in network-restricted setups [Slack: Red Hat, network restrictions](https://slack.com/archives/C02CTC89LAX/p1706540191539319), [Slack: Docker, offline](https://slack.com/archives/C02CTC89LAX/p1665377430565369), [Slack: Airflow, no internet](https://slack.com/archives/C02CTC89LAX/p1679910207387459). So, if you are experiencing this issue with

edr monitor

, you should expect the same behavior with

edr report

.

You need to populate the

dbt_packages

directory inside the internal Elementary dbt project (i.e., in the

elementary/monitor/dbt_project

folder within your Python site-packages), not in your own dbt project. This is because the Elementary CLI (

edr

) uses its own internal dbt project to run operations, and it specifically checks for and manages dependencies in its own

dbt_project

directory. If the required packages are not present there, it will attempt to download them—even if they exist in your main dbt project’s

dbt_packages

folder. This behavior is confirmed in several support threads and is a common source of confusion for users running in offline or restricted environments [Slack: Airflow, no internet](https://slack.com/archives/C02CTC89LAX/p1679910207387459), [Slack: Docker, offline](https://slack.com/archives/C02CTC89LAX/p1665377430565369), [Slack: dbt packages location](https://slack.com/archives/C02CTC89LAX/p1689854444623429). So, to run Elementary CLI commands like

edr report

or

edr monitor

offline, you must ensure the internal

dbt_packages

directory at

elementary/monitor/dbt_project

is pre-populated with the required packages. Populating only your main dbt project’s

dbt_packages

will not prevent the CLI from attempting to download packages.