<@U088WCP3B4P> Can we run tests only on the basis ...
# ask-elementary-ai
v
@kapa.ai Can we run tests only on the basis lineage flow and its connecting models. How to configure it and run it as a group
k
A message from kapa.ai
Based on the available knowledge sources, there is no direct, built-in way in Elementary (or dbt) to automatically select and run tests only on models that are part of a specific lineage flow or on models connected in the lineage graph. This is primarily due to dbt's current limitations: test definitions are attached to individual models or columns, and there is no native mechanism to define or execute tests based on lineage relationships or groups of connected models. However, Elementary Cloud provides advanced lineage features, including column-level lineage and the ability to analyze upstream and downstream impacts, which can help you visualize and understand the flow of data and dependencies between models. This can support workflows such as impact analysis and prioritizing issues for critical downstream assets, but it does not directly translate to running tests as a group based on lineage. You can use the lineage view to identify which models are connected, but you would still need to manually configure tests for those models or use dbt selectors to run tests on a set of models you specify by name or tag. There is no mention of a feature that allows you to select models for testing purely based on their lineage connections in an automated way [End-to-End Data Lineage](https://docs.elementary-data.com/features/data-lineage/lineage). If you want to run tests as a group, you can use dbt's tagging and selection features (e.g.,
dbt test --select tag:my_tag
), but you would need to manually tag the relevant models. Some users have suggested writing scripts to automate the generation of test YAMLs or to select models based on custom logic, but this is outside the scope of built-in Elementary or dbt functionality [Slack discussion on bulk test configuration](https://slack.com/archives/C02CTC89LAX/p1689113314530069). In summary: - There is no built-in way to run tests solely based on lineage flow and its connecting models. - You can use the lineage view in Elementary Cloud to identify relevant models, but test configuration and execution must still be managed manually or via dbt selectors/tags. - Bulk test configuration based on lineage is not currently supported in open source dbt or Elementary. If you need this feature, you may want to consider submitting a feature request to the Elementary team.
v
@kapa.ai can we integrate edr report generatio and elementary tests with jenkins
k
A message from kapa.ai
Yes, you can integrate Elementary (edr) report generation and Elementary tests with Jenkins. Elementary collects orchestrator information (like job IDs) at runtime and saves it to the dbt_invocations table. For Jenkins, you can access environment variables from the dbt run, and this information can be passed to Elementary. For example, you can pass Jenkins environment variables as dbt variables when running your dbt commands, which Elementary will then capture:
Copy code
dbt run --vars '{"orchestrator": "Jenkins", "job_name": "your_jenkins_job_name"}'
To automate the process, you would typically: 1. Run your dbt models and tests in Jenkins as part of your pipeline. 2. After the dbt run/test steps, execute the
edr
CLI commands (such as
edr monitor
or
edr report
) in your Jenkins job to generate the report and collect test results. You may need to ensure that the necessary environment variables from Jenkins are available to your dbt/Elementary process, and you can customize the macros in Elementary to capture additional Jenkins-specific metadata if needed. The integration is flexible and relies on standard CI/CD practices for running shell commands and passing environment variables [Slack discussion on Jenkins integration](https://slack.com/archives/C02CTC89LAX/p1693381415910399). In summary: - Yes, integration is possible and commonly done. - Use Jenkins to orchestrate dbt and Elementary CLI commands. - Pass Jenkins job metadata as variables if you want them tracked in Elementary. Let me know if you need a sample Jenkins pipeline script!