is there a way, for the same BQ project, to unders...
# ask-elementary-ai
i
is there a way, for the same BQ project, to understand where an invocation comes from using elementary artifacts?
k
A message from kapa.ai
Yes, you can understand where an invocation comes from in the same BigQuery project using Elementary artifacts. The key table for this is the
dbt_invocations
table, which stores metadata about each dbt run (invocation). This table includes columns such as
JOB_ID
,
JOB_NAME
,
JOB_RUN_ID
,
COMMAND
,
DBT_VERSION
,
ELEMENTARY_VERSION
,
TARGET_NAME
,
TARGET_DATABASE
,
TARGET_SCHEMA
,
PROJECT_NAME
,
ENV
,
ORCHESTRATOR
, and more. These fields can help you trace the origin of each invocation, including which orchestrator (like Airflow, Jenkins, etc.) triggered the run, the environment, and other contextual details. If you want to enrich this further (for example, to include Jenkins job IDs), you can pass orchestrator-specific environment variables or dbt variables and modify the macros Elementary uses to collect this information. This allows you to capture and store additional context about each invocation directly in the
dbt_invocations
table, making it easier to trace the source of each run within your BigQuery project. For Jenkins, for example, you can map Jenkins environment variables to the fields Elementary collects by editing the relevant macros or passing them as dbt variables during the run. This approach is described in more detail in the community discussions and documentation: - The
dbt_invocations
table structure and its fields: [dbt artifacts models](https://docs.elementary-data.com/dbt/dbt-artifacts#dbt-artifacts-models) - How to pass orchestrator/job info to Elementary: [Slack discussion on Jenkins integration](https://slack.com/archives/C02CTC89LAX/p1693381415910399) So, by querying the
dbt_invocations
table and ensuring you are populating the relevant fields, you can reliably determine where each invocation originated within the same BQ project.
i
@kapa.ai how can I use for example ORCHESTRATOR to pass an env var if the env var exists?
k
A message from kapa.ai
You can pass the ORCHESTRATOR environment variable to Elementary by setting it in your environment before running your dbt command. Elementary will automatically collect job metadata from environment variables if they exist. For example, to set the orchestrator as "airflow", you would run:
Copy code
bash ORCHESTRATOR=airflow dbt run
Alternatively, you can pass the orchestrator information as a dbt variable, which will take precedence over the environment variable:
Copy code
bash dbt run --vars '{"orchestrator": "airflow"}'
Elementary will capture this information and store it in the
dbt_invocations
table, allowing you to track the source of each invocation. This approach works for any supported orchestrator or custom value you want to pass. For more details, see the official documentation: [Collect jobs info from orchestrator](https://docs.elementary-data.com/oss/guides/collect-job-data)<source>https://docs.elementary-data.com/oss/guides/collect-job-data&lt;/source>.