I'm keen to use a Github Action to run Elementary ...
# ask-elementary-ai
n
I'm keen to use a Github Action to run Elementary in prod, I have an elementary profile exposed in my profiles.yml but I'm struggling to expose the value in the variable as I cannot link to the file as per the docs. Any suggestions on how to tweak the action to have this work?
k
A message from kapa.ai
New message text here
n
For 3, where is the reference profiles.yml to become the Github output? @kapa.ai
k
A message from kapa.ai
For Option 3 (generating profiles.yml content dynamically), the profiles.yml content isn't referenced from an existing file - it's actually generated within the GitHub workflow itself. Let me clarify how this works: In the example I shared, the
yaml_content
variable would contain the full profiles.yml content that you define directly in your workflow. Here's a more complete example showing how this is implemented:
Copy code
yaml - name: Generate YAML   id: generate_yaml   run: |     # Generate the YAML content     yaml_content="elementary:   outputs:     default:       type: 'snowflake'       account: '${{ secrets.ELEMENTARY_SNOWFLAKE_ACCOUNT }}'       user: '${{ secrets.ELEMENTARY_SNOWFLAKE_USER }}'       password: '${{ secrets.ELEMENTARY_SNOWFLAKE_PASSWORD }}'       role: '${{ secrets.ELEMENTARY_SNOWFLAKE_ROLE }}'       warehouse: '${{ secrets.ELEMENTARY_SNOWFLAKE_WH }}'       database: '${{ secrets.ELEMENTARY_SNOWFLAKE_DB }}'       schema: '${{ secrets.ELEMENTARY_SNOWFLAKE_SCHEMA }}'"      # Set the delimiter for heredoc     delimiter="$(openssl rand -hex 8)"          # Output the YAML content to GITHUB_OUTPUT     {       echo "profile_yml<<${delimiter}"       echo -e "$yaml_content"       echo "${delimiter}"     } >> $GITHUB_OUTPUT
This approach constructs the profiles.yml content directly in the workflow using GitHub secrets for sensitive information [Link to Slack discussion](https://slack.com/archives/C02CTC89LAX/p1682351403423809). The content is then passed to the Elementary action using the output variable. The benefit of this approach is that you don't need to store the entire profiles.yml as a single secret - you can use individual secrets for each credential and construct the file dynamically. Would you like me to provide more details on any specific part of this implementation?
n
I have the contents of the profiles.yml already available in the root directory so i would prefer to reference that somehow in the github action rather than maintaining a separate variable. It has references to env variables so ideally i want it to produce an output i can feed into the runner and save myself having to modify secrets later on. Is this possible @kapa.ai?
k
A message from kapa.ai
Yes, you can definitely use your existing profiles.yml file from the root directory in the GitHub Action without maintaining a separate variable. This is actually a supported approach for the Elementary GitHub Action. If your profiles.yml already exists in your project and uses environment variables, you can simply pass those environment variables directly to the action. Here's how you can do it:
Copy code
yaml - name: Checkout dbt project   uses: actions/checkout@v3  - name: Run Elementary   uses: elementary-data/run-elementary-action@v1.12   env:     # Add all environment variables referenced in your profiles.yml     PROFILES_YML_SNOWFLAKE_ACCOUNT: ${{ secrets.PROFILES_YML_SNOWFLAKE_ACCOUNT }}     PROFILES_YML_SNOWFLAKE_USER: ${{ secrets.PROFILES_YML_SNOWFLAKE_USER }}     PROFILES_YML_SNOWFLAKE_PASSWORD: ${{ secrets.PROFILES_YML_SNOWFLAKE_PASSWORD }}     # Add any other environment variables your profiles.yml needs   with:     warehouse-type: snowflake # or your warehouse type     project-dir: ./ # Points to the root directory where profiles.yml is located     edr-command: edr monitor # or any other edr command you need
The Elementary GitHub Action will automatically detect and use the profiles.yml in your project directory [Link to Slack discussion](https://slack.com/archives/C02CTC89LAX/p1675423462160059). Just make sure that: 1. Your profiles.yml contains an
elementary
profile section 2. All environment variables referenced in your profiles.yml are passed to the action 3. You specify the
project-dir
parameter if your profiles.yml is not in the default location This approach is ideal for your situation as it allows you to: - Use your existing profiles.yml file - Keep using environment variables for credentials - Avoid duplicating configuration in GitHub secrets - Easily update your profiles.yml without changing GitHub secrets Let me know if you need any clarification or have additional questions!
n
The link to slack discussion doesn't work and i believe i need to provide the profiles_yml variable in the github action for this to work @kapa.ai?
k
A message from kapa.ai
I apologize for the confusion with the Slack discussion links. You're right that they don't work as they're internal references from the knowledge sources. Regarding your question about the profiles.yml variable - there are two approaches you can take with the Elementary GitHub Action: 1. Using an existing profiles.yml in your project: If your profiles.yml is in your project repository and contains environment variables, you don't need to explicitly provide the profiles-yml parameter. Instead, you need to:
Copy code
yaml    - name: Checkout dbt project      uses: actions/checkout@v3     - name: Run Elementary      uses: elementary-data/run-elementary-action@v1.12      env:        # All environment variables referenced in your profiles.yml        VARIABLE_1: ${{ secrets.VARIABLE_1 }}        VARIABLE_2: ${{ secrets.VARIABLE_2 }}      with:        warehouse-type: snowflake # or your warehouse type        project-dir: ./ # Points to root directory with profiles.yml        edr-command: edr monitor # or other command
2. Explicitly providing profiles.yml content: Based on multiple user experiences in the knowledge sources, it seems that in many cases, you do need to explicitly provide the profiles-yml parameter for the GitHub Action to work properly, even if you have a profiles.yml file in your project:
Copy code
yaml    - name: Run Elementary      uses: elementary-data/run-elementary-action@v1.12      with:        warehouse-type: snowflake        profiles-yml: ${{ secrets.ELEMENTARY_PROFILES_YML }}        edr-command: edr monitor
From the knowledge sources, it appears that many users had success when explicitly providing the profiles-yml parameter, so this might be the more reliable approach. If you're still experiencing issues, you could try adding a step to read your existing profiles.yml file and pass its content to the action:
Copy code
yaml - name: Read profiles.yml   id: read_profiles   run: |     PROFILES_CONTENT=$(cat profiles.yml)     delimiter="EOF"     echo "profiles_content<<$delimiter" >> $GITHUB_OUTPUT     echo "$PROFILES_CONTENT" >> $GITHUB_OUTPUT     echo "$delimiter" >> $GITHUB_OUTPUT  - name: Run Elementary   uses: elementary-data/run-elementary-action@v1.12   env:     # Environment variables for your profiles.yml     VARIABLE_1: ${{ secrets.VARIABLE_1 }}     VARIABLE_2: ${{ secrets.VARIABLE_2 }}   with:     warehouse-type: snowflake     profiles-yml: ${{ steps.read_profiles.outputs.profiles_content }}     edr-command: edr monitor
This approach reads your existing profiles.yml file and passes its content to the action while still allowing environment variables to be resolved.
n
Is the order of edr commands monitor, report, send-report? @kapa.ai
k
A message from kapa.ai
Based on the knowledge sources, there isn't a strict required order for running the Elementary Data Reliability (EDR) CLI commands. Each command serves a different purpose: 1.
edr monitor
- Reads from the test results table and sends alerts for new failures (e.g., to Slack) 2.
edr report
- Generates the Elementary UI report locally 3.
edr send-report
- Generates the report and sends it to external platforms (Slack, S3, GCS, etc.) The commands can be run independently based on your needs. However, in GitHub Actions and production workflows, a common pattern is to run them in the sequence you mentioned:
Copy code
edr monitor edr report edr send-report
It's worth noting that
edr send-report
also generates a report, so running
edr report
before it is not strictly necessary. As mentioned in one of the knowledge sources: "send report also generates a report so no need to run both" [](https://slack.com/archives/C02CTC89LAX/p1708088624243309). If you're setting up a GitHub Action workflow, you can run these commands together as shown in many examples:
Copy code
edr monitor   --slack-token "..."   --slack-channel-name "..." && edr report   --file-path "report.html" && edr send-report   --slack-token "..."   --slack-channel-name "..."   # Additional parameters for S3/GCS/etc.
This sequence ensures you get alerts for failures, generate a local report, and share the report through your preferred channels. Is there a specific workflow or deployment scenario you're setting up with Elementary? I'm here to help you with any other questions about Elementary Data. Markdown formatting fixed.