Hi There, I am on dbt cloud and trying to set up g...
# community-support
k
Hi There, I am on dbt cloud and trying to set up github actions for elementary. I am not sure from where i should start? I already installed the package and able to generate report locally. Any help/advice is appreciated. Cheers!
Can i achieve this in community version?
@Maayan SalomHey @Maayan Salom Can you share your thoughts on this?
d
Do you have experience with Github actions? It should be not very complex to set it up. In actions you define platform you want to use, probably ubuntu:latest is the best choice for elementary. Check also if Elementary published their own Github action. Then you define a set of commands to execute: like
Copy code
pip install ...
dbt deps
edr report ...
#or
edr monitor ...
k
i have little context of github actions, generally our dev ops team do this for us. i found this doc on the website, I have defined profiles.yml with elementary connection, I want to run the workflow with only complete build job and sent report to a slack channel.
Hey @Denis Kokorin Would you be able to help me out here. I have updated the workflow but i am seeing errors. Here is my workflow:
Copy code
name: Run Elementary
on:
  # Run the action when a push to the main branch of the repository is made.
  push:
    branches: [ "main", "master" ]
  # Run the action when a pull request to the main branch is opened.
  pull_request:
    branches: [ "main", "master" ]
  # Run the action in a scheduled manner every hour.
  schedule:
    - cron: '0 9,13,19 * * 1-5'
  # Allows you to run this workflow manually from the Actions tab.
  workflow_dispatch:
env:
  SNOWFLAKE_ACCOUNT: ${{ secrets.GITHUB_TOKEN }}
  SNOWFLAKE_USER: ${{ secrets.SNOWFLAKE_USER }}
  SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD }}
  SNOWFLAKE_DB: ${{ secrets.SNOWFLAKE_DB }}
  SNOWFLAKE_WAREHOUSE: ${{ secrets.SNOWFLAKE_WAREHOUSE }}

jobs:
  elementary:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - name: Checkout dbt project
        uses: actions/checkout@v3
      - name: Run Elementary
        uses: elementary-data/run-elementary-action@v1.11
        with:
          warehouse-type: snowflake # Type of warehouse to use (bigquery, snowflake, redshift, etc.)
          adapter-version: 1.5.0 # The dbt-adapter version that should be used (If not provided the latest version will be used).
          profiles-yml: ${{ secrets.elementary }} # Content of ~/.dbt/profiles.yml, should have an `elementary` profile.
          edr-command:
            edr monitor
              --slack-token ${{ secrets.elementary_slack_token }}
              --slack-channel-name "tech-alerts-data-issues"
            &&
            edr report
              --file-path "report.html"
            &&
            edr send-report
              --slack-token ${{ secrets.elementary_slack_token }}
              --slack-channel-name "tech-alerts-data-issues"
      - name: Upload report
        uses: actions/upload-artifact@v3
        with:
          name: report.html
          path: report.html
      - name: Upload log
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: edr.log
          path: edr.log
This is the error:
Copy code
2024-06-05 13:07:54,200 Failed to get Elementary dbt package version: {'returncode': 1, 'cmd': ['dbt', '--log-format', 'json', 'run-operation', 'get_elementary_dbt_pkg_version', '--project-dir', '/edr_stager_dbt_project'], 'output': b'{"data": {"log_version": 3, "version": "=1.5.11"}, "info": {"category": "", "code": "A001", "extra": {}, "invocation_id": "0e01a258-16fb-43ea-9611-1657f3a6676a", "level": "info", "msg": "Running with dbt=1.5.11", "name": "MainReportVersion", "pid": 53, "thread": "MainThread", "ts": "2024-06-05T13:07:52.302819Z"}}\n{"data": {"profile_name": "elementary", "target_name": "default"}, "info": {"category": "", "code": "A005", "extra": {}, "invocation_id": "0e01a258-16fb-43ea-9611-1657f3a6676a", "level": "info", "msg": "target not specified in profile \'elementary\', using \'default\'", "name": "MissingProfileTarget", "pid": 53, "thread": "MainThread", "ts": "2024-06-05T13:07:52.307216Z"}}\n{"data": {"adapter_name": "snowflake", "adapter_version": "=1.5.0"}, "info": {"category": "", "c
Traceback (most recent call last):
  File "/entrypoint.py", line 148, in <module>
    main()
  File "/entrypoint.py", line 139, in main
    install_edr(args.adapter, args.project_dir, args.profile_target)
  File "/entrypoint.py", line 62, in install_edr
    command_results = subprocess.run(
  File "/usr/local/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['dbt', '--log-format', 'json', 'run-operation', 'get_elementary_dbt_pkg_version', '--project-dir', '/edr_stager_dbt_project']' returned non-zero exit status 1.
d
may be this:
"target not specified in profile \'elementary\', using \'default\'", "name": "MissingProfileTarget"
👀 1
k
Makes sense, Dumb questions do we need artifacts package install in order to have the github actions run?
and we are ignoring packages file right now from the repo, is it also required?
d
Elementary needs to install some packages for any EDR command
k
I am seeing this error now:
Copy code
2024-06-05 13:17:44,393 Failed to get Elementary dbt package version: {'returncode': 1, 'cmd': ['dbt', '--log-format', 'json', 'run-operation', 'get_elementary_dbt_pkg_version', '--project-dir', '/edr_stager_dbt_project'], 'output': b'{"data": {"log_version": 3, "version": "=1.5.11"}, "info": {"category": "", "code": "A001", "extra": {}, "invocation_id": "0f64e51d-98d8-4b57-a8f1-c2df71f4e659", "level": "info", "msg": "Running with dbt=1.5.11", "name": "MainReportVersion", "pid": 54, "thread": "MainThread", "ts": "2024-06-05T13:17:42.500196Z"}}\n{"data": {"adapter_name": "snowflake", "adapter_version": "=1.5.0"}, "info": {"category": "", "code": "E034", "extra": {}, "invocation_id": "0f64e51d-98d8-4b57-a8f1-c2df71f4e659", "level": "info", "msg": "Registered adapter: snowflake=1.5.0", "name": "AdapterRegistered", "pid": 54, "thread": "MainThread", "ts": "2024-06-05T13:17:42.805355Z"}}\n{"data": {"reason": "saved manifest not found. Starting full parse."}, "info": {"category": "", "code": "I024", "extra": {}, "i
Traceback (most recent call last):
  File "/entrypoint.py", line 148, in <module>
    main()
  File "/entrypoint.py", line 139, in main
    install_edr(args.adapter, args.project_dir, args.profile_target)
  File "/entrypoint.py", line 62, in install_edr
    command_results = subprocess.run(
  File "/usr/local/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['dbt', '--log-format', 'json', 'run-operation', 'get_elementary_dbt_pkg_version', '--project-dir', '/edr_stager_dbt_project']' returned non-zero exit status 1.
Elementary needs to install some packages for any EDR command
I would need to give permissions to write packages in this case
@Denis Kokorin You were an amazing brainstorming buddy, Thank you for your help. I think i am in a right direction now, able to make the correction. Need to spend some more time troubleshooting. thankyou
😎 1