https://linen.dev logo
Join Slack
Powered by
Can you help me with this issue <@U088WCP3B4P> <ht...
# ask-elementary-ai
n
k
A message from kapa.ai
I don't have direct access to GitHub issues, but based on the knowledge sources provided, I can help with common issues related to the Elementary GitHub Action. From the context in the knowledge sources, there have been several issues with the 
run-elementary-action
GitHub Action. Let me address some common problems and solutions: ## Common Issues with Elementary GitHub Action ### Version Compatibility Issues Several users have reported issues with version compatibility. For example: 1. When using newer versions of Elementary CLI (edr), there can be compatibility issues with the dbt package version. One user fixed their issue by pinning the Elementary CLI version to 0.16.1 when they were experiencing errors with 0.16.2 [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1739810287607029). 2. Another user fixed their issue by updating 
dbt-bigquery
to version 1.5.9 and changing the 
adapter-version
input in the action [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1711591268099699). ### Specifying Adapter Version If you're experiencing issues with a specific dbt adapter version, you can specify it in your workflow:
Copy code
yaml - name: Run Elementary   uses: elementary-data/run-elementary-action@v1.11   with:     warehouse-type: your_warehouse_type     adapter-version: 1.5.9  # Specify your required version here     profiles-yml: ${{ secrets.ELEMENTARY_PROFILES_YML }}     # other parameters...
This feature was added specifically to address version compatibility issues [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1685116355127589). ### Organization Restrictions on External Actions If you're in a private repository within an organization, you might encounter restrictions on using external actions. A workaround is to fork the Elementary action into your organization's GitHub account [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1680524725868009). ### Command Formatting Issues Make sure your commands in the 
edr-command
section are properly formatted with 
&&
between different commands [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1677164206225519). If none of these solutions address your specific issue, please provide more details about the error you're encountering, and I can try to provide more targeted assistance. I'm here to help you troubleshoot your Elementary GitHub Action issues!
n
Here is the issue @kapa.ai : This keeps failing for me when trying to run it in prod. It seems to be related to not finding the correct profile despite me specifying it. Any support would be much appreciated! Details below: Errors from Gitub Action under the 
Run Elementary
step:
Copy code
Notice:  A new release of pip is available: 23.0.1 -> 25.0.1
Notice:  To update, run: pip install --upgrade pip
2025-03-17 08:42:36,862 Setting up the environment.
2025-03-17 08:42:36,863 Getting Elementary *** package version.
2025-03-17 08:42:38,883 Failed to get Elementary *** package version: ***'returncode': 2, 'cmd': ['***', '--log-format', 'json', 'run-operation', 'get_elementary_***_pkg_version', '--project-dir', '/edr_stager_***_project'], 'output': b'***"data": ***"log_version": 3, "version": "=1.9.3"***, "info": ***"category": "", "code": "A001", "extra": ***, "invocation_id": "ff562d90-79f1-4810-92db-7fc06bc1ef2a", "level": "info", "msg": "Running with ***=1.9.3", "name": "MainReportVersion", "pid": 63, "thread": "MainThread", "ts": "2025-03-17T08:42:38.389120Z"***\n***"data": ***"profile_name": "elementary", "target_name": "default"***, "info": ***"category": "", "code": "A005", "extra": ***, "invocation_id": "ff562d90-79f1-4810-92db-7fc06bc1ef2a", "level": "info", "msg": "target not specified in profile \'elementary\', using \'default\'", "name": "MissingProfileTarget", "pid": 63, "thread": "MainThread", "ts": "2025-03-17T08:42:38.394143Z"***\n***"data": ***"exc": "Runtime Error\\n  The profile \'elementary\' does not have a target named \'default\'. The valid target names for this profile are:\\n   - dev\\n   - prod"***, "info": ***"category": "", "code": "Z002", "extra": ***, "invocation_id": "ff562d90-79f1-4810-92db-7fc06bc1ef2a", "level": "error", "msg": "Encountered an error:\\nRuntime Error\\n  The profile \'elementary\' does not have a target named \'default\'. The valid target names for this profile are:\\n   - dev\\n   - prod", "name": "MainEncounteredError", "pid": 63, "thread": "MainThread", "ts": "2025-03-17T08:42:38.394959Z"***\n', 'stderr': b''***
Traceback (most recent call last):
  File "/entrypoint.py", line 158, in <module>
    main()
  File "/entrypoint.py", line 149, in main
    install_edr(args.adapter, args.project_dir, args.profile_target)
  File "/entrypoint.py", line 65, in install_edr
    command_results = subprocess.run(
  File "/usr/local/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['***', '--log-format', 'json', 'run-operation', 'get_elementary_***_pkg_version', '--project-dir', '/edr_stager_***_project']' returned non-zero exit status 2.
Github Workflow file:
Copy code
name: schedule_and_push

on:
  schedule:
    - cron:  '30 9 * * *'
  push:
    branches:
      - main

env:
  DBT_PROFILES_DIR: ./

  DBT_GOOGLE_PROJECT_PROD: ${{ secrets.DBT_GOOGLE_PROJECT_PROD }}
  DBT_GOOGLE_BIGQUERY_DATASET_PROD: ${{ secrets.DBT_GOOGLE_BIGQUERY_DATASET_PROD }}
  # The DBT_GOOGLE_BIGQUERY_KEYFILE_PROD secret will be written to a json file below
  DBT_GOOGLE_BIGQUERY_KEYFILE_PROD: ./dbt-service-account.json

jobs:
  scheduled_run:
    name: scheduled_run
    runs-on: ubuntu-latest

    steps:
      - name: Check out
        uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12.2"

      - name: Authenticate using service account
        shell: bash
        env:
          KEYFILE: ${{secrets.DBT_GOOGLE_BIGQUERY_KEYFILE_PROD}}
        run: |
          echo "$KEYFILE" > ./dbt-service-account.json
      - name: Configure & Install dependencies
        run: |
          python -m pip install --upgrade pip-tools pip wheel
          python -m pip install setuptools==71.1.0 poetry==1.8.3
          pip install pyarrow dbt-core dbt-bigquery elementary_data
          dbt deps --target prod
      - name: Seed dbt
        run: dbt seed --target prod

      # - name: Run dbt models #Commented out until elementary works
      #   run: dbt run --target prod
      
      - name: Test dbt models
        id: dbt_test
        continue-on-error: true
        run: |
          if dbt test --target prod; then
            echo "DBT tests passed"
            echo "test_status=success" >> $GITHUB_OUTPUT
          else
            echo "DBT tests failed"
            echo "test_status=failed" >> $GITHUB_OUTPUT
          fi
      
      # - name: Read profiles.yml
      #   id: profiles
      #   run: |
      #     content=$(cat profiles.yml)
      #     content="${content//'%'/'%25'}"
      #     content="${content//$'\n'/'%0A'}"
      #     content="${content//$'\r'/'%0D'}"
      #     echo "content=$content" >> $GITHUB_OUTPUT
      
      - name: Run Elementary
        if: success() || steps.dbt_test.outcome == 'failure'
        uses: elementary-data/run-elementary-action@v1.12
        with:
          warehouse-type: bigquery
          edr-command: |
            edr send-report 
              --profiles-dir .
              --profile-target prod 
              --env prod 
              --slack-token "<https://hooks.slack.com/services/X/Y/Z>" 
              --google-service-account-path dbt-service-account.json 
              --gcs-bucket-name XXX
              --bucket-file-path report.html
              --report-url "<https://storage.cloud.google.com/XXX/report.html>"
            &&
            edr monitor 
              --report-url "<https://storage.cloud.google.com/XXX/report.html>" 
              --slack-webhook "<https://hooks.slack.com/services/X/Y/Z>" 
              --profile-target prod 
              --profiles-dir .
              --env prod 
              --group-by table 
              --suppression-interval 0
          bigquery-keyfile: ${{ secrets.DBT_GOOGLE_BIGQUERY_KEYFILE_PROD }}
          gcs-keyfile: ${{ secrets.DBT_GOOGLE_BIGQUERY_KEYFILE_PROD }}
profiles.yml
which is in the main root of the DBT project:
Copy code
dbt:
  target: dev
  outputs:
    dev:
      type: bigquery
      method: service-account
      keyfile: "{{ env_var('DBT_GOOGLE_BIGQUERY_KEYFILE_DEV') }}"
      project: "{{ env_var('DBT_GOOGLE_PROJECT_DEV') }}"
      dataset: "{{ env_var('DBT_GOOGLE_BIGQUERY_DATASET_DEV') }}"
      threads: 32
      location: EU
      timeout_seconds: 300
      priority: interactive

    prod:
      type: bigquery
      method: service-account
      keyfile: "{{ env_var('DBT_GOOGLE_BIGQUERY_KEYFILE_PROD') }}"
      project: "{{ env_var('DBT_GOOGLE_PROJECT_PROD') }}"
      dataset: "{{ env_var('DBT_GOOGLE_BIGQUERY_DATASET_PROD') }}"
      threads: 32
      location: EU
      timeout_seconds: 300
      priority: interactive

elementary:
  outputs:
    dev:
      type: bigquery
      method: service-account
      keyfile: "{{ env_var('DBT_GOOGLE_BIGQUERY_KEYFILE_DEV') }}"
      project: "{{ env_var('DBT_GOOGLE_PROJECT_DEV') }}"
      dataset: "{{ env_var('DBT_GOOGLE_BIGQUERY_DATASET_DEV') }}_elementary"
      threads: 32
      location: EU
      priority: interactive

    prod:
      type: bigquery
      method: service-account
      keyfile: "{{ env_var('DBT_GOOGLE_BIGQUERY_KEYFILE_PROD') }}"
      project: "{{ env_var('DBT_GOOGLE_PROJECT_PROD') }}"
      dataset: "{{ env_var('DBT_GOOGLE_BIGQUERY_DATASET_PROD') }}_elementary"
      threads: 32
      location: EU
      priority: interactive
k
A message from kapa.ai
I can see the issue in your GitHub Action configuration. The error message indicates that Elementary is trying to use a 'default' target in the 'elementary' profile, but your profiles.yml only has 'dev' and 'prod' targets defined. The specific error is:
Copy code
The profile 'elementary' does not have a target named 'default'. The valid target names for this profile are:    - dev    - prod
Based on the knowledge sources, there are two ways to fix this: 1. Add a default target to your elementary profile in profiles.yml:
Copy code
yaml elementary:   target: prod  # Add this line   outputs:     dev:       # your dev config     prod:       # your prod config
This solution was suggested in a similar case where a user was facing the same issue [in this Slack discussion](https://slack.com/archives/C02CTC89LAX/p1699363455840319). 2. Alternatively, make sure that the target is correctly specified in all your edr commands. In your workflow, you're using 
--profile-target prod
, which is correct, but there might be an issue with how the Elementary GitHub Action is processing this parameter. Try modifying your edr-command to ensure the target is consistently specified:
Copy code
yaml edr-command: |   edr send-report      -t prod  # Use -t as a shorthand for --profile-target     --profiles-dir .     --env prod      # rest of your parameters   &&   edr monitor      -t prod  # Use -t as a shorthand for --profile-target     --profiles-dir .     --env prod      # rest of your parameters
The first solution (adding 
target: prod
to your elementary profile) is the most straightforward and has been confirmed to work in similar situations. Note that the 
--env prod
parameter is only cosmetic and affects a field in the report/alerts, while 
-t prod
or 
--profile-target prod
is what actually determines which connection profile to use [as mentioned in this discussion](https://slack.com/archives/C02CTC89LAX/p1699363455840319). Let me know if you need any further clarification!
n
@kapa.ai if I'm specifying the target in my 
edr
command why would I need to also set up the 
target
default?
k
A message from kapa.ai
Based on the knowledge sources, even when you specify the target in your 
edr
command using 
--profile-target prod
or 
-t prod
, there's a specific reason why you might still need to set a default target in your profiles.yml. The issue relates to how Elementary initializes and processes profiles. When Elementary runs, it first tries to load the profile before applying the target parameter. If no default target is specified in the profile itself, it attempts to use a target named "default", which in your case doesn't exist. As explained in one of the Slack discussions: > "This requires an addition on our end, I'll do it soon. However, for the time being, you can define 'prod' as the default target of elementary. You can do so by adding 'target: prod' to the profile." [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1672333817993259) This is a known behavior in the Elementary CLI. When the tool starts up, it first tries to load the profile configuration before it processes command-line arguments like 
--profile-target
. If there's no default target specified in the profile itself, it looks for a target named "default" which doesn't exist in your configuration. Another user experienced the same issue and was advised: > "So you have two options to handle prod and dev projects: 1. single profile with different targets... And then pass the relevant target for your dbt / elementary runs" [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1691655066320649) This is why adding 
target: prod
to your elementary profile in profiles.yml resolves the issue - it provides a fallback target during the initial loading phase, before your command-line parameters are fully processed. It's a bit counterintuitive, but it's how the current implementation works. The Elementary team has acknowledged this as something they might improve in the future. Would you like me to explain any other aspects of how Elementary handles profiles and targets?
n
hey <!subteam^S08HYBWCF6E>, I'm really struggling with this one. I can't seem to get the Github Action to work and run, it works perfectly on local and just fails in Github. I've tried using the actions, and also tried doing it without them:
Copy code
name: schedule_and_push

on:
  schedule:
    - cron:  '30 9 * * *'
  push:
    branches:
      - main

env:
  DBT_PROFILES_DIR: ./

  DBT_GOOGLE_PROJECT_PROD: ${{ secrets.DBT_GOOGLE_PROJECT_PROD }}
  DBT_GOOGLE_BIGQUERY_DATASET_PROD: ${{ secrets.DBT_GOOGLE_BIGQUERY_DATASET_PROD }}
  # The DBT_GOOGLE_BIGQUERY_KEYFILE_PROD secret will be written to a json file below
  DBT_GOOGLE_BIGQUERY_KEYFILE_PROD: ./dbt-service-account.json

jobs:
  scheduled_run:
    name: scheduled_run
    runs-on: ubuntu-latest

    steps:
      - name: Check out
        uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12.2"

      - name: Authenticate using service account
        shell: bash
        env:
          KEYFILE: ${{secrets.DBT_GOOGLE_BIGQUERY_KEYFILE_PROD}}
        run: |
          echo "$KEYFILE" > ./dbt-service-account.json
      - name: Configure & Install dependencies
        run: |
          python -m pip install --upgrade pip-tools pip wheel
          python -m pip install setuptools==71.1.0 poetry==1.8.3
          pip install pyarrow dbt-core dbt-bigquery elementary_data
          dbt deps --target prod
      - name: Seed dbt
        run: dbt seed --target prod

      # - name: Run dbt models
      #   run: dbt run --target prod
      
      # - name: Test dbt models
      #   id: dbt_test
      #   continue-on-error: true
      #   run: |
      #     if dbt test --target prod; then
      #       echo "DBT tests passed"
      #       echo "test_status=success" >> $GITHUB_OUTPUT
      #     else
      #       echo "DBT tests failed"
      #       echo "test_status=failed" >> $GITHUB_OUTPUT
      #     fi
      
      - name: Run Elementary
        # if: success() || steps.dbt_test.outcome == 'failure'
        #uses: elementary-data/run-elementary-action@v1.12
        run: |
          echo "dbt service account output start"
          tail -50 dbt-service-account.json
          echo "dbt service account output end"
          edr monitor --report-url "<https://storage.cloud.google.com/elementary_data/report.html>" --slack-webhook "<https://hooks.slack.com/services/X/Y/Z>"  --project-profile-target dbt_abelini --profile-target prod --profiles-dir . --project-dir . --env prod --group-by table --suppression-interval 0
        #edr send-report --profiles-dir ./ --profile-target prod --env prod --slack-token "<https://hooks.slack.com/services/X/Y/Z>" --google-service-account-path ./dbt-service-account.json --gcs-bucket-name elementary_data --bucket-file-path report.html --report-url "<https://storage.cloud.google.com/elementary_data/report.html>"
        #bigquery-keyfile: ${{ secrets.DBT_GOOGLE_BIGQUERY_KEYFILE_PROD }}
        #gcs-keyfile: ${{ secrets.DBT_GOOGLE_BIGQUERY_KEYFILE_PROD }}
I still get an error (added in log file). Any idea on how I can get this to run?
github actions output.txt
👀 1
(I commented out testing and runs for a period just to see if it would work - even with them enabled it fails)
j
Hey! It seems this is more a Github Actions problem, where the runner is unable to find the keyfile (
[Errno 2] No such file or directory: './***-service-account.json'"***
) Maybe you can try something like:
Copy code
- name: Create google keyfile
      env:
        PROJECT_ID: ${{ secrets.PROJECT_ID }}
        PRIVATE_KEY_ID: ${{ secrets.PRIVATE_KEY_ID }}
        PRIVATE_KEY: ${{ secrets.PRIVATE_KEY }}
      run: |
        cat > ~/.dbt-service-account.json <<EOF
        {
          "type": "service_account",
          "project_id": "$PROJECT_ID",
          "private_key_id": "$PRIVATE_KEY",
          "private_key": "$PRIVATE_KEY",
          ..etc..
        }
        EOF
I can look into it a bit more tomorrow
n
Hey Joost, Thanks for this. I've tried this in a previous config using the Github Action runner you guys recommend and I just can't get that to work either, it keeps failing with a different error. I'd love some support as I've spent over 30 hours battling with Elementary in hope to get us up and running with it (which is far longer than I would've wanted to spend), I'd be happy to contribute to the docs and even make tutorials for simple tasks as I do love your mission, it's just a little harder to get up and running!
j
Ok will have a look at it today. Just to make sure, you are able to run 
dbt run
dbt test
etc with the same Github actions file?
n
Hey Joost! Thanks for that. Yes, dbt run and test work flawlessly, I'm just unable to do anything with 
edr
. I'm either using the commands wrong or something isn't right somewhere. I have not setup a profile for elementary in Github as a secret, it's available in the 
profiles.yml
in the main directory (shown in the first message of this thread)
Hey @Joost Boonzajer Flaes, Let me know if you need anything from me, I'm happy to even jump on a short call if that's easier (or send you a loom etc). Thanks,
n
Hey @Nitesh Sharoff, it seems like it's failing because it's looking for a default target in your elementary profile, but your profile only has dev and prod targets. Try this:
Copy code
with:
  warehouse-type: bigquery
  dbt-target: prod   # <--- Add this line
  edr-command: |
    edr send-report 
      --profiles-dir .
      --profile-target prod 
      ...
Let us know if this helps! 🙂
n
Thanks Noya, I'll try this today and confirm if it works. This command is not on the github actions readme and i thought passing the 
--target
in the 
edr
command would serve the same purpose, I'll explicitely try this and revert. Keeping my fingers crossed it's this!
j
hey @Nitesh Sharoff! Sorry only now had some time to look into it a bit. Ok so I can see there should be a 
./dbt-service-account.json
, but it is unable to find it. My first question is if you tried the configuration like this? (from the docs):
Copy code
- name: Run Elementary
        uses: elementary-data/run-elementary-action@v1.11
        with:
          warehouse-type: bigquery # Type of warehouse to use (bigquery, snowflake, redshift, etc.)
          adapter-version: 1.5.0 # The dbt-adapter version that should be used (If not provided the latest version will be used). 
          profiles-yml: ${{ secrets.ELEMENTARY_PROFILES_YML }} # Content of ~/.dbt/profiles.yml, should have an `elementary` profile.
          edr-command:
            edr monitor
              --slack-token "${{ secrets.SLACK_TOKEN }}"
              --slack-channel-name "${{ secrets.SLACK_CHANNEL_NAME }}"
            &&
            edr report
              --file-path "report.html"
            &&
            edr send-report
              --slack-token "${{ secrets.SLACK_TOKEN }}"
              --slack-channel-name "${{ secrets.SLACK_CHANNEL_NAME }}"
              --aws-access-key-id "${{ secrets.AWS_ACCESS_KEY_ID }}"
              --aws-secret-access-key "${{ secrets.AWS_SECRET_ACCESS_KEY }}"
              --s3-bucket-name "${{ secrets.S3_BUCKET_NAME }}"
              --google-service-account-path "/tmp/gcs_keyfile.json"
              --gcs-bucket-name "${{ secrets.GCS_BUCKET_NAME }}"
              --update-bucket-website "true"

          bigquery-keyfile: ${{ secrets.BIGQUERY_KEYFILE }} # If using BigQuery, the content of its keyfile.
          gcs-keyfile: ${{ secrets.GCS_KEYFILE }} # If using GCS, the content of its keyfile.
n
@Noya Offer the 
dbt-target
variable doesn't seem to be allowed (screenshot attached). @Joost Boonzajer Flaes I'm trying to avoid using the 
profiles-yml
variable as it's already exposed in the directory so it feels silly to specify it again as a secret. As well as that, the 
adapter-version
I've excluded so the latest version is provided. The rest looks pretty much the same to me with more specificity on my end. One thing I saw on your code is the use of 
--google-service-account-path "tmp/gcs_keyfile.json"
, is this the default location for the 
gcs-keyfile
specified in the variable and the one you recommend we use? If so, does the 
bigquery-keyfile
also have a default path we can use?
I'm just keen to get up and running with this - I'm frankly a little surprised no-one else is facing these issues as I'm not really doing anything "weird" and my approach is the recommended approach when using Github Actions and DBT and it works perfectly across 20+ clients. My aim was to test Elementary out on one OSS client and then see if the others would benefit from it but I'm really finding it tough to get up and running
i
Hi @Nitesh Sharoff, my name is Itamar, I'm the CTO here in Elementary. Would love to help to get to the bottom of this but this thread is a bit hard to follow 🙂 Can you please clarify: 1. How your 
profiles.yml
file is currently constructed. 2. The current configuration you are using? Are you currently trying to work with the Elementary action or running 
edr
directly? In any case, one thing that pops up to me is that it looks like you are specifying the keyfile json as a relative path (e.g. 
tmp/gcs_keyfile.json
).. This can be problematic, because 
edr
runs 
dbt
in an internal folder (the internal dbt project 
edr
uses), so any relative paths you set will be from there, and not from the local directory your GH action runs in. One thing you can try is to use an absolute path, e.g. 
/tmp/gcs_keyfile.json
. Another option is to provide the 
keyfile_json
setting directly in the YAML file rather than use an external json file (see here).
n
Hey @Itamar Hartstein, Thanks for jumping in. 1. profiles.yml (shared below)
Copy code
dbt_xxxx:
  target: dev
  outputs:
    dev:
      type: bigquery
      method: service-account
      keyfile: "{{ env_var('DBT_GOOGLE_BIGQUERY_KEYFILE_DEV') }}"
      project: "{{ env_var('DBT_GOOGLE_PROJECT_DEV') }}"
      dataset: "{{ env_var('DBT_GOOGLE_BIGQUERY_DATASET_DEV') }}"
      threads: 32
      location: EU
      timeout_seconds: 300
      priority: interactive

    prod:
      type: bigquery
      method: service-account
      keyfile: "{{ env_var('DBT_GOOGLE_BIGQUERY_KEYFILE_PROD') }}"
      project: "{{ env_var('DBT_GOOGLE_PROJECT_PROD') }}"
      dataset: "{{ env_var('DBT_GOOGLE_BIGQUERY_DATASET_PROD') }}"
      threads: 32
      location: EU
      timeout_seconds: 300
      priority: interactive

elementary:
  target: prod
  outputs:
    dev:
      type: bigquery
      method: service-account
      keyfile: "{{ env_var('DBT_GOOGLE_BIGQUERY_KEYFILE_DEV') }}"
      project: "{{ env_var('DBT_GOOGLE_PROJECT_DEV') }}"
      dataset: "{{ env_var('DBT_GOOGLE_BIGQUERY_DATASET_DEV') }}_elementary"
      threads: 32
      location: EU
      priority: interactive

    prod:
      type: bigquery
      method: service-account
      keyfile: "{{ env_var('DBT_GOOGLE_BIGQUERY_KEYFILE_PROD') }}"
      project: "{{ env_var('DBT_GOOGLE_PROJECT_PROD') }}"
      dataset: "{{ env_var('DBT_GOOGLE_BIGQUERY_DATASET_PROD') }}_elementary"
      threads: 32
      location: EU
      priority: interactive
2. I've tried both approaches. I've tried using the Action or running it directly in it's own step - both seem to fail Previously I wasn't trying 
tmp/gcs_keyfile.json
, this is something I tried today after seeing Joost's message. As I've exposed my 
profiles.yml
in the directory, I'm not comfortable exposing my keyfile there as it's a little too public. I'm now trying this without the Action, this is my config but still no luck sadly.
Copy code
name: schedule_and_push

on:
  schedule:
    - cron:  '30 9 * * *'
  push:
    branches:
      - main

env:
  DBT_PROFILES_DIR: ./
  DBT_GOOGLE_PROJECT_PROD: ${{ secrets.DBT_GOOGLE_PROJECT_PROD }}
  DBT_GOOGLE_BIGQUERY_DATASET_PROD: ${{ secrets.DBT_GOOGLE_BIGQUERY_DATASET_PROD }}
  DBT_GOOGLE_BIGQUERY_KEYFILE_PROD: ./dbt-service-account.json

jobs:
  scheduled_run:
    name: scheduled_run
    runs-on: ubuntu-latest

    steps:
      - name: Check out
        uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12.2"

      - name: Authenticate using service account
        shell: bash
        env:
          KEYFILE: ${{secrets.DBT_GOOGLE_BIGQUERY_KEYFILE_PROD}}
        run: |
          echo "$KEYFILE" > ./dbt-service-account.json
      - name: Configure & Install dependencies
        run: |
          python -m pip install --upgrade pip-tools pip wheel
          python -m pip install setuptools==71.1.0 poetry==1.8.3
          pip install pyarrow dbt-core dbt-bigquery elementary_data
          dbt deps --target prod
      - name: Seed dbt
        run: dbt seed --target prod

      # - name: Run dbt models
      #   run: dbt run --target prod
      
      - name: Test dbt models
        id: dbt_test
        continue-on-error: true
        run: |
          if dbt test --target prod; then
            echo "DBT tests passed"
            echo "test_status=success" >> $GITHUB_OUTPUT
          else
            echo "DBT tests failed"
            echo "test_status=failed" >> $GITHUB_OUTPUT
          fi
      
      - name: Run Elementary
        if: success() || steps.dbt_test.outcome == 'failure'
        env:
          DBT_PROFILES_DIR: ./
          DBT_GOOGLE_PROJECT_PROD: ${{ secrets.DBT_GOOGLE_PROJECT_PROD }}
          DBT_GOOGLE_BIGQUERY_DATASET_PROD: ${{ secrets.DBT_GOOGLE_BIGQUERY_DATASET_PROD }}
          KEYFILE: ${{secrets.DBT_GOOGLE_BIGQUERY_KEYFILE_PROD}}
        run: |
          echo "$KEYFILE" > ./dbt-service-account.json
          edr send-report --profiles-dir . --profile-target prod --env prod --slack-token "<https://hooks.slack.com/services/X/Y/Z>" --google-service-account-path ./dbt-service-account.json --gcs-bucket-name elementary_data --bucket-file-path report.html --report-url "<https://storage.cloud.google.com/elementary_data/report.html>"
          edr monitor --report-url "<https://storage.cloud.google.com/elementary_data/report.html>" --slack-webhook "<https://hooks.slack.com/services/X/Y/X>" --profile-target prod --profiles-dir . --env prod --group-by table --suppression-interval 0
Attached is the failure log for the run. I think my main concern is the error handling as I'm unable to clearly see where the error is to debug it. All these commands work perfectly on local but when I push to Github things start to break down. As I'm unclear what how 
edr
exactly works it's hard to figure out where the issue lies quickly.
github actions failure.txt
i
OK @Nitesh Sharoff thanks for sharing. So now I'm actually pretty sure it's what I wrote earlier - you're using a relative path for the keyfile - 
./dbt-service-account.json
. However, the 
dbt
run from within 
edr
does not run in the same directory as the one you're writing the file in, so that's not going to work. I saw that you tried 
tmp/dbt-service-account.json
but that's also a relative path, can you please try 
/tmp/dbt-service-account.json
? That's an absolute path so I think it should work.
n
Hey @Itamar Hartstein, That was exactly it. Thanks very much for your help here - silly me for not spotting it. I assume this was the same reason the Github Action failed as well? Any chance we can add a more detailed error log to the backlog to avoid this happening to anyone in the future? Thanks again for your help :) For anyone interested, here is my revised Action:
Copy code
name: schedule_and_push

on:
  schedule:
    - cron:  '30 9 * * *'
  push:
    branches:
      - main

env:
  DBT_PROFILES_DIR: ./
  DBT_GOOGLE_PROJECT_PROD: ${{ secrets.DBT_GOOGLE_PROJECT_PROD }}
  DBT_GOOGLE_BIGQUERY_DATASET_PROD: ${{ secrets.DBT_GOOGLE_BIGQUERY_DATASET_PROD }}
  DBT_GOOGLE_BIGQUERY_KEYFILE_PROD: /tmp/dbt-service-account.json

jobs:
  scheduled_run:
    name: scheduled_run
    runs-on: ubuntu-latest

    steps:
      - name: Check out
        uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12.2"

      - name: Create usable service account
        shell: bash
        env:
          KEYFILE: ${{secrets.DBT_GOOGLE_BIGQUERY_KEYFILE_PROD}}
        run: |
          echo "$KEYFILE" > /tmp/dbt-service-account.json
    
      ....
      
      - name: Run Elementary
        if: success() || steps.dbt_test.outcome == 'failure'
        run: |
          edr send-report --profiles-dir . --profile-target prod --env prod --slack-token "<https://hooks.slack.com/services/X/Y/Z>" --google-service-account-path /tmp/dbt-service-account.json --gcs-bucket-name elementary_data --bucket-file-path report.html --report-url "<https://storage.cloud.google.com/><bucket>/<filename>.html"
          edr monitor --report-url "<https://storage.cloud.google.com/><bucket>/<filename>.html" --slack-webhook "<https://hooks.slack.com/services/X/Y/Z>" --profile-target prod --profiles-dir . --env prod --group-by table --suppression-interval 0
drop corrupt happy 1
🙂 1
i
Hi @Nitesh Sharoff, really glad this worked for you! Yes, makes total sense, we'll think how to make this clearer.
👑 2
thankyou 1
6 Views
Open in Slack
PreviousNext
Powered by