Hi all! I run the tests once, but I get 4 duplicat...
# community-support
o
Hi all! I run the tests once, but I get 4 duplicates for each test, any ideas what this could be related to? test looks like
Copy code
- name: vacation_policy_assigned
  description: "{{ doc('vacation_policy_assigned') }}"
  tests:
    - dbt_expectations.expect_column_to_exist
    - dbt_expectations.expect_column_values_to_be_of_type:
        column_type: varchar
    - expect_column_values_to_match_regex:
        select_columns: "employee_id, vacation_policy_assigned"
        regex: "^[a-zA-Z0-9.,]+$"
d
I updated to most recent 0.12.1 and duplication is gone. Need to run 'dbt run --select elementary' after update however.
n
Hey @Oksana Gorbachenko Did upgrading the version work for you?
o
Hi! Yep, version
0.12.1
works fine for us, thanks a lot!
a parrot 1
a
Hi @Noa Kurman! I'm using version
0.13.2
and faced the same problem.
c
Hi @Noa Kurman we just started seeing the same thing, we're on
0.9.2
(and cannot, unfortunately, upgrade)
test runs also seems to be duplicated. note that i've confirmed in the metastore that duplicated records don't exist, so this seems to be happening during report generation
a
I discussed this with @Noy Arie. In my case, the problem occurs due to running tests/models in parallel. When you are running in parallel, the
on-run-end
hook will run in parallel as well. This hook collects and saves artifact data. Duplication in the artifact tables will cause this issue (the query of getting the test results includes a join with the artifact table). example:
elementary_test_results
1 record *
dbt_sources
2 records *
dbt_tests
6 records -> 12 records in the report
l
Hello everyone. Also using version 0.13.2 and also having this problem. Do we have any solution for this? Or some simple fix?
n
Hey @Luis Oliveira Did you make sure as Anna mentioned that your tests are not running in parallel?
l
I don't understand what you mean by running in parallel. We are running using 16 threads "Concurrency: 16 threads (target='prod_unity')"
a
Did you do multiple runs at the same time? These may be different tests, but in the same project.
I only found a workaround for this problem. I periodically run
dbt run --select elementary
, this does not delete important data, but gets rid of duplicates.
l
ahhh We do have multiple runs in the same project (different tests/tables)
I will try to apply the workaround. Doesn't that delete the historical data? @Anna Vasileva It is important for the freshness
a
No
I thought so, but when I tried to do it, I discovered that there was no history deletion
👍 1
c
Hi @Noy Arie I just found this thread as I'm having a similar issue. If we disable
disable_dbt_artifacts_autoupload: true
in our elementary runs and just run the artifact upload at deploy time, is it likely to fix it? (We do run multiple
dbt
commands at the same time, so seems like it could be the same problem)