I wonder if there might be something wrong in the latest package-version (

v0.11.2

)? (Could also be user error I'm sure). We use the

dbt_models

for many purposes. We have currently 2446 models in the project. In our production environment using

v0.11.1

,

dbt_models

contains all the models. However, in our development and test environments that both use

v0.11.2

, there are only 1203 models present in

dbt_models

. This causes downstream issues for us, and I cannot understand why it is so. Seems like I might have to downgrade.

@Sindre Grindheim can you tell me a little bit about your setup? do you have exactly the same code in all the envs? Are you using slim CI?

Same code in all environments. All environments write to different catalogs. Not using slim CI. We have disable_dbt_artifacts_autoupload set to true for normal runs, except for deploys - identified by flags.FULL_REFRESH being true. Never had a problem before now though.

Can you try to downgrade the other envs to 0.11.1 and see if this solves the issue?

Tried downgrading the test environment yesterday, but got a few errors:

14:06:44    Compilation Error in model dbt_models (models/edr/dbt_artifacts/dbt_models.sql)

Delta Lake does not support dropping columns from tables

14:06:44    Compilation Error in model dbt_snapshots (models/edr/dbt_artifacts/dbt_snapshots.sql)

Delta Lake does not support dropping columns from tables

did you get this after running

dbt run -s elementary

?

We used

dbt run --select state:modified+ --full-refresh

for the deploy. Since the package version was changed, all the elementary models were run as well.

@Sindre Grindheim it seems you're not able to downgrade because 0.11.2 added a new column to dbt_models and dbt_snapshots and delta lake cannot delete those columns. removing these columns manually can help you downgrade, but I would like to understand what is causing the missing models. I have a few suspicions as to what may be the issue so writing them here and let me know if you like to try it: 1. If you're running with a state selector I wonder if that means some of the elementary models did not run if they haven't changed, or weren't detected as changed. Would it be possible to try running

dbt run -s elementary

separately just in case and see if that changes anything? 2. if you have artifact upload disabled in some runs and enabled in others I wonder if this can cause the artifacts that are uploaded to somehow be partial. I'm not sure this is the case but to make sure you can run

dbt run --select elementary.edr.dbt_artifacts

. this is the operation that uploads the models to

dbt_models

. If any of those suggestions works this still doesn't mean we understand what happened but it will resolve the issue for you and will help us investigate. If it doesn't, it would be great if you can share your dbt.log with me, along with a name or a few names of models that are missing from your staging schema

Tried first running

dbt run --select elementary.edr.dbt_artifacts

and then

dbt run --select elementary.edr.dbt_artifacts --full-refresh

. Both runs gave the same result, where

dbt_models

has 1203 rows. As you can see in the attached screenshot, there are 2450 models. Here are the flags that are set in

dbt_project.yml

- the target name for scheduled runs is <env>_release: • disable_run_results: "{{ 'release' not in target.name }}" • disable_tests_results: "{{ 'release' not in target.name }}" • disable_dbt_artifacts_autoupload: "{{ 'release' not in target.name or not flags.FULL_REFRESH }}" • disable_dbt_invocation_autoupload: "{{ 'release' not in target.name }}"

wow thank you! we will look into it

Seems to me that the fix is easy - lines 76-79 should be moved to after line 80. Tested it and it worked like a charm.

Hi @Matthias Niehoff, can you confirm if this is the issue for you as well?

Will be able to test it tomorrow hopefully. might be related but for me its not model/test metadata but data about test/model runs

Yes @Matthias Niehoff we'll keep investigating your issue. Thanks for trying this out. @Sindre Grindheim we will release the fix soon and update you 🙏🏼

Hey @Matthias Niehoff and @Sindre Grindheim, we released

0.12.1

with a fix. Thank you for reporting!

Hey, thanks for the quick fix. I just tested it and it looks fine so far! 👍

Thanks for the fix @Maayan Salom. It did seem to fix the

dbt_models

issue. However, since upgrading to the altest

0.12.1

version, we're struggling with the

on-run-end

hook updating

dbt_columns

. As I've reported in #C02CTC89LAX, it hangs for over an hour and i had to manually cancel it after that (so can't say if it would ever finish). But now I've been forced to once more downgrade to

0.11.1

.

Hi @Sindre Grindheim, Checking this out.

Hi @Ella Katz. The first upload was indeed slow, but it worked fine. The second time however was when the problem occurred. We can't disable all the artifact uploads since we use them actively access control purposes.

hmm, that's interesting. Do you have a dbt.log of the run where it got stuck? can you send it to me in DM?