Hello, everyone! First I'd like to say that I am a...
# community-support
g
Hello, everyone! First I'd like to say that I am a huge fan of Elementary. Being using it in every dbt project I've ever been part of. In the latest platform I am building with dbt we are working on a multi-project approach. There is multiples dbt project, each one with elementary set up. The Warehouse is BigQuery. For the multi observability approach, we decided to go in a direction where each dbt project has its own Elementary set of tables. So imagine we set up Elementary in a way that in a specific dataset there is tables such as: bq_project.elementary.dbt_models_projectA bq_project.elementary.dbt_models_projectB In the central dbt project, where we evaluate the metadata, there is a group of modelling designed on top of Elementary models that will use BigQuery wildcard functionality to union all tables. Although it works with mostly every Elementary table, there is one in specific that presents an error whenever we call the staging model that sources it. The error is the following:
Wildcard tables are not supported over tables with columns pending to be dropped. First table: bv-pdbd-prd:trusted_tecn_platdados.elementary_test_results.
That being said, I am having a hard time finding out what elementary_test_results's column is a "column pending to be dropped" (BigQuery's lack of error log doesnt help). Is this column pending to be dropped something designed to be like this? Or perhaps we broke something due to customization? Anything has any idea what might be causing this error?
Some information that might be useful: 1. We are using dbt-core with dbt-bigquery adapter 2. The "metadata centralization" that I've mentioned is a group of models we do using multiple metadata packages we have, such as Elementary, dbt_project_evaluator and some custom built macros we did as well. We call it "dbt telemetry". 3. This error is happening in the staging model that calls the elementary_test_results as a source. Like this: select ... from {{ source('elementary', 'elementary_test_results') }} which will compile to something like this: select ... from bq_project.elementary.
elementary_test_results*
d
Hey @Gabriel Bernardo, I've tried to learn a bit about schema changes in BigQuery. It seems like when a column is dropped, BQ doesn't free it up immediately, so maybe this may create this "pending to be dropped" column. This still weird to me since we didn't change the schema of this table in the last versions.
Is there a chance you made some change to this table from your code ?