<#C02CTC89LAX|> When running the various anomaly t...
# community-supportn
Neil Merkling
12/15/2023, 11:28 PM#C02CTC89LAX When running the various anomaly tests, I believe the process creates a number of transient tables that (I think) are intended to be temporary.
Should the process delete these tables or is there a clean-up step I need to run? Maybe they are sticking around because of an error in my process that caused it to kick-out before hitting that point?
I couldn't find any information about this in the Documentation, but I might have just missed it.
👀 1
u
Unthread
12/18/2023, 7:02 AMHeya @Neil Merkling , thanks for reaching out! Those tables are indeed temporary and shouldn't be persisted. Could you perhaps attach a
edr.logn
Neil Merkling
12/19/2023, 9:12 PMNot sure how to get an old log file. I am currently using dbt Cloud.
I attempted to run the Anomaly tests today to see if I could get the log file with the errors, but it actually appeared to have worked correctly.
Now, I had been having trouble with dbt's <https://docs.getdbt.com/docs/build/custom-schemas%7C<target_schema>_<custom_schema>> protocol as my company does not allow us to create and delete individual schemas.
I modified the "generate_schema_name" macro in order to use either the target schema or the custom schema, but not to concatenate them.
I believe I did that between the time these temp tables failed to be removed and this latest attempt. Elementary is setup in our dbt_project.yml for a Custom Schema.
I am going to watch this in case it happens again.
Thank you for at least confirming that the behavior is unexpected.
s
Samuel Garfield
12/21/2023, 12:40 AMBeen encountering this on my end as well
Samuel Garfield
12/21/2023, 12:41 AMI also don't see
edr.logSamuel Garfield
12/21/2023, 12:46 AMif you're using databricks or a spark cluster you can use a pyspark script like this to drop all of them automatically though
Copy code
catalog = 'INSERT_HERE'
elementary_schema = 'INSERT_HERE'
spark.sql("use catalog " + catalog)
table_df = spark.sql("show tables in " + elementary_schema)
table_list = [row.tableName for row in table_df.select("tableName").collect()]
tables_to_keep = [table for table in table_list if '__tmp' not in table]
tables_to_drop = list(set(table_list) - set(tables_to_keep))
print("Tables to drop: \n\n" + '\n'.join(tables_to_drop) + '\n')
print("Tables to keep: \n\n" + '\n'.join(tables_to_keep))
for table in tables_to_drop:
spark.sql("drop table " + catalog + "." + elementary_schema + "." + table + ";")34 Views