i need use policy in schema bigquery, airbyte recr...
# replication-troubleshootingf
Francisco Viera
10/21/2022, 2:07 PMi need use policy in schema bigquery, airbyte recreate the tables always
✍️ 1
u
user
10/21/2022, 4:09 PM@[DEPRECATED] Marcos Marx turned this message into Zendesk ticket 3021 to ensure timely resolution!
u
[DEPRECATED] Marcos Marx
10/21/2022, 4:09 PM
Please use threads.
Link: https://cloud.google.com/bigquery/docs/column-level-security
u
user
10/21/2022, 5:04 PMFrancisco, let me know if that helps!
f
Francisco Viera
10/21/2022, 6:07 PM@[DEPRECATED] Marcos Marx @Nataly Merezhuk (Airbyte), i need load from a source with data sensitive, but when set a connect source to bigquery this recreate tables in bigquery
Francisco Viera
10/21/2022, 6:08 PMi dont want that recreate the tables
Francisco Viera
10/21/2022, 6:09 PMi have ready the taxonomy policy tags in datacalog
Francisco Viera
10/21/2022, 6:09 PMwhen to apply a tag to table in bigquery and run sync en airbyte
Francisco Viera
10/21/2022, 6:10 PMairbyte recrete the tables
Francisco Viera
10/21/2022, 6:10 PMand delete all policy tag apply
u
user
10/21/2022, 6:59 PMOK, let me look into this and I'll get back to you with some ideas!
m
Marcelo Pio de Castro
10/21/2022, 11:01 PMI think you can use DBT with custom transformations to apply the policy tags to the table after airbyte finishes any sync, so they will always apply
Marcelo Pio de Castro
10/21/2022, 11:02 PMIf that doesn't work, you can always get the DBT code that Airbyte generated, and customize it to add the policy tags, then you apply it with custom transformations
f
Francisco Viera
10/22/2022, 1:22 PM@Marcelo Pio de Castro you have an example?
Francisco Viera
10/22/2022, 1:26 PMthis option create in bigquery as table o view?
Francisco Viera
10/22/2022, 1:29 PMthe tables that I need have sensitive values and it is not clear to me if with dbt, I get the transformed bigquery table or create another table or view with the policy tag?
m
Marcelo Pio de Castro
10/22/2022, 2:11 PMMarcelo Pio de Castro
10/22/2022, 2:13 PMMarcelo Pio de Castro
10/22/2022, 2:15 PMI would try adding a policy_tag to the source column
Marcelo Pio de Castro
10/22/2022, 2:17 PMIf that doesn't work, you can get the generated DBT normalization code from Airbyte
https://docs.airbyte.com/operator-guides/transformation-and-normalization/transformations-with-dbt#exporting-dbt-normalization-project-outside-airbyte
f
Francisco Viera
10/22/2022, 2:31 PM@Marcelo Pio de Castro ok my concern is to have 2 tables, the first one generated by airbyte and without policy tag and another table generated by dtb with policy tag, I can't live with the first table without policy tag.
airbyte with dbt creates in the destination only the table transformed with dbt???
m
Marcelo Pio de Castro
10/22/2022, 2:34 PMAirbyte already uses DBT, the tables you see in bigquery are generated with it. Applying policy_tags to the source (if possible) is using the airbyte tables (the source). If you change the DBT code from Airbyte, you can change the policy_tags directly, without using source, so the airbyte tables will be generated with policy_tags
Marcelo Pio de Castro
10/22/2022, 2:36 PMFrom a security standpoint, changing the airbyte generated DBT code is safer, because that guarantee that it will have a policy tag no matter what
f
Francisco Viera
10/22/2022, 3:27 PMok thanks, i will try
Francisco Viera
10/22/2022, 8:50 PMhello, i use airbyte with helm, which is the pod that have the models?
m
Marcelo Pio de Castro
10/23/2022, 2:12 AMAh, that might be a problem:
https://github.com/airbytehq/airbyte/issues/8090
5 Views