When using Column Selection, is it expected that c...
# replication-ask-ai
m
When using Column Selection, is it expected that columns that are excluded will be present in the __airbyte_r_raw tables, or am I running into a bug with a custom connector? I have a custom bulk SFTP connector (almost identical to the Airbyte provided one, but with the ability to specificy a few extra options). I'm syncing from SFTP to Snowflake. I've de-selected a column that I don't ever want to hit the destination, and it's not present in the final, normalized tables. But when I look in the "raw" version of the table, I see it in the JSON. Any way to stop the column from showing up there?
k
Based on the information provided in the extracts, it appears that the behavior you're experiencing is expected. When you deselect a column using Column Selection, it will not appear in the final, normalized tables. However, the raw data tables, named
*_airbyte_raw_*
, contain the raw JSON data fetched from the source, including all columns [source](https://airbyte.com/tutorials/postgres-to-bigquery). This is because a core tenet of ELT philosophy, which Airbyte follows, is that data should be untouched as it moves through the "E" (Extract) and "L" (Load) stages so that the raw data is always accessible [source](https://airbyte.com/tutorials/postgres-to-bigquery). If you want to ensure that certain data never leaves the source, you would need to handle that at the extraction level, which would likely involve modifications to the source connector. However, please note that there is a discussion in the Airbyte community about the need for a feature to select columns, especially when there are PII data involved [source](https://discuss.airbyte.io/t/2933). You might want to follow this discussion for updates.
m
I noticed that another connector (SQL Server Source) correctly doesn't include the de-selected columns in the raw tables. Is this configured on a connector-by-connector basis?