Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

image.png

hey everyone, we've been using airbyte as a data integration tool for a while now, but recently we're facing some problems with data types in microsft sql server CDC. We have a legacy database with some custom data types that airbyte seems to completely ignore when syncing data.

is there a way of capturing these fields?

Hi, Matheus! Could you tell me what version of Airbyte/connectors you're using? I believe you could do this with dbt. We have a series of articles on this in the docs:
<https://docs.airbyte.com/operator-guides/transformation-and-normalization/transformations-with-sql>

Let me know if this helps!

we're running airbyte 0.40.26 and connector <https://docs.airbyte.com/integrations/sources/mssql|airbyte/source-mssql> 0.4.26

I'm not sure I understand the documentation you linked

I'm not sure I understand the documentation you linked

In the sync logs, I get something like this:

```2022-12-14 14:25:59 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Exporting data from table '&lt;REDACTED&gt;' (1 of 1 tables)
2022-12-14 14:25:59 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - 	 For table '&lt;REDACTED&gt;' using select statement: '&lt;SELECT &lt;REDACTED&gt;, &lt;REDACTED&gt;, &lt;REDACTED&gt;, &lt;REDACTED&gt;, &lt;REDACTED&gt;, &lt;REDACTED&gt; FROM &lt;REDACTED&gt;'```
I redacted the fields, but none of them is the `QT_ESTMIN` or `QT_ESTMAX` custom fields I showed in the screenshot

Would this transformation with DBT still work?

Wouldn't this transformation with DBT still work?

Ah, sorry that I misunderstood - dbt would not work as you need custom data types when fetching the data from the source, not when populating the destination. This is currently not supported by Airbyte. I encourage you to make a feature request on GitHub and contribute a PR if you've got the time!

that's what I thought, do you have any PRs we could use as a base for this?

any comments on the issue on where to start with a PR would be highly appreciated <https://github.com/airbytehq/airbyte/issues/21314>