@kapa.ai can airbyte create index on synced tables? api -> mongo, on k8s self hosted 2.0 version

There is a known issue with Airbyte's destination-mssql connector when syncing from a MongoDB source (CDC mode): Airbyte attempts to create indexes on internal CDC columns like

_ab_cdc_deleted_at

and

_ab_cdc_updated_at

, but these columns are created with the

VARCHAR(MAX)

data type, which SQL Server does not allow as index key columns. [Github Issue] The specific error reported is: >

Column '_ab_cdc_deleted_at' in table 'catalogServiceDb.newRestaurants' is of a type that is invalid for use as a key column in an index.

This causes the destination process to exit with code 1 and the sync to fail entirely. [Github Issue] Key points from the issue: • The root cause is that

_ab_cdc_*

columns (and potentially

_airbyte_raw_id

) are created with incompatible types (e.g.,

VARCHAR(MAX)

) that cannot be used as index keys in MSSQL. • Manually changing the column type to

datetime2

or

datetimeoffset

also causes errors (

No enum constant io.airbyte.integrations.destination.mssql.v2.convert.MssqlType.DATETIME2

or index dependency conflicts). • Indexes are not automatically recreated after changing column data types and restarting the job. • A community comment notes that ideally the destination column type should be derived from the source column definition rather than defaulting to

VARCHAR(MAX)

. [Github Issue] As of the knowledge available, this is an open, unresolved bug in the MSSQL destination connector when used with a MongoDB CDC source. There is no confirmed fix or workaround documented in the provided sources. If you are running this setup on self-hosted Airbyte 2.0 on Kubernetes, you are likely to encounter this issue. I'd recommend tracking or commenting on the GitHub issue to stay updated on any resolution. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

no my destination is mongodb and my source is a custom connector

Thank you for the clarification! Your setup is a custom source connector → MongoDB destination on self-hosted Airbyte 2.0 on Kubernetes. Regarding index creation on synced tables: the knowledge sources do not contain any information about Airbyte automatically creating indexes on collections in the MongoDB destination. What the MongoDB destination documentation does describe is the output schema for each synced stream — each collection will contain these fields: [MongoDB Destination] • `_id`: an identifier assigned to each document (type:

String

) • `_airbyte_emitted_at`: a timestamp for when the event was pulled from the source (type:

Timestamp

) • `_airbyte_data`: a JSON blob with the event data (type:

Object

) There is no mention of Airbyte creating indexes on MongoDB destination collections as part of the sync process. Index management on the MongoDB destination would need to be handled manually by you outside of Airbyte. If you have a specific error or behavior you're encountering with your custom source → MongoDB destination setup, please share more details and I'll do my best to help based on available information. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.