<@U04S8PLJWMT> i have column _id with type double,...
# replication-ask-ai
t
@kapa.ai i have column _id with type double, however it got failed during sync process with error : Unsupported _id type Double, does INT64 type is acceptable for airbyte version 2.0.0? or should it converted to string?
k
The knowledge sources don’t explicitly mention the exact error
Unsupported _id type Double
or a hard requirement for
_id
to be a specific type in Airbyte 2.0.0, so I can’t confirm the behavior with full certainty. From the general Airbyte type system: - Airbyte supports
number
(mapped to
double
) and
integer
(mapped to 64‑bit integer) as core numeric types.[[Data types](https://docs.airbyte.com/platform/understanding-airbyte/supported-data-types)] - For Avro/Parquet destinations,
number
is mapped to Avro
double
, and
integer
to Avro
long
.[[JSON–Avro rules](https://docs.airbyte.com/platform/understanding-airbyte/json-avro-conversion#conversion-rules); [S3 data changes](https://docs.airbyte.com/integrations/destinations/s3-migrations#data-changes)] - Destination best‑practice is to choose types “at least as large as needed” and to have a “failsafe” type (often string or JSON) when the source type is problematic.[[Data fidelity](https://docs.airbyte.com/platform/next/connector-development/partner-certified-destinations#data-fidelity-and-data-types)] For MongoDB specifically, the changelog notes that newer versions added support for binary
_id
types and warn (rather than fail) on different
_id
types, but it does not explicitly say that
_id
of type
double
is supported; the error you see suggests that particular
_id
type is not accepted by the connector you’re using.[[MongoDB v2 changelog](https://docs.airbyte.com/integrations/sources/mongodb-v2#changelog)] Given this: - I cannot confirm from the docs that mapping
_id
to
INT64
(integer) is officially supported for your exact case in 2.0.0. - The general guidance is: - Prefer a numeric type that matches the semantics (e.g., integer if it’s really an integer ID).[[Data fidelity](https://docs.airbyte.com/platform/next/connector-development/partner-certified-destinations#data-fidelity-and-data-types)] - If the connector/destination rejects that
_id
type, fall back to a “failsafe” type such as
string
to avoid sync failures.[[Data fidelity](https://docs.airbyte.com/platform/next/connector-development/partner-certified-destinations#data-fidelity-and-data-types)] So, if your current
_id
is a MongoDB
double
and the connector throws
Unsupported _id type Double
, the safest change—based on the documented guidance—is to convert
_id
to a string (or store it in a JSON/variant column if your destination supports that), rather than relying on an undocumented assumption that
INT64
will be accepted.
t
is there any action we can do from airbyte side without altering in database side to handle this error? column _id itself is a primary key
is there any action we can do from airbyte side without altering in database side to handle this error? column _id itself is a primary key @kapa.ai
k
A message from kapa.ai
The knowledge sources don’t describe a built‑in way in Airbyte to “override” or remap an unsupported MongoDB
_id
type (like
double
) at sync time without changing the database schema, so I can’t point to an official, documented switch or setting that fixes this automatically. What is documented: 1. MongoDB v2 connector behavior around
_id
types
The changelog mentions support for additional
_id
types (e.g., binary) and changing some `_id`‑type issues from “fail” to “warn”, but it does not state that
_id
of type
double
is supported or that Airbyte can coerce it for you.[[MongoDB v2 changelog](https://docs.airbyte.com/integrations/sources/mongodb-v2#changelog)] Your error (
Unsupported _id type Double
) indicates this specific type is not accepted by the connector version you’re using. 2. Type transformation is done in the source, but is schema‑driven and generic The Python CDK has a
Transformer
that can cast values to match the declared JSON schema (e.g., cast numbers to strings), but this is something a custom source connector author configures in code; it’s not exposed as a per‑column setting in the UI for existing connectors.[[Type transformation](https://docs.airbyte.com/platform/connector-development/cdk-python/schemas#type-transformation)] So, out of the box, you can’t just “turn on” a transformation for
_id
in the standard MongoDB connector via the UI. 3. General guidance on data types and failsafe types Airbyte’s destination guidelines say that when a good type cannot be chosen, connectors should fall back to a “failsafe” type like string/JSON to avoid sync failures.[[Data fidelity](https://docs.airbyte.com/platform/connector-development/partner-certified-destinations#data-fidelity-and-data-types)] However, this is guidance for connector implementers, not an end‑user configuration knob you can flip for a single column in an existing connector. Given these constraints, from the Airbyte side only, the documented options are essentially: - Upgrade to the latest MongoDB v2 connector and Airbyte version to benefit from any newer handling of
_id
types (the changelog shows ongoing work around
_id
handling and Debezium).[[MongoDB v2 changelog](https://docs.airbyte.com/integrations/sources/mongodb-v2#changelog)] This might change behavior (e.g., from fail to warn) but the docs do not guarantee support for
_id
as
double
. - Implement or fork a custom MongoDB source connector using the CDK and enable a
Transformer
to coerce
_id
to a supported type (e.g., string) before it reaches the destination.[[Type transformation](https://docs.airbyte.com/platform/connector-development/cdk-python/schemas#type-transformation)] This is the only documented way to change the type on the Airbyte side without touching the database, but it requires development work. The knowledge sources do not document: - Any UI‑level setting to cast
_id
from
double
to another type. - Any per‑stream/per‑column mapping feature in the standard MongoDB connector that would let you fix this without changing the DB or writing a custom connector. So, based on the docs: - There is no simple, built‑in Airbyte configuration to handle
Unsupported _id type Double
for a primary key without either: - changing the database schema, or - creating/forking a connector that transforms
_id
before output. If you’d like, I can outline how a custom connector would use the
Transformer
to cast
_id
to string, based strictly on the CDK docs.
t
@kapa.ai how to set custom connector? however in airbyte 2.0, during connection configuration, mongodb connector was set by default using mongodb v2 connector
k
A message from kapa.ai
You can work around the default MongoDB v2 connector by registering your own Docker-based custom connector and then using that in your connection instead. From the docs, the flow is: 1. Build your custom connector image For example, if you’ve forked/modified the MongoDB v2 connector, build its Docker image locally (or in CI) and push it to a registry you can access from Airbyte. The forum example shows building a modified connector with Gradle, then tagging the image and using it as a custom connector.[[Forum: modified connector](https://discuss.airbyte.io/t/1812)] 2. Make sure Airbyte can pull the image - Host the image in a Docker registry (private or public). - Configure your Airbyte deployment so it can authenticate to that registry (if private). You can verify this by running
docker pull <image-name>:<tag>
on the Airbyte host.[[Using custom connectors](https://docs.airbyte.com/platform/operator-guides/using-custom-connectors#4-use-your-custom-docker-connector-in-airbyte)] 3. Register the custom connector in the UI In the Airbyte UI (this applies to both “classic” and 2.0+ flows): 1. Go to Workspace settings. 2. Choose Sources (for MongoDB) or Destinations. 3. Click New connector → Add a new Docker connector. 4. Fill in: - Connector display name: e.g.
My MongoDB v2 Custom
. - Docker full image name: e.g.
<http://my-registry.com/my-mongodb-v2|my-registry.com/my-mongodb-v2>
. - Docker image tag: e.g.
dev
or
1.0.0
. - (Optional) Connector documentation URL.[[Using custom connectors](https://docs.airbyte.com/platform/operator-guides/using-custom-connectors#4-use-your-custom-docker-connector-in-airbyte)] 5. Click Add. After this, your custom connector appears in the catalog for that workspace. 4. Use the custom connector instead of the default MongoDB v2 When you create a new connection, choose your newly added custom source (
My MongoDB v2 Custom
) instead of the built‑in “MongoDB v2” source. The built‑in one will still exist, but you’re free to ignore it and use your custom one. If you’re on Airbyte “Next” / 2.x, the same “Add a new Docker connector” flow is documented there as well; the steps are effectively identical.[[Using custom connectors (Next)](https://docs.airbyte.com/platform/next/operator-guides/using-custom-connectors#4-use-your-custom-docker-connector-in-airbyte)] If you need to change the connector implementation (e.g., to handle
_id
types differently), that change must be done in the connector code before you build and register the custom image; there is no documented way to override the built‑in MongoDB v2 behavior purely via connection configuration.