@kapa.ai we're running into a CDC sync failure with the MySQL source connector (v3.51.5) on self-hosted Airbyte (Kubernetes). Error Failed to properly convert data value for 'oss.need_get_consents_08.created_at' of type DATETIME org.apache.kafka.connect.errors.DataException: Invalid value: null used for required field: "created_at", schema type: STRING at org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:224) ~[connect-api-3.9.0.jar:?] DDL table source required not null CREATE TABLE

need_get_consents_08

(

id

int unsigned NOT NULL AUTO_INCREMENT,

customer_id

int NOT NULL,

enrollment_id

varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,

state

varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,

last_usage_at

datetime NOT NULL,

last_usage_by

varchar(200) NOT NULL,

status

tinyint NOT NULL DEFAULT '0',

created_at

datetime NOT NULL, PRIMARY KEY (

id

), KEY

customer_id

(

customer_id

) ) ENGINE=InnoDB AUTO_INCREMENT=1377621 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci So, How to fix it?

The error you're seeing is a known issue with the MySQL CDC source connector. The

DataException: Invalid value: null used for required field

occurs when Debezium receives a

null

value for a column that is defined as

NOT NULL

in the schema (like your

created_at datetime NOT NULL

). Based on the knowledge sources available, here is what's relevant to your situation: Root Cause A bug was introduced in the MySQL source connector (around version 3.50.x) related to how CDC records handle null values during deserialization. Specifically, a code change caused primary key (and potentially other) fields with null values to be skipped entirely during CDC record deserialization, which then conflicts with schema validation that expects a non-null value. [MySQL CDC null PK issue] There is also a separate but related issue where the CDC connector's schema history can grow unbounded, causing progressive degradation — this was reported on version 3.51.5 specifically. [CDC schema history issue] What You Can Try 1. Check for a connector update: A fix (PR #68158) was merged and released in version

3.50.10

to ensure fields are always included in the record payload even when null. Since you're on

3.51.5

, it's worth verifying whether this fix is included in your version or if a newer patch is available. [MySQL CDC fix] 2. Investigate the actual data: Even though

created_at

is defined as

NOT NULL

in the DDL, CDC events (especially

DELETE

events or events from schema changes) can sometimes carry null values for columns. Check whether the failing CDC events correspond to delete operations or schema migration events. 3. Review the null GTID warm-start fix: There is also a recent fix (PR #74734) for MySQL CDC warm-start validation when the saved state contains a null GTID, which could cause unexpected behavior during incremental syncs. [null GTID fix] Limitations The knowledge sources do not contain a definitive fix specifically for the

Invalid value: null used for required field

error on

v3.51.5

. I'd recommend: • Opening or searching for a GitHub issue at airbytehq/airbyte with your exact error message and connector version. • Checking the MySQL source connector changelog for

3.51.x

to see if a patch addressing this has been released after

3.51.5

. I'm sorry I cannot provide a more definitive resolution — the knowledge sources do not contain enough detail about this specific error on

v3.51.5

to confirm a guaranteed fix. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai v3.51.5 is latest version, What's the reason cause the error?

Based on the knowledge sources available, the most directly relevant explanation for your error comes from a similar issue reported with the Postgres source connector, which shows the same underlying pattern: Root Cause The error

DataException: Invalid value: null used for required field: "created_at", schema type: STRING

is a Debezium type conversion error occurring during CDC incremental syncs. Specifically: • Debezium is receiving a

null

value for the

created_at

DATETIME column, but the schema expects a

STRING

type value. • This type of error has been observed to occur specifically during DELETE operations in the binary log, where the "before" image of a row may carry null or missing values for certain fields. [Postgres CDC type error] A very similar error was reported for the Postgres connector: > "Failed to properly convert data value for

createdAt

of type timestamp — `Invalid Java object for schema with type STRING: class java.lang.Long`" — occurring specifically on DELETE events in the WAL. [Postgres CDC type error] There is also a known MySQL CDC bug (introduced around version 3.50.1) where null values in fields are skipped entirely during CDC record deserialization, which can cause downstream schema validation failures. [MySQL CDC null PK issue] In Summary The error is caused by a mismatch between the value Debezium provides (null) and what the schema expects (a non-null STRING) during CDC event processing — most likely triggered by DELETE or UPDATE events in the MySQL binary log. The knowledge sources do not contain specific details about

v3.51.5

or a confirmed fix for this exact error in that version. I'd recommend checking the Airbyte GitHub issues for the latest status, or joining the Airbyte Community Slack for peer support. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai what is the ideal configuration of airbyte vlaues to have a perfect installation without editing

The knowledge sources provided don't contain a comprehensive "ideal default configuration" for Airbyte's

values.yaml

that covers a full installation without editing. However, based on what the documentation does say, here are the key resource-related configuration points: What the Docs Say About Defaults Airbyte's default resource allocations are designed to be sufficient for most use cases: > "We recommend that you start with the default resource allocations set by Airbyte. These should be sufficient for most use cases." [Configuring Connector Resources] When You May Need to Edit You would typically only need to modify

values.yaml

if: 1. Memory is insufficient — The Source worker reads up to 10,000 records in memory. For large row sizes (e.g. 0.5MB/row), you may need up to 5GB RAM. [Scaling Airbyte] 2. Disk space is low — A minimum of 30GB per node is recommended, as connector images are ~300MB each. [Scaling Airbyte] 3. Many concurrent syncs — Plan for

2 × <number of concurrent connections>

pods schedulable at once. [Scaling Airbyte] 4. High requested (but low used) CPU/memory — You can tune connector pod defaults in `values.yaml`:

global:
  edition: "enterprise"
  jobs:
    resources:
      limits:
        cpu: ## e.g. 250m
        memory: ## e.g. 500m
      requests:
        cpu: ## e.g. 75m
        memory: ## e.g. 150m

[Scaling Airbyte] Bottom Line The knowledge sources do not define a single "perfect" out-of-the-box configuration, as the ideal setup depends on your workload. The recommendation is to start with Airbyte's defaults, monitor resource usage via observability tools, and tune incrementally as needed. [Configuring Connector Resources] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai I'm running Airbyte on EKS Auto Mode using Helm chart airbyte/airbyte version 1.9.2. After a fresh helm upgrade --install, several pods are stuck in a restart loop: • airbyte-workload-api-server — 5 restarts • airbyte-workload-launcher — 5 restarts • airbyte-connector-builder-server — 4 restarts • airbyte-server — 1 restart (during DB migrations) Root causes I've identified: 1. workload-api-server, workload-launcher, and connector-builder-server have liveness/readiness/startup probes that appear to target the wrong port (8085), causing every probe check to fail with a connection refused, which triggers an immediate restart loop. 2. airbyte-server has its liveness probe initialDelaySeconds set to 0, which causes Kubernetes to kill the container while Flyway DB migrations are still running on a fresh database. Workaround I'm currently using (post-install kubectl patch): Remove all probes from workload-api-server, workload-launcher, and connector-builder-server • Set livenessProbe.initialDelaySeconds=600 on airbyte-server My questions: Is this a known bug in chart v1.9.2? Is it fixed in a newer version? What are the correct ports these probes should target for workload-api-server and workload-launcher? I'd prefer to fix the probe config rather than remove probes entirely. Can these probe settings be controlled via values.yaml overrides, or does the chart template hardcode them and ignore values? Setting livenessProbe: null in values appears to have no effect. What is the recommended way to handle the airbyte-server startup time on a fresh database with Flyway migrations running?

These are great detailed questions. Let me address each one based on what the knowledge sources actually contain. What the Knowledge Sources Confirm Probe Configuration via values.yaml The knowledge sources do confirm that liveness and readiness probe settings are controllable via

values.yaml

overrides — they are not hardcoded. For example, for `airbyte-server`:

server:
  livenessProbe:
    enabled: true
    initialDelaySeconds: 30
    periodSeconds: 10
    timeoutSeconds: 10
    failureThreshold: 3
    successThreshold: 1

  readinessProbe:
    enabled: true
    initialDelaySeconds: 10
    periodSeconds: 10
    timeoutSeconds: 10
    failureThreshold: 3
    successThreshold: 1

[Values.yaml reference] Similarly,

workload-launcher

exposes the same probe fields in

values.yaml

. [Values.yaml reference] Default initialDelaySeconds for airbyte-server The default

initialDelaySeconds

for the server's liveness probe is 30 seconds (not 0), per the documented defaults. [Values.yaml reference] If you're seeing

0

, that may be a chart-version-specific deviation from the documented defaults. What the Knowledge Sources Do NOT Cover I'm sorry, but the knowledge sources do not contain enough information to answer the following parts of your questions: 1. Whether this is a known bug in chart v1.9.2 — The sources do not mention this specific chart version or these probe issues. 2. The correct ports for workload-api-server and workload-launcher probes — The sources do not specify which ports the probes should target for these components. 3. Why

livenessProbe: null

has no effect — The sources do not document this behavior or any known limitation around nulling out probe configs. 4. Recommended handling of Flyway migration startup time — The sources do not address this scenario. Recommended Next Steps • For the

initialDelaySeconds

issue on

airbyte-server

, you can override it via

values.yaml

(confirmed supported)server livenessProbe: initialDelaySeconds: 600 • For the probe port issues on

workload-api-server

and

workload-launcher

, I'd recommend checking the Airbyte GitHub issues or the chart source at airbytehq/airbyte-platform directly, as the full

values.yaml

for v2 charts is referenced there but not fully reproduced in the knowledge sources. [Values.yaml reference] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.