@kapa.ai Found this error in airbyte-server logs, can you tell me what it might mean?

Unable to read configuration for notification on connectionId 'beb0c30e-0c9a-43fb-ab51-bdd90ff69766'. Non-blocking. Error:
java.lang.NullPointerException: Cannot invoke "io.airbyte.config.SyncStats.getSourceStateMessagesEmitted()" because "totalStats" is null
        at io.airbyte.persistence.job.tracker.TrackingMetadata.generateJobAttemptMetadata(TrackingMetadata.kt:173)
        at io.airbyte.persistence.job.JobNotifier.notifyJob(JobNotifier.kt:98)
        at io.airbyte.persistence.job.JobNotifier.notifyJob(JobNotifier.kt:72)
        at io.airbyte.persistence.job.JobNotifier.successJob(JobNotifier.kt:219)
        at io.airbyte.commons.server.handlers.JobsHandler.jobSuccessWithAttemptNumber(JobsHandler.kt:188)
        at io.airbyte.server.apis.controllers.JobsApiController.jobSuccessWithAttemptNumber$lambda$14(JobsApiController.kt:194)
        at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.kt:32)
        at io.airbyte.server.apis.controllers.JobsApiController.jobSuccessWithAttemptNumber(JobsApiController.kt:193)
        at io.airbyte.server.apis.controllers.$JobsApiController$Definition$Exec.dispatch(Unknown Source)
        at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invokeUnsafe(AbstractExecutableMethodsDefinition.java:461)
        at io.micronaut.context.DefaultBeanContext$BeanContextUnsafeExecutionHandle.invokeUnsafe(DefaultBeanContext.java:4438)
        at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:272)
        at io.micronaut.web.router.DefaultUriRouteMatch.execute(DefaultUriRouteMatch.java:38)
        at io.micronaut.http.server.RouteExecutor.executeRouteAndConvertBody(RouteExecutor.java:465)
        at io.micronaut.http.server.RouteExecutor.lambda$callRoute$5(RouteExecutor.java:442)
        at io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:92)
        at io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:232)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)

This error is a

NullPointerException

occurring in Airbyte's job tracking/notification code. Specifically, it happens when Airbyte tries to generate job attempt metadata for a notification (in this case, a success notification), but the

totalStats

object (of type

SyncStats

) is

null

, so calling

getSourceStateMessagesEmitted()

on it fails. The stack trace shows: •

TrackingMetadata.generateJobAttemptMetadata

tries to read sync statistics (like

sourceStateMessagesEmitted

) from

totalStats

•

totalStats

is

null

, causing the NPE • This is triggered during

JobNotifier.successJob

, meaning the sync itself completed successfully, but the post-sync notification/tracking step failed This is consistent with a known pattern in Airbyte where the job tracker attempts to collect metadata about a completed sync attempt, but the stats are missing or not populated. [Error running Sheets Sync] [Mysql connection issues] A related GitHub issue notes that a similar NPE can occur when a source's state message does not contain the expected

sourceStats

attribute (i.e., the count of records associated with the state message). [destination-clickhouse sync fails] Key points: • The error is described as "Non-blocking", meaning the sync itself likely succeeded — this is a metadata/notification tracking issue, not a data pipeline failure. • The root cause is that

totalStats

is

null

when Airbyte tries to read it during job completion tracking. • This can happen if the source connector does not emit proper stats/state messages, or if there is a bug in the version of Airbyte you are running. Recommended steps: 1. Check whether the sync actually completed and data was transferred — since the error is non-blocking, the sync may have succeeded. 2. If you are using a custom source connector, ensure it populates the

sourceStats

field in its state messages. 3. Check if upgrading Airbyte resolves the issue, as this has been a known bug in several versions. The knowledge sources do not contain a definitive fix for this exact stack trace, but the pattern is well-documented as a tracking/metadata issue rather than a core sync failure. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai There has been no changes to our instance for weeks, and the sync has been working fine until the last occurrence where I discovered only this error. Could it be due to log bucket or database issues (ie out-of-space)? Any other places I could like to fix this issue? (nb: upgrading seems like the last solution I would like to try as I do not have space in my schedule at the moment)

Based on the knowledge sources, yes — disk/storage space issues are a documented cause of sync failures and errors in Airbyte. Here's what's relevant to your situation: Storage/Space Issues as a Root Cause MinIO storage filling up is a known cause of sync jobs failing or hanging. The default MinIO PVC is only 500MB, which can fill up quickly with frequent syncs. Users have resolved similar issues by: 1. Deleting logs from MinIO storage — specifically from:[All sync jobs hanging] ◦

/storage/airbyte-storage/job-logging/workspace

◦

/storage/airbyte-storage/workload/output

1. Increasing MinIO storage — either by extending the PVC or setting a larger volume claim in `values.yaml`minio storage: volumeClaimValue: 2Gi[All sync jobs hanging] 2. Reducing Temporal history retention to prevent logs from accumulatingenv vars TEMPORAL_HISTORY_RETENTION_IN_DAYS: 7(Default is 30 days.) [All sync jobs hanging] Database (jobs DB) growing too large is also a documented issue. One user found that their jobs database grew until there was no disk space left, causing the

airbyte-db

container to go down and all syncs to fail. [Syncs and connections tests failing] Suggested Actions (without upgrading) • Check disk usage on your instance (e.g.,

df -h

) and on the MinIO/database volumes. • Clear old logs from MinIO storage as described above. • If on Docker, run

docker system prune

to reclaim space (though one user noted this alone didn't solve their problem). [Syncs and connections tests failing] • Set up log retention policies to prevent recurrence. The knowledge sources do not explicitly confirm that a full storage volume would produce exactly the

NullPointerException

you're seeing, but storage exhaustion is a well-documented cause of unexpected Airbyte failures, and it's a reasonable place to investigate given your description of a sudden failure with no configuration changes. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai, Unfortunately our instance is using google buckets instead of minio, and google cloud postgresql as the airbyte database. Any thoughts on how that may relate to the original issue?

Good context — your setup (GCS for storage + Cloud PostgreSQL as the Airbyte database) is directly relevant to the original error, and there are documented issues in the knowledge sources that match your configuration. GCS Storage Configuration Issues Several users on Kubernetes/GKE deployments using GCS instead of MinIO have encountered errors in the Airbyte server logs related to missing or misconfigured storage variables. The key finding is: Variables must be set on both the Worker and the Server pods. One user resolved a similar `NullPointerException`/storage-related error by adding the GCS storage variables to the server pod — they had only set them on the worker: [Cloud Storage Configs null for GCS] > "I managed to fix it by adding the variables listed above to the server pod!" The relevant variables are: •

STATE_STORAGE_GCS_APPLICATION_CREDENTIALS

•

STATE_STORAGE_GCS_BUCKET_NAME

•

WORKER_LOGS_STORAGE_TYPE=GCS

•

WORKER_STATE_STORAGE_TYPE=GCS

This is also consistent with the first forum post in the knowledge sources, where a user on GKE Autopilot added these same GCS variables and subsequently saw

Missing stats for job

errors in the server logs. [JobHistoryHandler Missing stats warning] Recommendation Before anything else, verify that all four GCS-related environment variables are correctly set on both the server and worker pods. If the server pod is missing them, it may fail to read job/attempt stats from GCS, which would explain the

totalStats

being

null

in your original error. The knowledge sources do not contain specific information about Cloud PostgreSQL causing this type of error, so I cannot draw a direct link there. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai, I am using Airbyte helm chart 2.0.19, the above variables are not there, but I did find that both server and worker pods have these common envvars (not sure if they are relevant): • STORAGE_TYPE=gcs • STORAGE_BUCKET_LOG=my-bucket • STORAGE_BUCKET_STATE=my-bucket • GCS_CREDENTIALS_SECRET_NAME=gcp-sa

Your environment variables look like the correct modern equivalents for Helm chart V2. However, there is a well-documented issue specifically with Helm chart V2 and GCS storage configuration that is very relevant to your situation. Known GCS Issues with Helm Chart V2 (around version 2.0.19) There is an open GitHub issue specifically for chart version v2.0.19 reporting that the documentation and Helm internal checks do not match on case sensitivity for the storage type. The Helm chart has case-sensitive comparisons against

"gcs"

(lowercase), but the documentation shows

"GCS"

(uppercase). When the case doesn't match, GCS credentials are not mounted to pods. [Documentation and helm checks case mismatch] Additionally, there is a separate known bug where the GCP JSON credentials are double base64 encoded in the

airbyte-secrets

secret, causing workers to fail to authenticate with GCS. [Double base64 encoded credentials] Key Things to Check 1. Case sensitivity: In your

values.yaml

, ensure

storage.type

is lowercase

"gcs"

, not

"GCS"

. [Documentation and helm checks case mismatch] 2. Credentials key: In Helm V2, the correct key is

credentialsJsonSecretKey

(not

credentialsJsonPath

). A working V2 configuration looks like: [Container launcher not using gcs-log-creds]

global:
  storage:
    type: "gcs"
    secretName: airbyte-config-secrets
    bucket:
      log: my-bucket
      state: my-bucket
      workloadOutput: my-bucket
    gcs:
      projectId: <your-project-id>
      credentialsJsonSecretKey: "gcp.json"

3. Container orchestrator secret mounting: There is a known bug where GCS credentials are not mounted to job pods. A documented workaround is to explicitly set: [GCP JSON not mounted to job pods]

global:
  workloads:
    containerOrchestrator:
      secretName: "airbyte-gcs-logs-creds"
      secretMountPath: "/secrets/gcs-log-creds"

(Replace

"airbyte"

in the secret name with your Helm release name.) Given that your sync was working fine until recently with no changes, it's possible a transient GCS authentication failure caused the stats not to be written/read correctly, resulting in

totalStats

being

null

. The knowledge sources do not explicitly confirm this causal chain, but misconfigured or inaccessible GCS credentials in V2 charts is a well-documented source of silent failures. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.