https://linen.dev logo
Join Slack
Powered by
Hi <@U04S8PLJWMT> i am seeing these errors in my p...
# replication-ask-ai
i
Hi @kapa.ai i am seeing these errors in my pipeline io.airbyte.commons.server.errors.IdNotFoundKnownException: Could not find attempt stats for job_id: 62971 and attempt no: 0 Warning from airbyte_platform: Something went wrong within the airbyte platform Activity with activityType='CreateNewAttemptNumber' failed: 'Activity task failed'. scheduledEventId=85, startedEventId=94, activityId=c69ab741-ea31-30bb-bfa0-2d059d5ee905, identity='1@airbyte-worker-7dc57677cf-dlxsv', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED
k
I’m sorry, but the knowledge sources don’t contain a concrete root cause or fix for this specific 
IdNotFoundKnownException
/ “Could not find attempt stats” error, so I can only give partial guidance. From what we do have: - Similar log lines like 
Missing stats for job X attempt Y
are known in Airbyte and come from the server trying to read per‑attempt statistics from the job persistence layer and not finding them. [[JobHistoryHandler warning](https://discuss.airbyte.io/t/4016); [JobHistoryHandler noise](https://discuss.airbyte.io/t/3804)] - In those forum threads, the jobs themselves were running successfully; the issue was mostly noisy logs rather than failed syncs. [[JobHistoryHandler warning](https://discuss.airbyte.io/t/4016); [JobHistoryHandler noise](https://discuss.airbyte.io/t/3804)] Your case is more severe because: - The exception is thrown as 
io.airbyte.commons.server.errors.IdNotFoundKnownException: Could not find attempt stats for job_id: 62971 and attempt no: 0
- Temporal reports 
ActivityType='CreateNewAttemptNumber'
failed with 
RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED
, which means the platform retried that internal activity until it hit its retry limit. The knowledge sources do not document: - Why 
CreateNewAttemptNumber
would fail specifically due to missing attempt stats. - Any official remediation steps for this exact 
IdNotFoundKnownException
path. Given those limits, the only grounded suggestions I can offer are: 1. Check whether the affected job actually ran or produced any data. In the similar “missing stats” cases, the jobs themselves were OK and only the stats lookup failed. [[JobHistoryHandler warning](https://discuss.airbyte.io/t/4016)] 2. Inspect job history and attempts for job 62971 in the UI or via API. Confirm: - Does job 62971 exist? - Does it show attempt 0? - Is there any indication of partial metadata (e.g., job exists but attempts list is empty)? 3. Look for broader job/attempt issues after upgrades or config changes. There is at least one example where an upgrade caused job/attempt–related errors (
IdNotFoundKnownException
in a different code path) and all syncs started failing after the version bump. [[Upgrade 0.40.4 issue](https://discuss.airbyte.io/t/2471)] If you recently: - Upgraded Airbyte, or - Changed storage backends (logs/state/job persistence), it’s worth correlating the timing. 4. Collect full server logs around the first occurrence. The snippets in the knowledge sources show that the important context is often a few lines before/after the “missing stats” message. [[JobHistoryHandler warning](https://discuss.airbyte.io/t/4016)] Look for: - Any stack trace around 
CreateNewAttemptNumber
- Any errors from job persistence or database access - Any migration or schema errors if this followed an upgrade Because the provided documentation and forum posts don’t describe this exact failure mode (CreateNewAttemptNumber + IdNotFoundKnownException on attempt stats), I can’t reliably tell you whether this is: - A benign stats‑metadata issue, or - A deeper corruption / persistence / migration problem in your deployment. Given that the Temporal activity is failing with maximum retries reached, I would treat it as a real platform issue and open a GitHub issue or forum post with: - Your Airbyte version - Deployment type (Docker / K8s, etc.) - Any recent upgrades or storage changes - The full stack trace around 
IdNotFoundKnownException
and 
CreateNewAttemptNumber
- Whether job 62971 appears correctly in the UI That will let the Airbyte team map it to the known “missing stats” behavior or identify a new bug.
i
@kapa.ai does temporal loads up the database ?
k
A message from kapa.ai
The knowledge sources provided do not explicitly describe how Temporal interacts with Airbyte’s databases, so I can’t give a definitive, detailed answer. From what is shown: - Temporal appears in stack traces as the workflow engine orchestrating activities like 
CreateNewAttemptNumber
, 
RunWithJobOutput
, 
CheckConnectionActivityImpl
, etc. These activities then call Airbyte server APIs or persistence layers, which in turn read/write to the jobs and attempts tables in the Jobs Database. [[Jobs DB schema](https://docs.airbyte.com/platform/understanding-airbyte/database-data-catalog#jobs-database)] - The database schema and access patterns (tables 
jobs
, 
attempts
, 
airbyte_metadata
) are documented as part of Airbyte’s own persistence layer, not Temporal’s. [[Jobs DB schema](https://docs.airbyte.com/platform/understanding-airbyte/database-data-catalog#jobs-database)] What is not documented in these sources: - A clear statement that “Temporal loads up the database” or directly manages the database contents. - Any description of Temporal performing schema migrations or bulk loading data into the Jobs Database. Based on the available docs, all we can say is: - Temporal runs workflows and activities. - Those activities use Airbyte’s server/persistence code, which is what actually reads/writes the Jobs Database. [[Jobs DB schema](https://docs.airbyte.com/platform/understanding-airbyte/database-data-catalog#jobs-database)] Anything more specific about Temporal “loading” the database would go beyond the information in the provided sources.
i
@kapa.ai what should be the spec for the airbyte's database ? Is there any recoemmendation ?
k
A message from kapa.ai
Airbyte’s docs don’t give a full “DB sizing spec” (CPU/RAM/IOPS), but they do contain a few concrete recommendations and constraints. ### 1. Use a dedicated Postgres instance For production, Airbyte recommends using an external, dedicated Postgres database (e.g. AWS RDS, GCP Cloud SQL) instead of the default in‑cluster Postgres. [[External DB](https://docs.airbyte.com/platform/deploying-airbyte/integrations/database)] > “For production deployments, we recommend using a dedicated database instance for better reliability, and backups (such as AWS RDS or GCP Cloud SQL) instead of the default internal Postgres database (
airbyte/db
) that Airbyte spins up within the Kubernetes cluster.” [[External DB](https://docs.airbyte.com/platform/deploying-airbyte/integrations/database)] ### 2. How to configure an external DB You disable the bundled Postgres and point Airbyte to your external instance via 
values.yaml
and a Kubernetes Secret: [[External DB](https://docs.airbyte.com/platform/deploying-airbyte/integrations/database)]
Copy code
yaml # values.yaml (Helm chart V1 example) postgresql:   enabled: false  global:   database:     type: external      # Secret name where database credentials are stored     secretName: "airbyte-config-secrets"      # The database host     host: "your-db-host"      # The database port     port: "5432"      # The database name     database: "airbyte"   # or "name" for Helm V2      # Keys in the secret for user/password     userSecretKey: "database-user"     passwordSecretKey: "database-password"
Secret example: [[External DB](https://docs.airbyte.com/platform/deploying-airbyte/integrations/database)]
Copy code
yaml apiVersion: v1 kind: Secret metadata:   name: airbyte-config-secrets type: Opaque stringData:   database-user: airbyte   database-password: your-password
### 3. DB-related configuration variables Airbyte expects standard Postgres connection parameters; you can override them via environment variables (internally mapped from 
values.yaml
): [[Configuring Airbyte](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte)] - 
DATABASE_USER
/ 
DATABASE_PASSWORD
- 
DATABASE_URL
– 
jdbc:postgresql://${DATABASE_HOST}:${DATABASE_PORT}/${DATABASE_DB}
- 
CONFIG_DATABASE_USER
/ 
CONFIG_DATABASE_PASSWORD
/ 
CONFIG_DATABASE_URL
- 
RUN_DATABASE_MIGRATION_ON_STARTUP
(whether to run migrations at boot) These don’t specify sizing, but they show that Airbyte uses two logical DBs (Jobs and Config) which can share the same Postgres instance. [[Configuring Airbyte](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte)] ### 4. Indirect hints from scaling docs The scaling guide focuses on CPU/RAM/disk for the Kubernetes nodes and job pods, not the DB itself, but it does highlight that Temporal (used by Airbyte) maintains multiple idle DB connections and may open more than expected: [[Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#temporal-db)] > “Temporal maintains multiple idle connections… might end up allowing 4–5 times more connections than expected… If you want to increase the amount of allowed idle connections, you will also need to increase 
SQL_MAX_CONNS
as well because 
SQL_MAX_IDLE_CONNS
is capped by 
SQL_MAX_CONNS
.” This implies your Postgres instance should be sized to handle a moderate number of concurrent connections (from Temporal + Airbyte services), but no specific numbers are given. --- Summary: The official docs recommend: - Use a dedicated, managed Postgres instance for production (RDS/Cloud SQL). [[External DB](https://docs.airbyte.com/platform/deploying-airbyte/integrations/database)] - Configure it via 
values.yaml
and a Kubernetes Secret as shown above. - Ensure it can handle multiple concurrent connections (Temporal + Airbyte), but there is no explicit guidance in the docs on CPU/RAM/IOPS or exact connection limits.
i
@kapa.ai what is the role of temporal in airbyte ?
k
A message from kapa.ai
Temporal is the orchestration engine Airbyte uses to run and manage jobs (syncs, checks, discoveries) reliably and at scale. From the docs: - Temporal is part of Airbyte’s core tech stack and is used specifically for orchestration. [[Tech stack](https://docs.airbyte.com/platform/understanding-airbyte/tech-stack)] - The Temporal Service (
airbyte-temporal
) “manages the scheduling and sequencing task queues and workflows.” [[Architecture overview](https://docs.airbyte.com/platform/understanding-airbyte/high-level-view)] - The Worker service (
airbyte-worker
) reads from Temporal’s task queues and executes the connection scheduling and sequencing logic, then calls the Workload API to launch the actual connector pods. [[Architecture overview](https://docs.airbyte.com/platform/understanding-airbyte/high-level-view)] Why Airbyte uses Temporal (from the FAQ): - It solves the challenge of orchestrating hundreds to thousands of jobs simultaneously, by: - Scaling state management (job state is serialized into a DB so it can be recovered). - Providing robust queue management for workflows and activities. - Because a job’s entire state is stored, it’s easy to: - Recover from failures. - Determine if a job was assigned and executed correctly. [[Tech stack](https://docs.airbyte.com/platform/understanding-airbyte/tech-stack)] So, in short: Temporal is the workflow engine that schedules, sequences, and tracks Airbyte jobs, while Airbyte’s worker and workload components do the actual data movement work under Temporal’s orchestration.
i
@kapa.ai i am seeing this in temporal {"level":"info","ts":"2025-12-20T202145.366Z","msg":"history client encountered error","service":"frontend","error":"consistent query buffer is full, this may be ca used by too many queries and workflow not able to process query fast enough","service-error-type":"serviceerror.ResourceExhausted","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/c lient/history/metric_client.go:103"}
k
A message from kapa.ai
This Temporal log message is known, but the knowledge sources only give partial context and no Airbyte-specific fix, so I can’t provide a definitive remediation. What it means (from Temporal / Airbyte usage): - The error is: 
consistent query buffer is full, this may be caused by too many queries and workflow not able to process query fast enough
with 
serviceerror.ResourceExhausted
. - This is a Temporal history/frontend service error indicating that the internal buffer for “consistent queries” is full because workflows aren’t processing them quickly enough. Where it shows up with Airbyte: - A very similar error appears in Airbyte deployments when Temporal is under pressure; for example, users have seen: 
Internal service error ... "consistent query buffer is full, cannot accept new consistent queries"
in the Temporal pod, associated with stuck or failing jobs. [[Normalization pod issue](https://discuss.airbyte.io/t/3132); [JobStatus FAILED thread](https://discuss.airbyte.io/t/3279)] - In those cases, Airbyte became unresponsive until the Temporal / Airbyte services were restarted. [[JobStatus FAILED thread](https://discuss.airbyte.io/t/3279)] What the docs/forums suggest indirectly: - The linked Temporal GitHub issue in one thread indicates this is a Temporal load / configuration problem rather than a connector bug. [[Normalization pod issue](https://discuss.airbyte.io/t/3132)] - Other Temporal-related 
RESOURCE_EXHAUSTED
and 
DEADLINE_EXCEEDED
errors in Airbyte are often correlated with insufficient resources (CPU/memory/DB performance) or Temporal limits being hit; increasing resources or upgrading to newer Airbyte versions with better Temporal defaults has helped in some cases. [[Salesforce DEADLINE_EXCEEDED](https://discuss.airbyte.io/t/1612); [DEADLINE_EXCEEDED workaround](https://discuss.airbyte.io/t/254); [Temporal RESOURCE_EXHAUSTED fix](https://github.com/airbytehq/airbyte/issues/48879)] Given the limited, indirect information, practical steps you can take: 1. Check if Airbyte jobs are hanging or the UI/API is slow. In prior reports, this error coincided with stuck syncs or an unresponsive instance. [[Normalization pod issue](https://discuss.airbyte.io/t/3132); [JobStatus FAILED thread](https://discuss.airbyte.io/t/3279)] 2. Temporarily restart Temporal / Airbyte services. In the outage example, restarting the containers cleared the “consistent query buffer is full” condition. [[JobStatus FAILED thread](https://discuss.airbyte.io/t/3279)] 3. Review load and resource usage: - How many syncs / workflows are running concurrently? - Are your Temporal and DB pods close to CPU/memory limits? Similar Temporal errors in Airbyte have been mitigated by increasing memory/CPU on nodes or Temporal. [[Salesforce DEADLINE_EXCEEDED](https://discuss.airbyte.io/t/1612); [DEADLINE_EXCEEDED workaround](https://discuss.airbyte.io/t/254)] 4. Check your Airbyte version and consider upgrading. Some Temporal-related 
RESOURCE_EXHAUSTED
issues were addressed by improved Temporal configuration in newer Airbyte releases. [[Temporal config fix](https://github.com/airbytehq/airbyte/issues/48879)] Because the sources don’t document a specific configuration knob in Airbyte to tune this “consistent query buffer” behavior, if the problem persists even after reducing load and ensuring sufficient resources, I’d recommend opening an issue or forum topic with: - Airbyte version and deployment type - Temporal pod logs around this error - Approximate number of concurrent jobs / connections so the Airbyte team can map it to known Temporal behavior or suggest version-specific fixes.
i
@kapa.ai does this make sense ? Activity failure size exceeds warning limit for mutable state
k
A message from kapa.ai
The knowledge sources don’t mention the exact message 
Activity failure size exceeds warning limit for mutable state
, but they do show closely related Temporal errors, so I can only answer partially. What we do see: - Temporal in Airbyte can emit size/limit‑related errors such as: - 
ScheduleActivityTaskCommandAttributes.Input exceeds size limit.
[[MySQL→Snowflake issue](https://discuss.airbyte.io/t/2991)] - 
consistent query buffer is full, cannot accept new consistent queries
/ 
serviceerror.ResourceExhausted
. [[Normalization pod issue](https://discuss.airbyte.io/t/3132); [JobStatus FAILED thread](https://discuss.airbyte.io/t/3279)] - These are all Temporal “resource/size” safeguards: when inputs, history, or internal buffers get too large, Temporal logs warnings or errors and may reject new work. So your message: > 
Activity failure size exceeds warning limit for mutable state
is consistent with that pattern: Temporal is warning that the recorded failure data for an activity (what gets stored in workflow mutable state/history) is large enough to cross a warning threshold. It’s essentially a size/limit warning from Temporal, not an Airbyte‑specific message. From the available info, the implications are: - The message itself “makes sense” in the context of Temporal’s behavior. - It likely indicates large error payloads (e.g., big stack traces or details) being stored in workflow state. - The Airbyte docs and forum posts do not provide a specific configuration or fix for this exact warning; they only show that similar size/limit issues can appear under heavy load or with large payloads. [[MySQL→Snowflake issue](https://discuss.airbyte.io/t/2991); [Normalization pod issue](https://discuss.airbyte.io/t/3132)] If this is only a warning and your syncs succeed, it may just be noise. If workflows start failing or getting 
RESOURCE_EXHAUSTED
errors alongside this, it’s a sign that Temporal is hitting internal limits and you may need to: - Reduce payload sizes (e.g., avoid extremely large error messages), or - Review load/resources and Airbyte version as in the other Temporal‑related threads.
5 Views
Open in Slack
PreviousNext
Powered by