https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> ```2026-02-16 00:51:46 source WARN ...
# replication-ask-ai
q
@kapa.ai
Copy code
2026-02-16 00:51:46 source WARN debezium-mongodbconnector-mongodb_srv___axlehire-prod-us-west1.vuags.mongodb.net_-replicator-fetcher-0 i.d.c.m.e.BufferingChangeStreamCursor$EventFetcher(enqueue):285 Unable to acquire buffer lock, buffer queue is likely full
2026-02-16 00:51:46 source WARN debezium-mongodbconnector-mongodb_srv___axlehire-prod-us-west1.vuags.mongodb.net_-replicator-fetcher-0 i.d.c.m.e.BufferingChangeStreamCursor$EventFetcher(enqueue):285 Unable to acquire buffer lock, buffer queue is likely full
k
The knowledge sources don’t explicitly mention the 
BufferingChangeStreamCursor$EventFetcher(enqueue): Unable to acquire buffer lock, buffer queue is likely full
warning, so I can’t give a definitive, documented root cause. However, there are closely related details about Debezium’s MongoDB CDC buffering and queue saturation. From the logs in the MongoDB CDC issues, Airbyte uses an internal bounded queue between Debezium’s change-stream fetcher threads and the Airbyte reader. When that queue reaches its capacity (e.g., size=1000, cap=0), Debezium logs that it is blocked or that the queue is full. [[Mongo OOM issue](https://github.com/airbytehq/airbyte/issues/48851)] In your case, the warning strongly suggests: - Debezium’s internal buffer/queue for MongoDB change events is full. - The consumer side (Airbyte’s DebeziumRecordIterator / source pipeline) is not draining events as fast as Debezium is fetching them. Related symptoms in the issues include: - Repeated logs about CDC queue stats at max size and being blocked on 
poll()
/ 
put()
. [[Mongo OOM issue](https://github.com/airbytehq/airbyte/issues/48851)] - Problems when processing large oplogs or high-activity collections, where timeouts and batching behavior can cause backpressure. [[Mongo 101-records issue](https://github.com/airbytehq/airbyte/issues/71046)] Given what’s documented, practical implications are: - It’s a backpressure signal: Debezium is still reading from MongoDB but can’t enqueue more events. - In heavy-change or large-oplog scenarios, you may need to tune connector settings (e.g., 
initial_wait_time
for CDC, sync frequency) or reduce the number/volume of streams so the consumer can keep up. [[Mongo 101-records issue](https://github.com/airbytehq/airbyte/issues/71046)] Because the docs and issues don’t provide a specific remediation for this exact log line, I can’t point to an official “fix,” only to the general pattern: queue saturation between Debezium’s fetcher and Airbyte’s reader. If you’re seeing stalled or failing syncs along with this warning, the next step would be to: - Check for accompanying Debezium queue stats or OOM/timeouts in the same run. [[Mongo OOM issue](https://github.com/airbytehq/airbyte/issues/48851)] - Review the MongoDB CDC troubleshooting guidance (oplog size/retention, document size, cluster tier) to ensure the environment isn’t compounding the issue. [[MongoDB troubleshooting](https://docs.airbyte.com/integrations/sources/mongodb-v2/mongodb-v2-troubleshooting)]
q
@kapa.ai how to change queue size
k
You can change the queue size in the MongoDB v2 source configuration (UI or via Terraform). 1. In the Airbyte UI For a MongoDB v2 source: 1. Go to Sources → your MongoDB v2 source → Edit. 2. Expand Advanced settings. 3. Find “Size of the queue (Advanced)”. 4. Set the desired value (default is *10,000*; changing it affects memory usage and efficiency, so be careful). [Mongo config params; CDC best practices] 2. Via Terraform Use the 
queue_size
field in `airbyte_source_mongodb_v2`:
Copy code
resource "airbyte_source_mongodb_v2" "my_source_mongodbv2" {
  configuration = {
    database_config = {
      self_managed_replica_set = {
        auth_source       = "admin"
        connection_string = "mongodb://example1.host.com:27017/"
        databases         = ["mydb"]
        password          = "..."
        username          = "..."
      }
    }
    queue_size = 4000  # <--- change queue size here
  }

  definition_id = "b2e713cd-cc36-4c0a-b5bd-b47cb8a0561e"
  name          = "my-mongodb-source"
  workspace_id  = "..."
}
[Terraform mongodb_v2] Best practice from CDC docs is to keep this at the default 10,000 unless you have a specific reason or guidance from support, since it directly impacts memory and stability. [CDC best practices] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
q
@kapa.ai i change Size of the queue (Advanced) to 40000 but in log still
Copy code
Queue size is overridden to 10000
k
The knowledge sources indicate that MongoDB v2’s queue size is hard-capped at 10,000, regardless of what you set in the UI. In the MongoDB v2 issue logs, when a higher value is configured, Airbyte logs: > 
Queue size is overridden to 10000 , which is the max allowed for safety.
[Mongo offset issue] And in the CDC best practices, 10,000 is documented as the recommended/default queue size, with a warning that improper sizing can impact memory and stability. [CDC best practices] So: • Your change to 40,000 is being ignored and clamped to 10,000 by the connector. • This is expected behavior; 10,000 is the maximum allowed queue size for MongoDB v2 in Airbyte today. If you’re hitting buffer-full warnings, you’ll need to address them via other levers (e.g., reducing load per sync, adjusting 
initial_waiting_seconds
, sync frequency, or MongoDB/cluster-side tuning), not by increasing 
queue_size
beyond 10,000. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
q
@kapa.ai i change Size of the queue (Advanced) to 40000 but in log still
Copy code
Queue size is overridden to 10000
k
Airbyte is intentionally overriding your setting. For the MongoDB v2 source, when you set Size of the queue (Advanced) above a safety limit, the connector clamps it and logs: > 
Queue size is overridden to 10000 , which is the max allowed for safety.
[Mongo offset issue] So even if you configure 
40000
, the effective queue size is forced back to 10,000, which is both the default and maximum allowed value for this connector. [Mongo config params; CDC best practices] In short: this behavior is expected; you currently cannot increase the MongoDB v2 CDC queue size beyond 10,000. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
q
@kapa.ai
Copy code
2026-02-15 23:50:27 source INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):48 CDC events queue stats: size=9161, cap=839, puts=6, polls=0
2026-02-15 23:50:29 source INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):87 CDC events queue poll(): blocked for PT0.0000277S after 86 previous call(s) which were not logged.
2026-02-15 23:50:29 source INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):184 CDC events queue poll(): returned a change event with "source": {"version":"2.6.2.Final","connector":"mongodb","name":"mongodb_srv___axlehire-prod-us-west1.vuags.mongodb.net_","ts_ms":1771173698000,"snapshot":null,"db":"prod","sequence":null,"ts_us":1771173698000000,"ts_ns":1771173698000000000,"collection":"routing_route","ord":332,"lsid":null,"txnNumber":null,"wallTime":1771173698543}.
2026-02-15 23:50:29 source WARN debezium-mongodbconnector-mongodb_srv___axlehire-prod-us-west1.vuags.mongodb.net_-replicator-fetcher-0 i.d.c.m.e.BufferingChangeStreamCursor$EventFetcher(enqueue):285 Unable to acquire buffer lock, buffer queue is likely full
2026-02-15 23:50:30 source WARN debezium-mongodbconnector-mongodb_srv___axlehire-prod-us-west1.vuags.mongodb.net_-replicator-fetcher-0 i.d.c.m.e.BufferingChangeStreamCursor$EventFetcher(enqueue):285 Unable to acquire buffer lock, buffer queue is likely full
2026-02-15 23:50:31 source WARN debezium-mongodbconnector-mongodb_srv___axlehire-prod-us-west1.vuags.mongodb.net_-replicator-fetcher-0 i.d.c.m.e.BufferingChangeStreamCursor$EventFetcher(enqueue):285 Unable to acquire buffer lock, buffer queue is likely full
2026-02-15 23:50:32 source WARN debezium-mongodbconnector-mongodb_srv___axlehire-prod-us-west1.vuags.mongodb.net_-replicator-fetcher-0 i.d.c.m.e.BufferingChangeStreamCursor$EventFetcher(enqueue):285 Unable to acquire buffer lock, buffer queue is likely full
2026-02-15 23:50:33 source WARN debezium-mongodbconnector-mongodb_srv___axlehire-prod-us-west1.vuags.mongodb.net_-replicator-fetcher-0 i.d.c.m.e.BufferingChangeStreamCursor$EventFetcher(enqueue):285 Unable to acquire buffer lock, buffer queue is likely full
2026-02-15 23:50:34 source WARN debezium-mongodbconnector-mongodb_srv___axlehire-prod-us-west1.vuags.mongodb.net_-replicator-fetcher-0 i.d.c.m.e.BufferingChangeStreamCursor$EventFetcher(enqueue):285 Unable to acquire buffer lock, buffer queue is likely full
2026-02-15 23:50:34 source INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):87 CDC events queue poll(): blocked for PT0.00001723S after 88 previous call(s) which were not logged.
2026-02-15 23:50:34 source INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):184 CDC events queue poll(): returned a change event with "source": {"version":"2.6.2.Final","connector":"mongodb","name":"mongodb_srv___axlehire-prod-us-west1.vuags.mongodb.net_","ts_ms":1771173698000,"snapshot":null,"db":"prod","sequence":null,"ts_us":1771173698000000,"ts_ns":1771173698000000000,"collection":"routing_route","ord":456,"lsid":null,"txnNumber":null,"wallTime":1771173698789}.
2026-02-15 23:50:36 source WARN debezium-mongodbconnector-mongodb_srv___axlehire-prod-us-west1.vuags.mongodb.net_-replicator-fetcher-0 i.d.c.m.e.BufferingChangeStreamCursor$EventFetcher(enqueue):285 Unable to acquire buffer lock, buffer queue is likely full
k
The knowledge sources don’t describe this exact Debezium warning in detail, but they do show the same CDC queue behavior and limits, so the explanation below is based on that. From your log:
Copy code
CDC events queue stats: size=9161, cap=839, puts=6, polls=0
...
BufferingChangeStreamCursor$EventFetcher(enqueue): Unable to acquire buffer lock, buffer queue is likely full
What this means in Airbyte’s MongoDB v2 connector: 1. CDC queue is near its max capacity Airbyte uses an internal bounded queue between Debezium (which reads MongoDB change streams) and the Airbyte pipeline (which consumes those events). When the queue is close to full, you see stats like 
size=9161, cap=839
(size + cap ≈ 10,000). Similar stats appear in other MongoDB v2 runs. [Mongo offset issue; Mongo OOM issue] 2. Debezium can’t enqueue more events The warning:
Copy code
Unable to acquire buffer lock, buffer queue is likely full
indicates Debezium’s fetcher thread is trying to push more change events into its buffer/queue but can’t, because the queue is saturated and the lock can’t be acquired quickly. This is consistent with the “queue is full / blocked” patterns seen in other MongoDB CDC logs. [Mongo OOM issue] 3. Raising “Size of the queue” above 10,000 won’t help Even if you set Size of the queue (Advanced) to 40,000, the connector clamps it:
Copy code
Queue size is overridden to 10000 , which is the max allowed for safety.
So the effective maximum queue size is 10,000, and that’s what you’re seeing (size ≈ 9,161, remaining capacity ≈ 839). [Mongo offset issue; Mongo config params] ------------------------- What you can adjust (within documented behavior) Since you can’t increase the queue beyond 10,000, mitigation has to come from reducing pressure or giving Debezium/Airbyte more time: • Keep queue_size ≤ 10,000 (anything higher is forced to 10,000). [Mongo offset issue; CDC best practices] • Tune 
initial_waiting_seconds
 (Initial Waiting Time in Seconds) to better handle large oplogs / high activity; this is explicitly called out as a lever for MongoDB CDC timing issues. [Mongo 101-records issue; Mongo config params] • Review MongoDB CDC troubleshooting items (oplog size/retention, cluster tier, document size) to ensure the environment isn’t compounding backpressure problems. [MongoDB troubleshooting] The knowledge sources don’t provide a connector-level switch to remove this warning or increase the buffer beyond 10,000, so the main takeaway is: the warning is a symptom of the CDC queue running near its hard limit; you need to reduce load per sync or tune timing rather than increasing the queue size. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
q
@kapa.ai i already set Initial Waiting Time in Seconds (Advanced) to 1200. that the max value
k
You’re right: for MongoDB v2, 1200 seconds is the documented maximum for Initial Waiting Time in Seconds (Advanced), so you’re already at the upper limit. [Mongo config params; Terraform mongodb_v2] Given the knowledge sources, there are no additional tunables beyond: • Queue size: hard-capped at 10,000, even if you configure more. [Mongo offset issue; CDC best practices] • Initial Waiting Time: max 1200s, which you already use. [Mongo config params; Mongo 101-records issue] • Initial Load Timeout in Hours: controls how long the initial snapshot can run before switching to CDC. [Mongo config params; CDC best practices 2.0] The docs and issues don’t provide a further documented way to eliminate the “buffer queue is likely full” warnings once you’ve hit those limits. At this point, the only options mentioned in the knowledge sources are more “architectural”: • Reduce load per sync (fewer high‑churn collections, or split into multiple connections) so the consumer can drain the queue faster. • Increase sync frequency so each run processes fewer accumulated changes, which is also recommended in the MongoDB troubleshooting guide to avoid oplog/CDC pressure. [MongoDB troubleshooting] • Ensure oplog size/retention is sufficient so CDC isn’t also fighting oplog churn. [MongoDB troubleshooting] The knowledge sources do not describe any additional MongoDB v2 connector setting that would further increase buffer capacity or relax this behavior, so I can’t point you to another knob beyond these. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
q
explain for Queue size what if i set 40000 and what if i set 10000 @kapa.ai
k
For MongoDB v2 in Airbyte, the behavior is: • The “Size of the queue (Advanced)” controls the size of the internal CDC events buffer in memory. Larger = can absorb more change events during bursts, but uses more memory; smaller = less memory, but less tolerant of bursts. [Mongo config params; CDC best practices] • However, for this connector there is a hard maximum of 10,000. If you configure a higher value (e.g. 
40000
), the connector overrides it back to 10,000 and logs: > 
Queue size is overridden to 10000 , which is the max allowed for safety.
[Mongo offset issue] So in practice: • If you set 10,000 ◦ This is both the default and maximum effective queue size. ◦ Airbyte uses a queue of 10,000 CDC events. [Mongo config params; CDC best practices] • If you set 40,000 ◦ The connector ignores the extra and clamps it to 10,000 for safety. ◦ You will still see logs indicating the queue size is overridden to 10,000, and the effective behavior is identical to setting 10,000. [Mongo offset issue] There is no documented way in the current MongoDB v2 connector to use a queue larger than 10,000. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
16 Views
Open in Slack
PreviousNext
Powered by