https://linen.dev logo
Join Slack
Powered by
what are my options to speed up a mongodb cdc back...
# replication-ask-ai
a
what are my options to speed up a mongodb cdc backfill job? It has been 24 hours for less than 40Gb processed. Size of the queue? Pod resources?
k
There are a few options you can consider to speed up a MongoDB CDC backfill job: 1. Adjust the Queue Size: The allocated queue size used to buffer Debezium events is hard coded to 10,000. Making this a configurable field could potentially improve the speed of your CDC backfill job. This is similar to what was done for the MongoDB v2 connector. However, this would require changes in the codebase. [Source](https://github.com/airbytehq/airbyte/issues/21498) 2. Increase Pod Resources: If you're running Airbyte on Kubernetes, you could try increasing the resources allocated to your worker pods. This could potentially speed up the processing of your CDC backfill job. For example, in a forum post, a user was advised to increase the worker memory when facing sync job failures. [Source](https://discuss.airbyte.io/t/705) 3. Reduce Sync Timeout: If your CDC cursor is becoming invalid due to long sync times, you could consider reducing the default sync timeout. For instance, for source-mongodb connections, it was suggested to reduce the default value from 24 hours to 6 hours to prevent users from running into an error state. [Source](https://github.com/airbytehq/airbyte/issues/36457) Remember to monitor your system's performance and adjust these settings as necessary to find the optimal configuration for your specific use case.
a
@kapa.ai what queue size do you recommend then ? my worker has 6gb of memory
k
New message text here
m
@kapa.ai Is allocated queue size used to buffer Debezium hardcoded? Is there any way to make it bigger?
k
A message from kapa.ai
Based on the available knowledge sources, the allocated queue size used to buffer Debezium events in Airbyte is hardcoded in some connectors. For example, in the context of MongoDB CDC, the queue size is mentioned as being hardcoded to 10,000, and making it configurable is suggested as a potential improvement. However, as of the referenced discussion, there is no built-in way for users to increase this queue size through configuration—it would require changes in the codebase to make it adjustable. So, currently, you cannot make the Debezium buffer queue bigger via configuration; it is hardcoded and not user-configurable at this time [source-jira](https://github.com/airbytehq/airbyte/issues/21498).
m
@kapa.ai What is queue size advanced parameter in Mongo DB source connector then?
k
A message from kapa.ai
The "Size of the queue" advanced parameter in the MongoDB source connector controls the size of the internal queue used by the connector. This queue buffers records/events as they are read from MongoDB before they are processed further. The documentation specifically notes: > The size of the internal queue. This may interfere with memory consumption and efficiency of the connector, please be careful. Adjusting this parameter can impact both performance and memory usage: increasing the queue size may help with throughput but will also increase memory consumption. The default and valid range for this parameter is not explicitly stated in the documentation, but it is exposed as a configurable field in the connector settings [MongoDB Source Reference](https://docs.airbyte.com/integrations/sources/mongodb-v2#reference) [Configuration Parameters](https://docs.airbyte.com/integrations/sources/mongodb-v2#configuration-parameters). So, unlike some other connectors where the queue size is hardcoded, in the MongoDB v2 source connector, you can adjust the queue size via this advanced parameter.
m
@kapa.ai Is it the same as debezium buffer?
k
A message from kapa.ai
The knowledge sources do not explicitly state whether the "Size of the queue" advanced parameter in the MongoDB source connector is exactly the same as the Debezium buffer queue size. The documentation describes it as "the size of the internal queue" that can affect memory consumption and efficiency, but does not clarify if this queue is directly the Debezium event buffer or a separate Airbyte-managed queue [Configuration Parameters](https://docs.airbyte.com/integrations/sources/mongodb-v2#configuration-parameters). Therefore, while it is likely related to buffering records/events during CDC, I cannot confirm with certainty from the provided sources that it is the same as the Debezium buffer. If you need a definitive answer, further investigation in the connector's implementation or official clarification from the Airbyte team would be required.
5 Views
Open in Slack
PreviousNext
Powered by